Repository navigation
Reimplemented parse_anything.py to handle SBOMs with incorrect file extensions. #424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,33 +22,25 @@ | |
|
|
||
|
|
||
| def parse_file(fn): | ||
| buildermodule = jsonyamlxmlbuilders | ||
| read_data = False | ||
| if fn.endswith(".rdf") or fn.endswith(".rdf.xml"): | ||
| parsing_module = rdf | ||
| buildermodule = rdfbuilders | ||
| elif fn.endswith(".spdx"): | ||
| parsing_module = rdf | ||
| buildermodule = rdfbuilders | ||
| elif fn.endswith(".tag"): | ||
| parsing_module = tagvalue | ||
| buildermodule = tagvaluebuilders | ||
| read_data = True | ||
| elif fn.endswith(".json"): | ||
| parsing_module = jsonparser | ||
| elif fn.endswith(".xml"): | ||
| parsing_module = xmlparser | ||
| elif fn.endswith(".yaml") or fn.endswith(".yml"): | ||
| parsing_module = yamlparser | ||
| else: | ||
| raise FileTypeError("FileType Not Supported" + str(fn)) | ||
| buildermodules = [rdfbuilders, jsonyamlxmlbuilders, jsonyamlxmlbuilders, jsonyamlxmlbuilders, tagvaluebuilders] | ||
| parsing_modules = [rdf, xmlparser, yamlparser, jsonparser, tagvalue] | ||
| read_datas = [False, False, False, False, True] | ||
| for i in range(len(buildermodules)): | ||
| parsing_module = parsing_modules[i] | ||
| buildermodule = buildermodules[i] | ||
| read_data = read_datas[i] | ||
| try: | ||
| p = parsing_module.Parser(buildermodule.Builder(), StandardLogger()) | ||
| if hasattr(p, "build"): | ||
| p.build() | ||
| with open(fn) as f: | ||
| if read_data: | ||
| data = f.read() | ||
| return p.parse(data) | ||
| else: | ||
| return p.parse(f) | ||
| except: | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This would basically |
||
| if i == len(buildermodules) - 1: | ||
| raise FileTypeError("FileType Not Supported" + str(fn)) | ||
|
|
||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This code will work, but I would personally find it more readable if on the exception you just continued and raise the error after the for loop since if you made it there, all file types would fail parsing. I would also change the message to something like "The file cannot be parsed by the RDF, XML, YAML, JSON or Tag/Value parsers". |
||
|
|
||
| p = parsing_module.Parser(buildermodule.Builder(), StandardLogger()) | ||
| if hasattr(p, "build"): | ||
| p.build() | ||
| with open(fn) as f: | ||
| if read_data: | ||
| data = f.read() | ||
| return p.parse(data) | ||
| else: | ||
| return p.parse(f) | ||
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # example of an SBOM with several packages and filesAnalyzed=False | ||
| # from https://github.com/spdx/spdx-spec/issues/439 | ||
| SPDXID: "SPDXRef-DOCUMENT" | ||
| spdxVersion: "SPDX-2.2" | ||
| creationInfo: | ||
| created: "2020-07-23T18:30:22Z" | ||
| creators: | ||
| - "Organization: Example Inc." | ||
| - "Person: Thomas Steenbergen" | ||
| licenseListVersion: "3.9" | ||
| name: "xyz-0.1.0" | ||
| dataLicense: "CC0-1.0" | ||
| documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz" | ||
| documentDescribes: | ||
| - "SPDXRef-Package-xyz" | ||
| packages: | ||
| - SPDXID: "SPDXRef-Package-xyz" | ||
| summary: "Awesome product created by Example Inc." | ||
| copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved." | ||
| downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78" | ||
| filesAnalyzed: false | ||
| homepage: "https://example.com/products/xyz" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc" | ||
| name: "xyz" | ||
| versionInfo: "0.1.0" | ||
| - SPDXID: "SPDXRef-Package-curl" | ||
| description: "A command line tool and library for transferring data with URL syntax, supporting \ | ||
| HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \ | ||
| IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features." | ||
| copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many | ||
| contributors, see the THANKS file." | ||
| downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz" | ||
| filesAnalyzed: false | ||
| homepage: "https://curl.haxx.se/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "curl" | ||
| name: "curl" | ||
| packageFileName: "./libs/curl" | ||
| versionInfo: "7.70.0" | ||
| - SPDXID: "SPDXRef-Package-openssl" | ||
| description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone." | ||
| copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved." | ||
| downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72" | ||
| filesAnalyzed: false | ||
| homepage: "https://www.openssl.org/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0" | ||
| packageFileName: "./libs/openssl" | ||
| name: "openssl" | ||
| versionInfo: "1.1.1g" | ||
| relationships: | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-curl" | ||
| relationshipType: "CONTAINS" | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-openssl" | ||
| relationshipType: "CONTAINS" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # example of an SBOM with several packages and filesAnalyzed=False | ||
| # from https://github.com/spdx/spdx-spec/issues/439 | ||
| SPDXID: "SPDXRef-DOCUMENT" | ||
| spdxVersion: "SPDX-2.2" | ||
| creationInfo: | ||
| created: "2020-07-23T18:30:22Z" | ||
| creators: | ||
| - "Organization: Example Inc." | ||
| - "Person: Thomas Steenbergen" | ||
| licenseListVersion: "3.9" | ||
| name: "xyz-0.1.0" | ||
| dataLicense: "CC0-1.0" | ||
| documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz" | ||
| documentDescribes: | ||
| - "SPDXRef-Package-xyz" | ||
| packages: | ||
| - SPDXID: "SPDXRef-Package-xyz" | ||
| summary: "Awesome product created by Example Inc." | ||
| copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved." | ||
| downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78" | ||
| filesAnalyzed: false | ||
| homepage: "https://example.com/products/xyz" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc" | ||
| name: "xyz" | ||
| versionInfo: "0.1.0" | ||
| - SPDXID: "SPDXRef-Package-curl" | ||
| description: "A command line tool and library for transferring data with URL syntax, supporting \ | ||
| HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \ | ||
| IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features." | ||
| copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many | ||
| contributors, see the THANKS file." | ||
| downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz" | ||
| filesAnalyzed: false | ||
| homepage: "https://curl.haxx.se/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "curl" | ||
| name: "curl" | ||
| packageFileName: "./libs/curl" | ||
| versionInfo: "7.70.0" | ||
| - SPDXID: "SPDXRef-Package-openssl" | ||
| description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone." | ||
| copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved." | ||
| downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72" | ||
| filesAnalyzed: false | ||
| homepage: "https://www.openssl.org/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0" | ||
| packageFileName: "./libs/openssl" | ||
| name: "openssl" | ||
| versionInfo: "1.1.1g" | ||
| relationships: | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-curl" | ||
| relationshipType: "CONTAINS" | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-openssl" | ||
| relationshipType: "CONTAINS" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # example of an SBOM with several packages and filesAnalyzed=False | ||
| # from https://github.com/spdx/spdx-spec/issues/439 | ||
| SPDXID: "SPDXRef-DOCUMENT" | ||
| spdxVersion: "SPDX-2.2" | ||
| creationInfo: | ||
| created: "2020-07-23T18:30:22Z" | ||
| creators: | ||
| - "Organization: Example Inc." | ||
| - "Person: Thomas Steenbergen" | ||
| licenseListVersion: "3.9" | ||
| name: "xyz-0.1.0" | ||
| dataLicense: "CC0-1.0" | ||
| documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz" | ||
| documentDescribes: | ||
| - "SPDXRef-Package-xyz" | ||
| packages: | ||
| - SPDXID: "SPDXRef-Package-xyz" | ||
| summary: "Awesome product created by Example Inc." | ||
| copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved." | ||
| downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78" | ||
| filesAnalyzed: false | ||
| homepage: "https://example.com/products/xyz" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc" | ||
| name: "xyz" | ||
| versionInfo: "0.1.0" | ||
| - SPDXID: "SPDXRef-Package-curl" | ||
| description: "A command line tool and library for transferring data with URL syntax, supporting \ | ||
| HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \ | ||
| IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features." | ||
| copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many | ||
| contributors, see the THANKS file." | ||
| downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz" | ||
| filesAnalyzed: false | ||
| homepage: "https://curl.haxx.se/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "curl" | ||
| name: "curl" | ||
| packageFileName: "./libs/curl" | ||
| versionInfo: "7.70.0" | ||
| - SPDXID: "SPDXRef-Package-openssl" | ||
| description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone." | ||
| copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved." | ||
| downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72" | ||
| filesAnalyzed: false | ||
| homepage: "https://www.openssl.org/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0" | ||
| packageFileName: "./libs/openssl" | ||
| name: "openssl" | ||
| versionInfo: "1.1.1g" | ||
| relationships: | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-curl" | ||
| relationshipType: "CONTAINS" | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-openssl" | ||
| relationshipType: "CONTAINS" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # example of an SBOM with several packages and filesAnalyzed=False | ||
| # from https://github.com/spdx/spdx-spec/issues/439 | ||
| SPDXID: "SPDXRef-DOCUMENT" | ||
| spdxVersion: "SPDX-2.2" | ||
| creationInfo: | ||
| created: "2020-07-23T18:30:22Z" | ||
| creators: | ||
| - "Organization: Example Inc." | ||
| - "Person: Thomas Steenbergen" | ||
| licenseListVersion: "3.9" | ||
| name: "xyz-0.1.0" | ||
| dataLicense: "CC0-1.0" | ||
| documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz" | ||
| documentDescribes: | ||
| - "SPDXRef-Package-xyz" | ||
| packages: | ||
| - SPDXID: "SPDXRef-Package-xyz" | ||
| summary: "Awesome product created by Example Inc." | ||
| copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved." | ||
| downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78" | ||
| filesAnalyzed: false | ||
| homepage: "https://example.com/products/xyz" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc" | ||
| name: "xyz" | ||
| versionInfo: "0.1.0" | ||
| - SPDXID: "SPDXRef-Package-curl" | ||
| description: "A command line tool and library for transferring data with URL syntax, supporting \ | ||
| HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \ | ||
| IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features." | ||
| copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many | ||
| contributors, see the THANKS file." | ||
| downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz" | ||
| filesAnalyzed: false | ||
| homepage: "https://curl.haxx.se/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "curl" | ||
| name: "curl" | ||
| packageFileName: "./libs/curl" | ||
| versionInfo: "7.70.0" | ||
| - SPDXID: "SPDXRef-Package-openssl" | ||
| description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone." | ||
| copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved." | ||
| downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72" | ||
| filesAnalyzed: false | ||
| homepage: "https://www.openssl.org/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0" | ||
| packageFileName: "./libs/openssl" | ||
| name: "openssl" | ||
| versionInfo: "1.1.1g" | ||
| relationships: | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-curl" | ||
| relationshipType: "CONTAINS" | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-openssl" | ||
| relationshipType: "CONTAINS" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # example of an SBOM with several packages and filesAnalyzed=False | ||
| # from https://github.com/spdx/spdx-spec/issues/439 | ||
| SPDXID: "SPDXRef-DOCUMENT" | ||
| spdxVersion: "SPDX-2.2" | ||
| creationInfo: | ||
| created: "2020-07-23T18:30:22Z" | ||
| creators: | ||
| - "Organization: Example Inc." | ||
| - "Person: Thomas Steenbergen" | ||
| licenseListVersion: "3.9" | ||
| name: "xyz-0.1.0" | ||
| dataLicense: "CC0-1.0" | ||
| documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz" | ||
| documentDescribes: | ||
| - "SPDXRef-Package-xyz" | ||
| packages: | ||
| - SPDXID: "SPDXRef-Package-xyz" | ||
| summary: "Awesome product created by Example Inc." | ||
| copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved." | ||
| downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78" | ||
| filesAnalyzed: false | ||
| homepage: "https://example.com/products/xyz" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc" | ||
| name: "xyz" | ||
| versionInfo: "0.1.0" | ||
| - SPDXID: "SPDXRef-Package-curl" | ||
| description: "A command line tool and library for transferring data with URL syntax, supporting \ | ||
| HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \ | ||
| IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features." | ||
| copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many | ||
| contributors, see the THANKS file." | ||
| downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz" | ||
| filesAnalyzed: false | ||
| homepage: "https://curl.haxx.se/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "curl" | ||
| name: "curl" | ||
| packageFileName: "./libs/curl" | ||
| versionInfo: "7.70.0" | ||
| - SPDXID: "SPDXRef-Package-openssl" | ||
| description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone." | ||
| copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved." | ||
| downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72" | ||
| filesAnalyzed: false | ||
| homepage: "https://www.openssl.org/" | ||
| licenseConcluded: "NOASSERTION" | ||
| licenseDeclared: "Apache-2.0" | ||
| packageFileName: "./libs/openssl" | ||
| name: "openssl" | ||
| versionInfo: "1.1.1g" | ||
| relationships: | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-curl" | ||
| relationshipType: "CONTAINS" | ||
| - spdxElementId: "SPDXRef-Package-xyz" | ||
| relatedSpdxElement: "SPDXRef-Package-openssl" | ||
| relationshipType: "CONTAINS" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a necessary change from my perspective, but it would be nice to attempt parsing based on the file type as the first attempt just for performance reasons.
You could do something like leaving in the code for setting the parsing module and attempting that parse first and only looping through if the parsing_module is not set or if the parse fails.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @goneall here. I would hope that by default the files have correct file types and we could parse them based on that. This as the default first try should therefore improve performance. Clearly, as your example from the DocFest shows, this is not always true, so I understand why this feature can be useful.
Additionally I would generate a list of triplets here and iterate through this list as this would be less error-prone. If for example a new buildder and parser is added, one could easily forget to adapt the
read_dataslist or the order could change so that builder and parser module don't match.