Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 21 additions & 29 deletions spdx/parsers/parse_anything.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,33 +22,25 @@


def parse_file(fn):
buildermodule = jsonyamlxmlbuilders
read_data = False
if fn.endswith(".rdf") or fn.endswith(".rdf.xml"):
parsing_module = rdf
buildermodule = rdfbuilders
elif fn.endswith(".spdx"):
parsing_module = rdf
buildermodule = rdfbuilders
elif fn.endswith(".tag"):
parsing_module = tagvalue
buildermodule = tagvaluebuilders
read_data = True
elif fn.endswith(".json"):
parsing_module = jsonparser
elif fn.endswith(".xml"):
parsing_module = xmlparser
elif fn.endswith(".yaml") or fn.endswith(".yml"):
parsing_module = yamlparser
else:
raise FileTypeError("FileType Not Supported" + str(fn))
buildermodules = [rdfbuilders, jsonyamlxmlbuilders, jsonyamlxmlbuilders, jsonyamlxmlbuilders, tagvaluebuilders]
parsing_modules = [rdf, xmlparser, yamlparser, jsonparser, tagvalue]
read_datas = [False, False, False, False, True]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a necessary change from my perspective, but it would be nice to attempt parsing based on the file type as the first attempt just for performance reasons.

You could do something like leaving in the code for setting the parsing module and attempting that parse first and only looping through if the parsing_module is not set or if the parse fails.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @goneall here. I would hope that by default the files have correct file types and we could parse them based on that. This as the default first try should therefore improve performance. Clearly, as your example from the DocFest shows, this is not always true, so I understand why this feature can be useful.

Additionally I would generate a list of triplets here and iterate through this list as this would be less error-prone. If for example a new buildder and parser is added, one could easily forget to adapt the read_datas list or the order could change so that builder and parser module don't match.

for i in range(len(buildermodules)):
parsing_module = parsing_modules[i]
buildermodule = buildermodules[i]
read_data = read_datas[i]
try:
p = parsing_module.Parser(buildermodule.Builder(), StandardLogger())
if hasattr(p, "build"):
p.build()
with open(fn) as f:
if read_data:
data = f.read()
return p.parse(data)
else:
return p.parse(f)
except:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would basically except everything, I am not sure if it is possible, but I would try to make sure that this only excepts "basic" failure if the parser doesn't match the content of the file. So e.g. if we try to use the rdf parser for a file which has a .json ending but is in tag-value format. This should not except any other Errrors that may occur during parsing, otherwise these errors would simply no longer be visible.

if i == len(buildermodules) - 1:
raise FileTypeError("FileType Not Supported" + str(fn))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code will work, but I would personally find it more readable if on the exception you just continued and raise the error after the for loop since if you made it there, all file types would fail parsing.

I would also change the message to something like "The file cannot be parsed by the RDF, XML, YAML, JSON or Tag/Value parsers".


p = parsing_module.Parser(buildermodule.Builder(), StandardLogger())
if hasattr(p, "build"):
p.build()
with open(fn) as f:
if read_data:
data = f.read()
return p.parse(data)
else:
return p.parse(f)
223 changes: 223 additions & 0 deletions tests/data/parse_anything_test_files/SPDXJsonExample.json

Large diffs are not rendered by default.

223 changes: 223 additions & 0 deletions tests/data/parse_anything_test_files/SPDXJsonExample.json.rdf

Large diffs are not rendered by default.

223 changes: 223 additions & 0 deletions tests/data/parse_anything_test_files/SPDXJsonExample.json.spdx

Large diffs are not rendered by default.

223 changes: 223 additions & 0 deletions tests/data/parse_anything_test_files/SPDXJsonExample.json.spdx.yml

Large diffs are not rendered by default.

223 changes: 223 additions & 0 deletions tests/data/parse_anything_test_files/SPDXJsonExample.json.tag

Large diffs are not rendered by default.

223 changes: 223 additions & 0 deletions tests/data/parse_anything_test_files/SPDXJsonExample.json.xml

Large diffs are not rendered by default.

223 changes: 223 additions & 0 deletions tests/data/parse_anything_test_files/SPDXJsonExample.json.yaml

Large diffs are not rendered by default.

305 changes: 305 additions & 0 deletions tests/data/parse_anything_test_files/SPDXRdfExample.rdf

Large diffs are not rendered by default.

305 changes: 305 additions & 0 deletions tests/data/parse_anything_test_files/SPDXRdfExample.rdf.json

Large diffs are not rendered by default.

305 changes: 305 additions & 0 deletions tests/data/parse_anything_test_files/SPDXRdfExample.rdf.spdx

Large diffs are not rendered by default.

305 changes: 305 additions & 0 deletions tests/data/parse_anything_test_files/SPDXRdfExample.rdf.spdx.yml

Large diffs are not rendered by default.

305 changes: 305 additions & 0 deletions tests/data/parse_anything_test_files/SPDXRdfExample.rdf.tag

Large diffs are not rendered by default.

305 changes: 305 additions & 0 deletions tests/data/parse_anything_test_files/SPDXRdfExample.rdf.xml

Large diffs are not rendered by default.

305 changes: 305 additions & 0 deletions tests/data/parse_anything_test_files/SPDXRdfExample.rdf.yaml

Large diffs are not rendered by default.

58 changes: 58 additions & 0 deletions tests/data/parse_anything_test_files/SPDXSBOMExample.spdx.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# example of an SBOM with several packages and filesAnalyzed=False
# from https://github.com/spdx/spdx-spec/issues/439
SPDXID: "SPDXRef-DOCUMENT"
spdxVersion: "SPDX-2.2"
creationInfo:
created: "2020-07-23T18:30:22Z"
creators:
- "Organization: Example Inc."
- "Person: Thomas Steenbergen"
licenseListVersion: "3.9"
name: "xyz-0.1.0"
dataLicense: "CC0-1.0"
documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz"
documentDescribes:
- "SPDXRef-Package-xyz"
packages:
- SPDXID: "SPDXRef-Package-xyz"
summary: "Awesome product created by Example Inc."
copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved."
downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78"
filesAnalyzed: false
homepage: "https://example.com/products/xyz"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc"
name: "xyz"
versionInfo: "0.1.0"
- SPDXID: "SPDXRef-Package-curl"
description: "A command line tool and library for transferring data with URL syntax, supporting \
HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \
IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features."
copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many
contributors, see the THANKS file."
downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz"
filesAnalyzed: false
homepage: "https://curl.haxx.se/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "curl"
name: "curl"
packageFileName: "./libs/curl"
versionInfo: "7.70.0"
- SPDXID: "SPDXRef-Package-openssl"
description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone."
copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved."
downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72"
filesAnalyzed: false
homepage: "https://www.openssl.org/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0"
packageFileName: "./libs/openssl"
name: "openssl"
versionInfo: "1.1.1g"
relationships:
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-curl"
relationshipType: "CONTAINS"
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-openssl"
relationshipType: "CONTAINS"
58 changes: 58 additions & 0 deletions tests/data/parse_anything_test_files/SPDXSBOMExample.spdx.yml.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# example of an SBOM with several packages and filesAnalyzed=False
# from https://github.com/spdx/spdx-spec/issues/439
SPDXID: "SPDXRef-DOCUMENT"
spdxVersion: "SPDX-2.2"
creationInfo:
created: "2020-07-23T18:30:22Z"
creators:
- "Organization: Example Inc."
- "Person: Thomas Steenbergen"
licenseListVersion: "3.9"
name: "xyz-0.1.0"
dataLicense: "CC0-1.0"
documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz"
documentDescribes:
- "SPDXRef-Package-xyz"
packages:
- SPDXID: "SPDXRef-Package-xyz"
summary: "Awesome product created by Example Inc."
copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved."
downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78"
filesAnalyzed: false
homepage: "https://example.com/products/xyz"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc"
name: "xyz"
versionInfo: "0.1.0"
- SPDXID: "SPDXRef-Package-curl"
description: "A command line tool and library for transferring data with URL syntax, supporting \
HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \
IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features."
copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many
contributors, see the THANKS file."
downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz"
filesAnalyzed: false
homepage: "https://curl.haxx.se/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "curl"
name: "curl"
packageFileName: "./libs/curl"
versionInfo: "7.70.0"
- SPDXID: "SPDXRef-Package-openssl"
description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone."
copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved."
downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72"
filesAnalyzed: false
homepage: "https://www.openssl.org/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0"
packageFileName: "./libs/openssl"
name: "openssl"
versionInfo: "1.1.1g"
relationships:
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-curl"
relationshipType: "CONTAINS"
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-openssl"
relationshipType: "CONTAINS"
58 changes: 58 additions & 0 deletions tests/data/parse_anything_test_files/SPDXSBOMExample.spdx.yml.rdf
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# example of an SBOM with several packages and filesAnalyzed=False
# from https://github.com/spdx/spdx-spec/issues/439
SPDXID: "SPDXRef-DOCUMENT"
spdxVersion: "SPDX-2.2"
creationInfo:
created: "2020-07-23T18:30:22Z"
creators:
- "Organization: Example Inc."
- "Person: Thomas Steenbergen"
licenseListVersion: "3.9"
name: "xyz-0.1.0"
dataLicense: "CC0-1.0"
documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz"
documentDescribes:
- "SPDXRef-Package-xyz"
packages:
- SPDXID: "SPDXRef-Package-xyz"
summary: "Awesome product created by Example Inc."
copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved."
downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78"
filesAnalyzed: false
homepage: "https://example.com/products/xyz"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc"
name: "xyz"
versionInfo: "0.1.0"
- SPDXID: "SPDXRef-Package-curl"
description: "A command line tool and library for transferring data with URL syntax, supporting \
HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \
IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features."
copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many
contributors, see the THANKS file."
downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz"
filesAnalyzed: false
homepage: "https://curl.haxx.se/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "curl"
name: "curl"
packageFileName: "./libs/curl"
versionInfo: "7.70.0"
- SPDXID: "SPDXRef-Package-openssl"
description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone."
copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved."
downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72"
filesAnalyzed: false
homepage: "https://www.openssl.org/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0"
packageFileName: "./libs/openssl"
name: "openssl"
versionInfo: "1.1.1g"
relationships:
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-curl"
relationshipType: "CONTAINS"
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-openssl"
relationshipType: "CONTAINS"
58 changes: 58 additions & 0 deletions tests/data/parse_anything_test_files/SPDXSBOMExample.spdx.yml.spdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# example of an SBOM with several packages and filesAnalyzed=False
# from https://github.com/spdx/spdx-spec/issues/439
SPDXID: "SPDXRef-DOCUMENT"
spdxVersion: "SPDX-2.2"
creationInfo:
created: "2020-07-23T18:30:22Z"
creators:
- "Organization: Example Inc."
- "Person: Thomas Steenbergen"
licenseListVersion: "3.9"
name: "xyz-0.1.0"
dataLicense: "CC0-1.0"
documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz"
documentDescribes:
- "SPDXRef-Package-xyz"
packages:
- SPDXID: "SPDXRef-Package-xyz"
summary: "Awesome product created by Example Inc."
copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved."
downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78"
filesAnalyzed: false
homepage: "https://example.com/products/xyz"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc"
name: "xyz"
versionInfo: "0.1.0"
- SPDXID: "SPDXRef-Package-curl"
description: "A command line tool and library for transferring data with URL syntax, supporting \
HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \
IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features."
copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many
contributors, see the THANKS file."
downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz"
filesAnalyzed: false
homepage: "https://curl.haxx.se/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "curl"
name: "curl"
packageFileName: "./libs/curl"
versionInfo: "7.70.0"
- SPDXID: "SPDXRef-Package-openssl"
description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone."
copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved."
downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72"
filesAnalyzed: false
homepage: "https://www.openssl.org/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0"
packageFileName: "./libs/openssl"
name: "openssl"
versionInfo: "1.1.1g"
relationships:
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-curl"
relationshipType: "CONTAINS"
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-openssl"
relationshipType: "CONTAINS"
58 changes: 58 additions & 0 deletions tests/data/parse_anything_test_files/SPDXSBOMExample.spdx.yml.tag
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# example of an SBOM with several packages and filesAnalyzed=False
# from https://github.com/spdx/spdx-spec/issues/439
SPDXID: "SPDXRef-DOCUMENT"
spdxVersion: "SPDX-2.2"
creationInfo:
created: "2020-07-23T18:30:22Z"
creators:
- "Organization: Example Inc."
- "Person: Thomas Steenbergen"
licenseListVersion: "3.9"
name: "xyz-0.1.0"
dataLicense: "CC0-1.0"
documentNamespace: "http://spdx.org/spdxdocs/spdx-document-xyz"
documentDescribes:
- "SPDXRef-Package-xyz"
packages:
- SPDXID: "SPDXRef-Package-xyz"
summary: "Awesome product created by Example Inc."
copyrightText: "copyright 2004-2020 Example Inc. All Rights Reserved."
downloadLocation: "git+ssh://gitlab.example.com:3389/products/xyz.git@b2c358080011af6a366d2512a25a379fbe7b1f78"
filesAnalyzed: false
homepage: "https://example.com/products/xyz"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0 AND curl AND LicenseRef-Proprietary-ExampleInc"
name: "xyz"
versionInfo: "0.1.0"
- SPDXID: "SPDXRef-Package-curl"
description: "A command line tool and library for transferring data with URL syntax, supporting \
HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, MQTT, FILE, \
IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features."
copyrightText: "Copyright (c) 1996 - 2020, Daniel Stenberg, <daniel@haxx.se>, and many
contributors, see the THANKS file."
downloadLocation: "https://github.com/curl/curl/releases/download/curl-7_70_0/curl-7.70.0.tar.gz"
filesAnalyzed: false
homepage: "https://curl.haxx.se/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "curl"
name: "curl"
packageFileName: "./libs/curl"
versionInfo: "7.70.0"
- SPDXID: "SPDXRef-Package-openssl"
description: "OpenSSL is a robust, commercial-grade, full-featured Open Source Toolkit for the Transport Layer Security (TLS) protocol formerly known as the Secure Sockets Layer (SSL) protocol. The protocol implementation is based on a full-strength general purpose cryptographic library, which can also be used stand-alone."
copyrightText: "copyright 2004-2020 The OpenSSL Project Authors. All Rights Reserved."
downloadLocation: "git+ssh://github.com/openssl/openssl.git@e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72"
filesAnalyzed: false
homepage: "https://www.openssl.org/"
licenseConcluded: "NOASSERTION"
licenseDeclared: "Apache-2.0"
packageFileName: "./libs/openssl"
name: "openssl"
versionInfo: "1.1.1g"
relationships:
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-curl"
relationshipType: "CONTAINS"
- spdxElementId: "SPDXRef-Package-xyz"
relatedSpdxElement: "SPDXRef-Package-openssl"
relationshipType: "CONTAINS"
Loading