importlib.metadata.metadata("package") fails with email.errors.HeaderParseError when README contains valid markdown link

Bug report

Bug description:

Here is a minimal example project (tested on Python 3.10):

pyproject.toml

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test"
readme = "README.md"
version = "0.1.0"

README.md

# Test

[MIT](<https://choosealicense.com/licenses/mit/>)

test package:

__init__.py (empty)
bug.py: import importlib.metadata; print(importlib.metadata.metadata("test"))

To reproduce:
pip install -e .
python test/bug.py

Traceback:

Traceback (most recent call last):
  File "bug.py", line 1, in <module>
    import importlib.metadata; print(importlib.metadata.metadata("test"))
  File "lib/python3.10/email/message.py", line 135, in __str__
    return self.as_string()
  File "lib/python3.10/email/message.py", line 158, in as_string
    g.flatten(self, unixfrom=unixfrom)
  File "/home/mint/miniforge3/lib/python3.10/email/generator.py", line 116, in flatten
    self._write(msg)
  File "lib/python3.10/email/generator.py", line 199, in _write
    self._write_headers(msg)
  File "lib/python3.10/email/generator.py", line 226, in _write_headers
    self.write(self.policy.fold(h, v))
  File "lib/python3.10/email/_policybase.py", line 326, in fold
    return self._fold(name, value, sanitize=True)
  File "lib/python3.10/email/_policybase.py", line 369, in _fold
    parts.append(h.encode(linesep=self.linesep, maxlinelen=maxlinelen))
  File "lib/python3.10/email/header.py", line 389, in encode
    raise HeaderParseError("header value appears to contain "
email.errors.HeaderParseError: header value appears to contain an embedded header: '# Test\n[MIT](<https://choosealicense.com/licenses/mit/>)'

It appears that [MIT] is seen as a header, as it would be in a TOML file. That's my best guess really. Any help is appreciated!

CPython versions tested on:

3.10

Operating systems tested on:

Linux

Today I encountered this error as well on a file with RST in the description:

 ~ @ pip-run build -- -c "import build.util; print(build.util.project_wheel_metadata('~/code/jaraco/jaraco.packaging'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import build.util; print(build.util.project_wheel_metadata('~/code/jaraco/jaraco.packaging'))
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-u6naqgo3/build/util.py", line 43, in project_wheel_metadata
    builder = ProjectBuilder.from_isolated_env(
        env,
        source_dir,
        runner=runner,
    )
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-u6naqgo3/build/_builder.py", line 179, in from_isolated_env
    return cls(
        source_dir=source_dir,
        python_executable=env.python_executable,
        runner=_wrap_subprocess_runner(runner, env),
    )
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-u6naqgo3/build/_builder.py", line 154, in __init__
    _validate_source_directory(source_dir)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-u6naqgo3/build/_builder.py", line 52, in _validate_source_directory
    raise BuildException(msg)
build._exceptions.BuildException: Source ~/code/jaraco/jaraco.packaging is not a directory
 ~ [1] @ pip-run build -- -c "import build.util; print(build.util.project_wheel_metadata('code/jaraco/jaraco.packaging'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import build.util; print(build.util.project_wheel_metadata('code/jaraco/jaraco.packaging'))
                       ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/email/message.py", line 165, in __str__
    return self.as_string()
           ~~~~~~~~~~~~~~^^
  File "/opt/python/lib/python3.13/email/message.py", line 188, in as_string
    g.flatten(self, unixfrom=unixfrom)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/email/generator.py", line 115, in flatten
    self._write(msg)
    ~~~~~~~~~~~^^^^^
  File "/opt/python/lib/python3.13/email/generator.py", line 198, in _write
    self._write_headers(msg)
    ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/opt/python/lib/python3.13/email/generator.py", line 225, in _write_headers
    self.write(self.policy.fold(h, v))
               ~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/python/lib/python3.13/email/_policybase.py", line 326, in fold
    return self._fold(name, value, sanitize=True)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/email/_policybase.py", line 369, in _fold
    parts.append(h.encode(linesep=self.linesep, maxlinelen=maxlinelen))
                 ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/email/header.py", line 385, in encode
    raise HeaderParseError("header value appears to contain "
        "an embedded header: {!r}".format(value))
email.errors.HeaderParseError: header value appears to contain an embedded header: ".. image:: https://img.shields.io/pypi/v/jaraco.packaging.svg\n   :target: https://pypi.org/project/jaraco.packaging\n.. image:: https://img.shields.io/pypi/pyversions/jaraco.packaging.svg\n.. image:: https://github.com/jaraco/jaraco.packaging/actions/workflows/main.yml/badge.svg\n   :target: https://github.com/jaraco/jaraco.packaging/actions?query=workflow%3A%22tests%22\n   :alt: tests\n.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json\n    :target: https://github.com/astral-sh/ruff\n    :alt: Ruff\n.. image:: https://readthedocs.org/projects/jaracopackaging/badge/?version=latest\n   :target: https://jaracopackaging.readthedocs.io/en/latest/?badge=latest\n.. image:: https://img.shields.io/badge/skeleton-2024-informational\n   :target: https://blog.jaraco.com/skeleton\nTools for packaging.\nsphinx\n======\nThis package provides a Sphinx extension that will inject into the config\nthe following values from the project's package metadata (as presented by\ndistutils):\n - project (from name)\n - author\n - copyright (same as author)\n - version\n - release (same as version)\n - package_url (from url)\nTo enable, include 'jaraco.packaging' in the requirements and add\n'jaraco.packaging.sphinx' to the list of extensions in a Sphinx config\nfile::\n    extensions=['jaraco.packaging.sphinx']\nThe extension by default builds the project in an isolated environment in\norder to extract the metadata. For offline builds, set\n``BUILD_ENVIRONMENT=current`` and ensure the build dependencies are\nmet in the current environment.\nDeprecated: To build the documentation offline,\nprovide an already built wheel by setting the environment variable\n``JARACO_PACKAGING_SPHINX_WHEEL`` to the path of an existing wheel.\nmake-tree\n=========\nA utility for taking output from ``pipdeptree --json`` and producing a tree\nrooted at a given package.\nUsage::\n    pipdeptree --json | python -m jaraco.packaging.make-tree mypkg\nmetadata\n=========\nA wrapper around ``build.util.project_wheel_metadata`` to enable dowstream\npackagers to indicate that they need an isolated build. Set the environment\nvariable ``BUILD_ENVIRONMENT=current`` to bypass build isolation and use the\ncurrent isolation for loading metadata from a project."

I believe this is a bug in the email module incorrectly inferring an error in the (multi-line) description.

~~A workaround is to not print the metadata, but instead print each of the keys:~~

md = importlib.metadata.metadata('test')
for key in md:
    print(key, md[key], sep=": ")

Edit: on further consideration, that approach will fail to emit unique values for headers of the same key. A better workaround is needed.

The error is easily reproduced without involving importlib.metadata:

>>> import email.message
>>> msg = email.message.Message()
>>> msg.add_header('Foo', 'Test\n[bar](https://example.com)\n')
>>> print(msg)
Traceback (most recent call last):
...
email.errors.HeaderParseError: header value appears to contain an embedded header: 'Test\n[bar](https://example.com)'

The regex that's rejecting the header value is:

cpython/Lib/email/header.py

Lines 50 to 52 in 7595e67

    
           # Find a header embedded in a putative header value.  Used to check for 
        
           # header injection attack. 
        
           _embedded_header = re.compile(r'\n[^ \t]+:')

It's detecting the [bar](https as a header key and //example.com) as the value.

Aha. So maybe importlib.metadata is implicated, since it's repairing the headers thus:

cpython/Lib/importlib/metadata/_adapters.py

Lines 63 to 73 in 6f1d448

    
           def _repair_headers(self): 
        
               def redent(value): 
        
                   "Correct for RFC822 indentation" 
        
                   if not value or '\n' not in value: 
        
                       return value 
        
                   return textwrap.dedent(' ' * 8 + value) 
        
               headers = [(key, redent(value)) for key, value in vars(self)['_headers']] 
        
               if self._payload: 
        
                   headers.append(('Description', self.get_payload())) 
        
               return headers

The email module is assuming the headers are internally represented in their messy, escaped format, but importlib.metadata adapts that to store friendly values internally.

Probably that means that importlib.metadata should also override the rendering behavior.

I get this with iminuit, too. There's it's actually in a header - the license.

assuming the headers are internally represented in their messy, escaped format

The looks like it's just adding indent to already-indented headers. I don't think they could be broken that way. But are the headers actually escaped? The email module doesn't handle unicode (at least for the body), and the specification says it's RFC822 + unicode. And headers aren't allowed to have a new newline in the writers. So most writers customize the writing, and I don't think they escape anything.

Just for context: In emails, a header values cannot contain \n. Long headers can be folded into multiple lines with a line break and the next line starting with white-space, but that's only for transport. Semantically, a folded header value is still just a single line of text. Folding is reversed by removing the white-space and the line-break and stitching the value back together with a single white-space. The line-breaks are lost in the process. So, using the email module for key/value pairs with multi-line values may be tricky. The current approach sounds like a very fragile way to handle this type of data.

I would love if Python package metadata was JSON instead of not-really-email format. :) There's a way to express metadata (starting with 2.1 or 2.2, forget which) as JSON, but no standard filename, etc. for it.

The original "long-description" section, before it was moved to the payload, used | like so:

To support empty lines and lines with indentation with respect to the RFC 822 format, any > CRLF character has to be suffixed by 7 spaces followed by a pipe (“|”) char. As a result, > the Description field is encoded into a folded field that can be interpreted by RFC822 parser.

Example:
Description: This project provides powerful math functions
     |For example, you can use `sum()` to sum numbers:
     |
     |Example::
     |
     |    >>> sum(1, 2)
     |    3
     |
This encoding implies that any occurrences of a CRLF followed by 7 spaces and a pipe char > have to be replaced by a single CRLF when the field is unfolded using a RFC822 reader.

Alternatively, the distribution’s description may instead be provided in the message body > (i.e., after a completely blank line following the headers, with no indentation or other special formatting necessary).

(TBH, payloads aren't unicode, so this requires more hacks.)

But this was never done for the license field, such as in this example:

License: This software may only be obtained by sending the
      author a postcard, and then the user promises not
      to redistribute it.

(See https://packaging.python.org/en/latest/specifications/core-metadata/#license)

When I've tried to validate single line for the one-line summary, I had to go back because people were loading paragraphs into the summary and were mad that it broke. 🤷

The issue here, though, is that there seems to be some escaping needed for things like markdown and restructure text code in a field (License for example), and I'm trying to find out if there's some escaping we can do to make the parser happy.

fix: use old license text (#1050)

Currently, I can't reproduce the error. The same version of iminuit I was running is not reporting an error even when I try it on a range of Pythons.

# /// script
# dependencies = ["nox"]
# ///

import nox
import nox.__main__


@nox.session(python=["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"], venv_backend="uv")
def check(session):
    session.install("iminuit")
    session.run("python", "-c", "import importlib.metadata; importlib.metadata.metadata('iminuit')['License']")

if __name__ == "__main__":
    nox.__main__.main()

In importlib_metadata 8.6.1, it now honors blank lines as well. Please try it out and confirm it addresses your concerns; it should get merged into Python 3.14.

Bump importlib-metadata from 8.5.0 to 8.6.1 (#3177)

pypi-importlib_metadata: Autospec creation for update from version 8.…

Bump the dependencies group in /.config with 12 updates (#4379)

python3-importlib-metadata: upgrade 8.5.0 -> 8.6.1

deps: Update importlib-metadata requirement from <=8.5.0,>=3.6 to >=3…

⬆️ Bump the pip group with 32 updates (#892)

This change didn't land in Python 3.14, but is landing in 3.15 as #137885

Although this change is a bug, it also affects the behavior, so maybe shouldn't be backported. My inclination is not to backport the change, but if someone wishes to request / advocate for a backport, I'd consider it.

build(deps): bump importlib-metadata from 8.5.0 to 8.7.0 in /tools/pu…

Update importlib-metadata requirement from <7.0.0,>=4.8.0 to >=4.8.0,…

deps: update importlib-metadata requirement from <=8.6.1,>=7.0.0 to >…

deps: update importlib-metadata requirement from <=8.7.0,>=7.0.0 to >…

Navigation Menu

Repository navigation

importlib.metadata.metadata("package") fails with email.errors.HeaderParseError when README contains valid markdown link #119650

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

5 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# Find a header embedded in a putative header value. Used to check for
	# header injection attack.
	_embedded_header = re.compile(r'\n[^ \t]+:')

	def _repair_headers(self):
	def redent(value):
	"Correct for RFC822 indentation"
	if not value or '\n' not in value:
	return value
	return textwrap.dedent(' ' * 8 + value)

	headers = [(key, redent(value)) for key, value in vars(self)['_headers']]
	if self._payload:
	headers.append(('Description', self.get_payload()))
	return headers

importlib.metadata.metadata("package") fails with email.errors.HeaderParseError when README contains valid markdown link #119650

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

jaraco commented on Jun 21, 2024

jaraco commented on Jun 21, 2024

jaraco commented on Jun 21, 2024

jaraco commented on Jun 21, 2024

henryiii commented on Oct 9, 2024

defnull commented on Oct 9, 2024

henryiii commented on Oct 9, 2024

henryiii commented on Oct 11, 2024

5 remaining items

jaraco commented on Jan 20, 2025

jaraco commented on Aug 17, 2025

jaraco commented on Aug 18, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions