Regression on initialization and new non-deterministic behavior issue in mimetypes

@davidkhess

Bug report

Mimetypes should NOT include defaults when initialized with file(s)

This is a follow up from https://bugs.python.org/issue4963 that led to this merged PR by @davidkhess #3062

I reported the issue in https://bugs.python.org/issue4963#msg384730 :

The changes introduced by this ticket in 9fc720e#r45794801 are problematic.

I discovered this from having tests failing when testing on Python 3.7 and up

The bug is that calling mimetypes.init(files) will NOT use my files, but instead use both my files and knownfiles.
This was not the case before as knownfiles would be ignored as expected when I provide my own files list.

This is a breaking API change IMHO and introduces a buggy instability : even if I want to ignore knownfiles by providing my list of of files, knownfiles will always be added and this results in erratic and buggy behaviour as the content of "knownfiles" is completely random based on the OS version and else.

The code I am using is here https://github.com/nexB/typecode/blob/ba07c04d23441d3469dc5de911376d408514ebd8/src/typecode/contenttype.py#L308

I think we should reopen to fix (or create a new ticket)

Actually this is problematic on multiples counts:

the behaviour changes and this is a regression

even if that new buggy behaviour was the one to use, it should not give preference to knownfiles over init-provided files, but at least take the provided files first and knownfiles second.

See also this thread 9fc720e#r45794801

@pombredanne
pombredanne on Jan 9, 2021 Contributor

This introduces a buggy unstability starting with Python 3.7: even if I want to ignore knownfiles by providing my list of of files, knownfiles will always be added and this results in erratic and buggy behaviour as the content of "knownfiles" is completely random based on the OS version and else.

@adeadman
adeadman 2 hours ago

@davidkhess this has resulted in an undocumented change to the behaviour of calling mimetypes.init(files=["mimetype_list"]) as now the defaults are included where they weren't before Python 3.7.

I'm not sure how this can be fixed without breaking compatibility or principle of least surprise somewhere - perhaps by adding a new kwarg to init() that ignores knownfiles?

@pombredanne
pombredanne 1 hour ago Contributor

@adeadman re:

I'm not sure how this can be fixed without breaking compatibility or principle of least surprise somewhere - perhaps by adding a new kwarg to init() that ignores knownfiles?

IMHO the current un-documented behaviour is a bug so fixing this would not be an API breakage.
Also the changelog is lying:

Fixed non-deterministic behavior related to mimetypes extension mapping and module reinitialization.

The behaviour used to be mostly deterministic and is now non-deterministic.

And @adeadman also posted in #3062 (comment)

This change has resulted in a difference in behaviour from previous versions of Python in that even when specifying files as a parameter to init(), the known files are loaded, which is not the case in previous versions. This causes problems when an application wishes to use the mimetypes database with an internal-only mapping of mimetypes, to work around inconsistencies with mimetypes in the known files list (which can vary between systems).

Unfortunately this change wasn't documented in the changelog either, nor in the docstring comments, so it can only be found by examining the commit history for the code in cpython.

Would it be appropriate to add another kwarg to init() indicating that knownfiles are to be ignored/excluded from the DB initialization?

Linked PRs

Thanks for pinging me. I'll share what I know and can remember.

The original bug regarded the non-deterministic behavior of the guess_extension method/function. It would return different answers upon different executions of the python interpreter and import of the mimetypes module. I suggest reading through all of https://bugs.python.org/issue4963 as it is very informative as to the discussion and approaches to a solution that were kicked around.

The change you've fingered is related to this comment by me in that issue:

I modified the init() function to check the files argument as r.david.murray suggested. If it is supplied, then the existing database is used and the files are added to it. If it is not supplied, then the module reinitializes from scratch. I'll update the documentation to reflect this if the commit passes muster.

That comment is referring to R. David Murray's comment in that issue: https://bugs.python.org/issue4963#msg108934

It's worth reading that comment carefully and in its entirety to understand how the change to init() tried to walk a very fine line.

I think this current issue is a result of that analysis and my fix related to it missing the use case where files is definitely not meant to be additive - it should be the only thing loaded.

In terms of a workaround, I think the easiest thing that should work is:

    import mimetypes
    mimetypes.knownfiles = []
    mimetypes.init(files=...)

Perhaps a longer-term fix would be to add a boolean argument to init() that controls whether or not knownfiles are used additively.

Just my $0.02.

bpo-4963: Fix for initialization and non-deterministic behavior issue…

@davidkhess

Thanks for your detailed comment @davidkhess and thanks for opening the report @pombredanne.

From R. David Murray's analysis in the original issue:

It's not so clear what the behavior should be when you pass init one or more files. It is possible, even highly probable, that there is code out there that depends on the fact that doing so is additive.

I'm not clear exactly what was meant by "additive", in this context. However, an examination of the init() code before the change would suggest that this was not always the case, as previously a call to init() passing in a list of files used only the files supplied, and used the knownfiles as a default if no files were specified. A caller would have to explicitly pass the known files in like this: mimetypes.init(files=['my_mimetype_file'] + mimetypes.knownfiles).

I believe his analysis was perhaps referring to calling MimeTypes() a second time would not reinitialise the global db, as the inited flag prevents that from happening, but not a call to mimetypes.init() explicitly.

Regardless of the validity of this point of analysis, the fact remains that the current behaviour of mimetypes.init (since Python 3.7) is to load from the knownfiles as well as the files specified, and so Hyrum's Law would suggest that to put the behavior back as it was will inconvenience someone.

Perhaps as a first step we could update the docstring to clearly mark that the behaviour changed between Python 3.6 and 3.7, so people that must maintain supporting for multiple versions can be aware of the difference.

Facing this issue when investigate [CVE-2024-3220] Default mimetype known files writeable on Windows

The workaround to ignore the knownfiles in the email is wrong:

To work-around this issue a user can call mimetypes.init() with an empty
list (“[]”) on Windows platforms to avoid using the default list of known
file locations.

So I create #141393 to update the doc.

pythongh-93417: Improve test_mimetypes

gh-93417: Improve test_mimetypes (GH-150725)

#150733 adds the use_system_defaults to mimetypes.init() and MimeTypes(). It controls whther to load MIME information from knownfiles files and Windows registry.

It is a draft for now, I'll add documentation after merging other documentation PR.

[3.13] gh-93417: Improve test_mimetypes (GH-150725) (GH-153750)

[3.14] gh-93417: Improve test_mimetypes (GH-150725) (GH-153749)

[3.15] gh-93417: Improve test_mimetypes (GH-150725) (GH-153748)

Navigation Menu

Repository navigation

Regression on initialization and new non-deterministic behavior issue in mimetypes #93417

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Regression on initialization and new non-deterministic behavior issue in mimetypes #93417

Description

Linked PRs

Activity

davidkhess commented on Jun 1, 2022

adeadman commented on Jun 1, 2022

bebound commented on Nov 11, 2025

serhiy-storchaka commented on Jun 1, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions