Repository navigation
Deprecate the unused tokenize._compile function #143009
Description
Activity
@ZeroIntensity
stdlibtopic-parser
CC-ing @pablogsal @Yhg1s @lysnikolaouTalked with @ZeroIntensity, the general guideline is to avoid breaking things, so I'll rephrase this issue to instead propose deprecating first.
- changed the title
[-]Remove the unused `tokenize._compile` function[/-][+]Deprecate the unused `tokenize._compile` function[/+]on Dec 20, 2025 I am not against it but I think is weird to deprecate a private function and the gain for maintainance is very little as we don't have any good reason for doing it right now (security, bugs...etc)
the gain for maintainance is very little as we don't have any good reason for doing it right now (security, bugs...etc)
I see where you're coming from. Do you think it's better to wait for a good reason to deprecate or remove, instead of doing it right now, when there is no immediate benefit?
the gain for maintainance is very little as we don't have any good reason for doing it right now (security, bugs...etc)
I see where you're coming from. Do you think it's better to wait for a good reason to deprecate or remove, instead of doing it right now, when there is no immediate benefit?
Pretty much. In general these kind of changes have more chances to annoy some users then to really improve the maintainance of the code. It's a bit unfortunate that people rely on private functions but at this stage there is no much obvious benefit and I just see risk
Reacted by Bartosz SławeckiI can see the problem. I'll close this -- I also think it's not worth the risk. Thanks @pablogsal!
Feature or enhancement
Proposal:
Remove the unusedEDIT: Deprecate the unusedtokenize._compilefunction.tokenize._compilefunction.It very much looks like the internal
tokenize._compilefunction hasn't been needed since GH-104323 when it was initially removed, but it was mechanically brought back totokenizein GH-104722 to fix GH-104719.From a quick search, it seems that, prior to GH-104323,
tokenize._compilewas only used internally intokenize._tokenize.Most of the fairly fresh code I skimmed through on GitHub operating on
tokenize._compileproperly checks if Python is version <3.10 /tokenize._compileexists.Except
CheetahTemplate3, so removingtokenize._compileimmediately could break them -- we need to deprecate first.In terms of benefits of removing this at some point: it wouldn't necessarily improve the import time of
tokenizeas long asrestill importsfunctoolsto cache template compilation. Therefore, removingtokenize._compilein the future is just a small cleanup that yields no other benefits than smaller, less bloated code.Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response