You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consider not consuming all the buffer in one go in the tokenizer module #105069
Seems that some tools were relying on the implementation detail that the readline-like callable that is provided to functions in the tokenize module is called as tokens are emitted and not consumed in one go. Although this was never part of the contract and technically we don't need to change, it would make the implementation more efficient as we don't need to hold the entire input in memory at the same time and we won't break these tools.
Seems that some tools were relying on the implementation detail that the
readline-like callable that is provided to functions in thetokenizemodule is called as tokens are emitted and not consumed in one go. Although this was never part of the contract and technically we don't need to change, it would make the implementation more efficient as we don't need to hold the entire input in memory at the same time and we won't break these tools.Linked PRs