Document how to profile and optimize mypy

We now have some tooling that helps with investigating performance bottlenecks in mypy, and I've had some success using these to improve the performance of mypy. It would be good to document these workflows so that more contributors can help with this work.

Is there also tooling that allows me to see if/where there are specific parts of my stubs that are a performance bottleneck?

Yes, we have something like that, but I haven't used it in a long time. I'll look into it as well.

I started writing the documentation here: https://github.com/python/mypy/wiki/Profiling-and-Optimizing-Mypy

It's still work in progress, but there is already something there.

@ilevkivskyi

Now it has at least the basics covered.

cc @ilevkivskyi @sterliakov who've been looking at performance bottlenecks.

Nice! FWIW I started looking at why colours is so slow. It turns out more than half of all time is spent on type-checking just two expressions. Both involve a long list (few dozens) of np.array() calls nested within dict(zip()). I am currently looking at special-casing np.array() similar to how we special-case list literals and/or adding an expression cache.

@ilevkivskyi

Thanks, that's a nice writeup!

@ilevkivskyi then #19408 is related - also an expression cache for colour:)

I am currently looking at special-casing np.array() similar to how we special-case list literals and/or adding an expression cache.

Anything I can help with on the numpy side of things?

I vaguely remember seeing a repo one time which is a mypy/mypyc benchmark "service" which uses the github runner to compare self-check performance between 2 specific commits, but now I can't find it. Does this ring any bells for any of you?

Perhaps you're thinking of https://github.com/sterliakov/mypy-issues?

Ayo my man! That's the one, thank you!

…

On Wed, Aug 6, 2025 at 6:37 PM Brian Schubert ***@***.***> wrote: *brianschubert* left a comment (python/mypy#19472) <#19472 (comment)> Perhaps you're thinking of https://github.com/sterliakov/mypy-issues? — Reply to this email directly, view it on GitHub <#19472 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQ3HIHVBS74JB4BNVSHSVE33MJ7R5AVCNFSM6AAAAACBXRQB7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNRRHAZTEOJXHE> . You are receiving this because you commented.Message ID: ***@***.***>

@ilevkivskyi

It turns out more than half of all time is spent on type-checking just two expressions. Both involve a long list (few dozens) of np.array() calls nested within dict(zip()).

Do you by perhaps remember which colour expressions @ilevkivskyi ? Because I'd like to see if I can do something about those. It currently takes 9 minutes to run in mypy_primer, which for scipy-stubs is a huge CI performance bottlenece (everything else runs in <2 minutes).

update: I'm guessing it's https://github.com/colour-science/colour/blob/ae990bd75ce8d4238f4d41c7f6a99e5b15e91ce6/colour/characterisation/datasets/colour_checkers/chromaticity_coordinates.py . Optimizing it seems pretty straightforward luckily.

@jorenham

@jorenham Why do you run interpreted mypy in CI? Also there should be no bottleneck anymore, I fixed it like a month ago.

@jorenham Why do you run interpreted mypy in CI?

Huh, am I? I basically use the same config as numpy and ruff do, so I kinda assumed that mypy_primer would compile it by default, tbh...

Also there should be no bottleneck anymore, I fixed it like a month ago.

Hmm, maybe we're talking about a different bottleneck, or perhaps the something changed about the colour codebase in the meantime 🤔. Either way, what I was talking about can be seen in e.g. https://github.com/scipy/scipy-stubs/actions/runs/17163275474/job/48697488986?pr=831, namely that colour takes ~530 seconds, whereas the next slowest project runs in ~106 seconds.

Two days ago I did manage to reduce that by one minute colour-science/colour#1358, but even with that it's still over 4 times slower than the next thing.

Anyway, I'll try and see if I can get mypy to compile then, which I'm guessing is gonna help quite a bit. Thanks for the tip!

Huh, am I? I basically use the same config as numpy and ruff do, so I kinda assumed that mypy_primer would compile it by default, tbh...

I guess you are, on current master compiled mypy should be able to check colour in couple minutes or so (depending on hardware).

Hmm, maybe we're talking about a different bottleneck

I am talking about my original comment, now there is no obvious hot spot in colour code where mypy spends most of the time

Huh, am I? I basically use the same config as numpy and ruff do, so I kinda assumed that mypy_primer would compile it by default, tbh...

I guess you are, on current master compiled mypy should be able to check colour in couple minutes or so (depending on hardware).

Ah I see, that explains. So no need for running mypy_primer with --mypyc-compile-level=3 then.

I see you are using last PyPI version of mypy in CI. That one doesn't have a lot of recent perf improvements (including the fix I am talking about).

Navigation Menu

Repository navigation

Document how to profile and optimize mypy #19472

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Document how to profile and optimize mypy #19472

Description

Activity

jorenham commented on Jul 17, 2025

JukkaL commented on Jul 17, 2025

JukkaL commented on Jul 17, 2025

JukkaL commented on Jul 18, 2025

ilevkivskyi commented on Jul 18, 2025

sterliakov commented on Jul 18, 2025

jorenham commented on Jul 18, 2025

BobTheBuidler commented on Aug 6, 2025

brianschubert commented on Aug 6, 2025

BobTheBuidler commented on Aug 6, 2025

jorenham commented on Aug 24, 2025

ilevkivskyi commented on Aug 25, 2025

jorenham commented on Aug 25, 2025

ilevkivskyi commented on Aug 25, 2025

jorenham commented on Aug 25, 2025

ilevkivskyi commented on Aug 25, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions