Repository navigation
Remove support for JSON serialization format #19697
Description
Now that we have a new binary fixed format for serialization, it would be good to eventually drop the old JSON based serialization code, since maintaining duplicate implementations slows down development, and there is a risk that the implementations will get out of sync.
At work we have some tools that consume the JSON based cache format, and it would be non-trivial to update them to directly use the binary format. As a workaround, we could provide a tool that converts a binary cache into JSON files that resemble the current JSON format (they don't need to be 100% compatible since the format isn't documented anywhere).
To implement this, we can make the binary format "self-describing", i.e. it would have enough redundancy that we can write a simple generic parser that reads arbitrary data serialized using the format and converts it into JSON, and that doesn't need to know too many details of each possible object type that can be serialized (to simplify maintenance).
Here is one possible to way to make the format self-describing:
- Each value/object is serialized as
<type tag><data>, where the format of data depends on the type tag. The type tag is a 8-bit integer. - Simple values are also encoded as a type tag followed by data. These include integers, strings, booleans, floats and
None. - We can have similar generic encoding for lists and dicts, such as
<type tag for list><number of items><arbitrary value>.... - AST nodes would also have distinct type tags, so that it would be possible to mix simple values and AST nodes.
- I would propose that AST nodes and types would be encoded using a format like this:
<type tag><field tag><arbitrary value>...<end tag>. Field tags are also 8-bit integers that map to JSON keys. The end tag is a reserved 8-bit number that is distinct from all type tags.
- I would propose that AST nodes and types would be encoded using a format like this:
Now we can implement a generic parser. It needs to know how to parse all simple values and container values, and it must have a mapping from valid type tags for AST nodes (including type objects) and the field tags to strings.
I think that this format would also make it easy to implement lazy deserialization. We can easily find the end of an arbitrary object in a serialized byte stream, without having to deserialize it. Instead of deserializing a FuncDef, for example, we could just find the and of the serialized representation and put the serialized byte string into a symbol table.
cc @ilevkivskyi
Activity
JukkaL commentedon Aug 20, 2025
The main changes to the current [de]deserialization code would be these:
KIND: Final = 56; 56 is the field tag value). The JSON key name can be derived automatically from the constant name (KIND->"kind").write_int_field(buf, KIND, o.kind)).This would a little less efficient that the current format, but I don't expect the performance to degrade much, since all the new features can be compiled to fast machine operations (no new dynamic allocations etc. that would slow things down). If this enables lazy deserialization, I'd expect overall performance to improve significantly.
ilevkivskyi commentedon Aug 21, 2025
TBH I don't like this. IMO we should simply drop the JSON format at some point, period. More details:
from mypy.nodes import MypyFile; from native_internal import Buffer; tree = MypyFile.read(Buffer(data)). A requirement to inspect mypy cache files while not having mypy available looks very niche.<end tag>will help with lazy deserialization. You can't simply scan the byte string looking for this magic byte (it may appear accidentally in other serialized values). IMO it can be implemented in a simpler (and probably more efficient) way by storing total byte size in front of each symbol:FuncDef,Var, etc)JukkaL commentedon Aug 21, 2025
If there's strong opposition to this, we could implement and maintain the JSON converter internally at Dropbox. However, we have no reliable way of knowing ahead of time whether other mypy users are parsing the JSON files and would have to change their workflows. I remember discussing the things we do with cache files at conferences, and it's possible that others have borrowed the idea or came up with it by themselves. In any case I volunteer to implement the required functionality and I don't think it would add a significant maintenance burden (but clearly it would have some impact, even if small).
I think the field tags are redundant and can be dropped from the proposal. I also think that the remaining bits would be useful when implemeting lazy deserialization. We could also use the "quickly skip some objects during deserialization" functionality to implement a low-tech incremental parser that will skip serialized nodes until reaching line N.
If we'd drop the field tags and only keep the type tags, which could useful for lazy deserialization, there might not be any impact. This at least will be easy to measure once we have a prototype implementation.
I hadn't thought about this. I will experiment with this. If it looks like this can support the use cases we have at work, we can document this new approach in mypy release notes and perhaps add some simple example under
misc/that does something like this. In that case the JSON converter wouldn't be needed.It will be used for tracking the nesting depth of objects. The byte value of the end tag only needs to not overlap with field tags (or type tags if we drop field tags). Serialized simple values such as integers and strings can have arbitrary binary data with no reserved bytes. We just need a way to skip until the end of the deserialized data for each possible data type (i.e. simple types, container types, and object types). All object types are similar, but without the end tag we don't know when the object ends without some extra metadata (which would require additional maintenance and thus wouldn't be ideal). I can explain this is more detail if this is still unclear.
cc @msullivan who mentioned potential impact to debugging issues with incremental mode.
ilevkivskyi commentedon Aug 21, 2025
I think when we will flip the default to use fixed format cache, and then later will emit deprecation warnings if people use
--json-cache, they will likely start complaining. Then we will see how much it is needed.Yeah, I understand this "galloping mode" read. But whether this is really needed or we can just insert total size for few selected classes depends on the details of the proposal. I see you commented in a dedicated issue, we can discuss it there.
Adding just the type tags may be OK, this really depends on the details of how it will be implemented (e.g if adding type tags will be "automated" for primitive types etc).
It is not like I am putting my foot down, but I think we need to weight all options before committing to anything here. Btw yet another possible option may be to have a separate file that will have a list of attribute names for each class in the order they appear in cache. This file would be kept up to date by people who are interested in parsing the cache files without loading (assuming this will be actually needed).
A5rocks commentedon Aug 30, 2025
BTW Trio relies on JSON serialization to quickly check whether an attribute is publicly exported in mypy's view. While we can do this by calling mypy directly, this is significantly slower than populating cache and then pulling from it! I think we can use this workaround though:
But FYI this might be more popular than you expected. (Though I also only did this because I've contributed to mypy and knew the internals)
JukkaL commentedon Oct 14, 2025
We'll add sufficient type tags (i.e start tags) and end tags so that we can find the end of a serialized object with a generic algorithm that doesn't understand all the specifics of serialized objects (see #17924 (comment) for context). It would be good to have this for the next mypy public release (release tracked at #19964), but not absolutely essential.
We can possibly use
MypyFile.read(Buffer(data))as a replacement for the current JSON serialization format (see discussion above). This should be documented at least in the release notes.However, we still have the option of writing a binary cache to JSON converter, which would perhaps only support a subset of the data exposed in the current JSON format, but enough for common use cases. I think the most interesting bit is symbol table data -- we could start by providing an export tool that dumps module and class symbol tables, with kinds of nodes (e.g variable of function), cross_refs and some other relevant information (maybe also static types).
JukkaL commentedon Oct 14, 2025
I started prototyping and it doesn't look hard to write a tool that calls
MypyFile.read(...)on a fixed format cache file and then converts the result to JSON that mostly looks like the current JSON cache format. This looks like a reasonable solution, and we can allow the tool to be maintained on a best effort basis (i.e. not all information added to the symbol might be exported). Once we have this tool, I think we can drop support for the JSON serialization format after a reasonable deprecation period.Add tool to convert binary cache files to JSON (#20071)
JukkaL commentedon Oct 17, 2025
Now we have a tool to convert .data.ff files to JSON (#20071). The tool will still need to be updated to support binary meta cache files, once mypy supports them (meta are files are still JSON currently, but we should switch them to use the binary format as well).