Skip to content

Fallback to full run if protocol deps are missing/invalid instead of crashing/failing #5004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ilevkivskyi merged 2 commits into from
May 4, 2018
Merged

Fallback to full run if protocol deps are missing/invalid instead of crashing/failing #5004

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
faf9acd
Fallback to full run if protocol deps are missing or invalid
May 4, 2018
d5e11a5
Call dispatch recursively instead of manually reloading graph
May 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions mypy/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1088,7 +1088,7 @@ def get_protocol_deps_cache_name(manager: BuildManager) -> Tuple[str, str]:
contains hashes of all source files at the time the protocol dependencies
were written, and data file contains the protocol dependencies.
"""
name = os.path.join(_cache_dir_prefix(manager), 'proto_deps')
name = os.path.join(_cache_dir_prefix(manager), '@proto_deps')
return name + '.meta.json', name + '.data.json'


Expand Down Expand Up @@ -2342,11 +2342,14 @@ def dispatch(sources: List[BuildSource], manager: BuildManager) -> Graph:
# after we loaded cache for whole graph. The `read_protocol_cache` will also validate
# the protocol cache against the loaded individual cache files.
TypeState.proto_deps = read_protocol_cache(manager, graph)
if TypeState.proto_deps is None and manager.stats.get('fresh_trees', 0) > 0:
if TypeState.proto_deps is None and manager.stats.get('fresh_metas', 0) > 0:

@JukkaL JukkaL May 4, 2018

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to have a test case for this, since this is something that could easily get stale. Maybe create an issue about adding a test (maybe also for other related cache corruption issues)?

@ilevkivskyi ilevkivskyi May 4, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have #5002, do we need a separate issue? I can just add a comment that we need to also check performance/behavior with corrupted caches.

# There were some cache files read, but no protocol dependencies loaded.
manager.errors.set_file(_cache_dir_prefix(manager), None)
manager.errors.report(0, 0, "Error reading protocol dependencies cache -- aborting",
blocker=True)
manager.log("Error reading protocol dependencies cache -- aborting cache load")
manager.cache_enabled = False
TypeState.proto_deps = {}
manager.log("Falling back to full run -- reloading graph...")
return dispatch(sources, manager)

else:
process_graph(graph, manager)
if manager.options.cache_fine_grained or manager.options.fine_grained_incremental:
Expand Down