Repository navigation
Conversation
A daemon started with --use-fine-grained-cache looks at every source on its first check: validate_meta stats each file (and hashes it when the cache's mtime does not match, which a downloaded cache's never does), and the fswatcher then stats them all again to find what changed since the cache. For a large repository on a network or FUSE filesystem, that is the whole cost of a cold start, though a tool that knows what changed - git, or a filesystem that keeps a journal - could say so in one step. --changed-since-cache-file names that file: every path that may differ from the files the cache was generated from, one per line. For any other source whose cache metadata names the same path, validate_meta accepts the metadata without a stat, State creation does not ask whether it is a directory, and the first fine-grained update looks only at the named files and at sources the cache has nothing for, through the same update_changed path dmypy recheck --update takes. A missing file is warned about and ignored, so every file is looked at as before. Like --quickstart-file, the option is hidden: its user is a wrapper that knows how the cache was made.
Before its first check, dmypy turns each source path into a module by crawling up its directories for __init__ files. With the cache trusted for files not named in --changed-since-cache-file, that is the one look left at each source's directories, and over a network or FUSE filesystem the largest part of a cold start. The cache already says it: each module's metadata has its path and id, and a module's components are the directories above the file, so its base directory follows without looking. A file's module is as when the cache was made unless the file, or an __init__ in a directory above it, changed since, so for every other file the first check takes its module from the cache; create_source_list takes these as known_modules, and crawls for the rest and for any path whose directories do not match its module.
Contributor
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is targetted at optimizing for a Dropbox internal use case where we have begun using a FUSE VFS to serve our git repository.
For lazy FUSE repos, crawling the filesystem is expensive, but the file notifications from the filesystem are extremely reliable, so this flag allows the caller to tell mypy which files have changed since the cache was in place, allowing mypy to skip most of its work.
I think it is still useful for non-FUSE users, but less required.
The flag is not documented.