Skip to content

Add undocumented flag for use with trusted file watchers - #22516

Open
jhance wants to merge 2 commits into
masterfrom
efes-cache-changes
Open

jhance wants to merge 2 commits into
masterfrom
efes-cache-changes

Conversation

@jhance

@jhance jhance commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

This PR is targetted at optimizing for a Dropbox internal use case where we have begun using a FUSE VFS to serve our git repository.

For lazy FUSE repos, crawling the filesystem is expensive, but the file notifications from the filesystem are extremely reliable, so this flag allows the caller to tell mypy which files have changed since the cache was in place, allowing mypy to skip most of its work.

I think it is still useful for non-FUSE users, but less required.

The flag is not documented.

A daemon started with --use-fine-grained-cache looks at every source on
its first check: validate_meta stats each file (and hashes it when the
cache's mtime does not match, which a downloaded cache's never does), and
the fswatcher then stats them all again to find what changed since the
cache. For a large repository on a network or FUSE filesystem, that is the
whole cost of a cold start, though a tool that knows what changed - git,
or a filesystem that keeps a journal - could say so in one step.

--changed-since-cache-file names that file: every path that may differ
from the files the cache was generated from, one per line. For any other
source whose cache metadata names the same path, validate_meta accepts
the metadata without a stat, State creation does not ask whether it is a
directory, and the first fine-grained update looks only at the named
files and at sources the cache has nothing for, through the same
update_changed path dmypy recheck --update takes. A missing file is
warned about and ignored, so every file is looked at as before.

Like --quickstart-file, the option is hidden: its user is a wrapper that
knows how the cache was made.
Before its first check, dmypy turns each source path into a module by
crawling up its directories for __init__ files. With the cache trusted for
files not named in --changed-since-cache-file, that is the one look left
at each source's directories, and over a network or FUSE filesystem the
largest part of a cold start.

The cache already says it: each module's metadata has its path and id,
and a module's components are the directories above the file, so its
base directory follows without looking. A file's module is as when the
cache was made unless the file, or an __init__ in a directory above it,
changed since, so for every other file the first check takes its module
from the cache; create_source_list takes these as known_modules, and
crawls for the rest and for any path whose directories do not match its
module.
@github-actions

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant