diff --git a/LAB_II.md b/LAB_II.md new file mode 100644 index 0000000000000..58bc68218387f --- /dev/null +++ b/LAB_II.md @@ -0,0 +1,135 @@ +# Lab II — Generic Type Suggestions for `dmypy suggest` + +## Overview + +This project extends mypy's `dmypy suggest` command to recommend **generic function signatures** when relationships between input and output types can be inferred from the function body and multiple call sites. + +For example, given: + +```python +def identity(value): + return value + +identity(42) +identity("hello") +``` + +`dmypy suggest` can now output: + +```text +[T](T) -> T +``` + +This corresponds to the Python 3.12+ annotation `def identity[T](value: T) -> T:`. The command **only suggests** a signature; it does not edit the user's Python source file. + +## Supported relationships + +The current implementation recognizes these common patterns: + +| Function body pattern | Suggested signature | +| --- | --- | +| `return value` | `[T](T) -> T` | +| `return items[0]` (lists) | `[T](list[T]) -> T` | +| `return items[0]` (homogeneous tuples) | `[T](tuple[T, ...]) -> T` | +| `return mapping[key]` | `[K, V](dict[K, V], K) -> V` | +| `return callback(value)` | `[T, R](Callable[[T], R], T) -> R` | + +The implementation also supports a simple instance-method case, such as `Box.echo(self, item)` returning `item`. + +A candidate is only considered when the observed call sites provide suitable, differing type information and its structure matches one of the supported patterns. Unsupported or ambiguous cases fall back to ordinary type suggestions. + +## How it works + +1. **Collect call-site types.** The existing `SuggestionEngine` gathers argument types observed at calls to the target function. The new `get_generic_callsites()` method normalizes these observations for the inference helper. +2. **Inspect the function AST.** Visitors in `mypy/generic_suggestions.py` recognize direct returns, indexed collection access, dictionary lookup, and single-argument callback application. A separate visitor rejects selected patterns in which relevant arguments are rebound. +3. **Build generic candidates.** `infer_generic_candidates()` creates `CallableType` signatures using mypy's `TypeVarType`, `Instance`, and related internal types. The TypeVars generated for suggestions have the dedicated namespace `dmypy_suggest`. +4. **Validate candidates.** `SuggestionEngine` rechecks candidate signatures with `try_type()`. It rejects candidates that introduce new reported errors compared with the original diagnostics, and it compares the remaining candidates with normal suggestions using error counts and type scores. +5. **Format the result.** Accepted type variables appear in the suggested signature, for example `[K, V](dict[K, V], K) -> V`. The JSON suggestion format may also include a `type_params` field. + +The error comparison is intentionally conservative: changes in the text or location of an error may cause a candidate to be rejected. This is a safety mechanism, not a formal proof of correctness for arbitrary programs. + +## Additional extension: configurable call-site threshold + +The `--generic-min-calls=N` option sets the minimum number of usable observed calls required before generic inference is attempted. The default is **2**. + +For example: + +```bash +dmypy suggest --generic-min-calls=3 demo_generic.identity +``` + +If the function has fewer than three suitable call sites, `dmypy suggest` falls back to its ordinary, non-generic suggestion. The current implementation disables generic inference when `N < 2` rather than reporting a CLI validation error. + +## Try it locally + +Run these commands from the repository root with the project installed in your Python environment: + +```bash +dmypy start +dmypy check demo_generic.py +dmypy suggest demo_generic.identity +``` + +The committed `demo_generic.py` contains three calls with different argument types, so the expected generic output is: + +```text +[T](T) -> T +``` + +You can also demonstrate the configuration option without changing the file: + +```bash +dmypy suggest --generic-min-calls=4 demo_generic.identity +# Falls back to a non-generic signature: (object) -> object + +dmypy suggest --generic-min-calls=3 demo_generic.identity +# [T](T) -> T +``` + +To inspect additional supported patterns, see the integration examples in `mypy/test/testgeneric_suggestions.py` and `test-data/unit/fine-grained-suggest.test`. + +## Tests and checks + +The following commands were run successfully during development: + +```bash +# Generic inference unit tests and end-to-end integration tests +python -m pytest -n 0 -q mypy/test/testgeneric_suggestions.py + +# Existing and new dmypy suggest scenarios +python -m pytest -n 0 -q mypy/test/testfinegrained.py -k "Suggest" + +# General type-checker regression suite +python -m pytest -q mypy/test/testcheck.py + +# Project type self-check +python runtests.py self + +# Broader fast regression suite +python runtests.py pytest-fast +``` + +Ruff linting and formatting checks were also used for the modified Python files. The tests include positive examples (identity, collections, callbacks, methods), negative examples (insufficient or uniform calls, reassigned arguments, constrained function bodies), and checks for new diagnostic messages. + +## Key changed files + +| File | Purpose | +| --- | --- | +| `mypy/generic_suggestions.py` | AST analysis and generic candidate construction | +| `mypy/suggestions.py` | Call-site normalization, candidate validation, and formatting | +| `mypy/typeanal.py` | Handling of suggestion-generated type variables during reanalysis | +| `mypy/dmypy/client.py` | CLI option `--generic-min-calls` | +| `mypy/test/testgeneric_suggestions.py` | Unit and end-to-end tests | +| `mypy/test/testfinegrained.py` and `test-data/unit/fine-grained-suggest.test` | Data-driven `dmypy suggest` tests | +| `demo_generic.py` | Minimal manual demonstration | + +## Limitations and design decisions + +- The feature intentionally handles **recognizable patterns**, not arbitrary generic relationships or full control-flow/type inference. +- Generic candidates need at least two suitable calls by default. They may not be produced when call-site information is incomplete or contains `Any`. +- The normalization currently targets simple one-actual-argument-per-parameter calls. Complex `*args` / `**kwargs` forms are not supported by this generic path. +- Indexed access is only recognized in limited direct-return forms. Tuple inference uses homogeneous element types; callback inference handles a single argument. +- Reassignment detection and diagnostic comparison reduce false suggestions, but they cannot guarantee that every possible runtime behavior is represented. +- The feature reuses mypy's existing AST, type representation, daemon, and suggestion mechanisms, and does not require new runtime dependencies. + +These restrictions are deliberate: when there is insufficient evidence, the tool prefers the existing non-generic suggestion to an unsupported guess. diff --git a/demo_generic.py b/demo_generic.py new file mode 100644 index 0000000000000..a2ddc1eeb516b --- /dev/null +++ b/demo_generic.py @@ -0,0 +1,7 @@ +def identity(value): + return value + + +a: int = identity(42) +b: str = identity("hello") +c: list[int] = identity([1, 2, 3]) diff --git a/mypy/dmypy/client.py b/mypy/dmypy/client.py index 722aea4243575..d189b3ad3000d 100644 --- a/mypy/dmypy/client.py +++ b/mypy/dmypy/client.py @@ -178,6 +178,11 @@ def __init__(self, prog: str, **kwargs: Any) -> None: type=int, help="Set the maximum number of types to try for a function (default 64)", ) +p.add_argument( + "--generic-min-calls", + type=int, + help="Set the minimum number of callsites required for generic inference (default 2)", +) inspect_parser = p = subparsers.add_parser( "inspect", help="Locate and statically inspect expression(s)" @@ -520,6 +525,7 @@ def do_suggest(args: argparse.Namespace) -> None: flex_any=args.flex_any, use_fixme=args.use_fixme, max_guesses=args.max_guesses, + generic_min_calls=args.generic_min_calls, ) check_output(response, verbose=False, junit_xml=None, perf_stats_file=None) @@ -594,7 +600,7 @@ def show_stats(response: Mapping[str, object]) -> None: # Special case text output to display just 40 characters of text value = repr(value)[1:-1] if len(value) > 50: - value = f"{value[:40]} ... {len(value)-40} more characters" + value = f"{value[:40]} ... {len(value) - 40} more characters" print("%-24s: %s" % (key, value)) continue print("%-24s: %10s" % (key, "%.3f" % value if isinstance(value, float) else value)) diff --git a/mypy/generic_suggestions.py b/mypy/generic_suggestions.py new file mode 100644 index 0000000000000..176b8f5b54a82 --- /dev/null +++ b/mypy/generic_suggestions.py @@ -0,0 +1,645 @@ +"""Helpers for detecting generic type relationships.""" + +from collections.abc import Callable, Sequence + +from mypy.erasetype import remove_instance_last_known_values +from mypy.nodes import ( + ARG_OPT, + ARG_POS, + AssignmentExpr, + AssignmentStmt, + CallExpr, + DelStmt, + Expression, + ForStmt, + FuncDef, + IndexExpr, + IntExpr, + ListExpr, + NameExpr, + OperatorAssignmentStmt, + ReturnStmt, + StarExpr, + TupleExpr, + Var, + WithStmt, +) +from mypy.traverser import TraverserVisitor +from mypy.types import ( + AnyType, + CallableType, + Instance, + TupleType, + Type, + TypeOfAny, + TypeVarId, + TypeVarType, + get_proper_type, +) + + +class ReturnedArgumentFinder(TraverserVisitor): + """Find arguments returned directly by a function.""" + + def __init__(self, func: FuncDef) -> None: + self.arguments = [arg.variable for arg in func.arguments] + self.returned: list[int | None] = [] + + def visit_return_stmt(self, stmt: ReturnStmt) -> None: + expr = stmt.expr + + if isinstance(expr, NameExpr): + for i, arg in enumerate(self.arguments): + if expr.node is arg: + self.returned.append(i) + return + + self.returned.append(None) + + def visit_func_def(self, node: FuncDef) -> None: + # Ignore returns inside nested functions. + pass + + +def find_direct_returned_argument(func: FuncDef) -> int | None: + """Return the index of an argument returned directly.""" + + finder = ReturnedArgumentFinder(func) + func.body.accept(finder) + + if not finder.returned: + return None + + first = finder.returned[0] + + if first is None: + return None + + if any(index != first for index in finder.returned): + return None + + if is_argument_reassigned(func, first): + return None + + return first + + +def make_direct_return_generic( + base: CallableType, returned_arg: int, object_type: Instance +) -> CallableType | None: + """Build a generic candidate linking an argument to the return type.""" + + if base.variables: + return None + + if not 0 <= returned_arg < len(base.arg_types): + return None + + if not isinstance(get_proper_type(base.arg_types[returned_arg]), AnyType): + return None + + if not isinstance(get_proper_type(base.ret_type), AnyType): + return None + + type_var = TypeVarType( + "T", + "T", + TypeVarId(-1, namespace="dmypy_suggest"), + [], + object_type, + AnyType(TypeOfAny.from_omitted_generics), + ) + + new_arg_types = base.arg_types.copy() + new_arg_types[returned_arg] = type_var + + return base.copy_modified(arg_types=new_arg_types, ret_type=type_var, variables=[type_var]) + + +class IndexedReturnFinder(ReturnedArgumentFinder): + """Find arguments whose elements are returned directly.""" + + def visit_return_stmt(self, stmt: ReturnStmt) -> None: + expr = stmt.expr + + if ( + isinstance(expr, IndexExpr) + and isinstance(expr.base, NameExpr) + and isinstance(expr.index, IntExpr) + ): + for i, arg in enumerate(self.arguments): + if expr.base.node is arg: + self.returned.append(i) + return + + self.returned.append(None) + + +def find_indexed_returned_argument(func: FuncDef) -> int | None: + """Find an argument returned through integer indexing.""" + + finder = IndexedReturnFinder(func) + func.body.accept(finder) + + if not finder.returned: + return None + + first = finder.returned[0] + + if first is None: + return None + + if any(index != first for index in finder.returned): + return None + + if is_argument_reassigned(func, first): + return None + + return first + + +def make_collection_element_generic( + base: CallableType, indexed_arg: int, collection_type: Instance, object_type: Instance +) -> CallableType | None: + """Build a generic candidate for list[T] or tuple[T, ...].""" + + if base.variables: + return None + + if not 0 <= indexed_arg < len(base.arg_types): + return None + + if collection_type.type.fullname not in ("builtins.list", "builtins.tuple"): + return None + + if not isinstance(get_proper_type(base.arg_types[indexed_arg]), AnyType): + return None + + if not isinstance(get_proper_type(base.ret_type), AnyType): + return None + + type_var = TypeVarType( + "T", + "T", + TypeVarId(-1, namespace="dmypy_suggest"), + [], + object_type, + AnyType(TypeOfAny.from_omitted_generics), + ) + + new_arg_types = base.arg_types.copy() + new_arg_types[indexed_arg] = collection_type.copy_modified(args=[type_var]) + + return base.copy_modified(arg_types=new_arg_types, ret_type=type_var, variables=[type_var]) + + +def make_list_element_generic( + base: CallableType, indexed_arg: int, list_type: Instance, object_type: Instance +) -> CallableType | None: + if list_type.type.fullname != "builtins.list": + return None + + return make_collection_element_generic(base, indexed_arg, list_type, object_type) + + +def make_tuple_element_generic( + base: CallableType, indexed_arg: int, tuple_type: Instance, object_type: Instance +) -> CallableType | None: + if tuple_type.type.fullname != "builtins.tuple": + return None + + return make_collection_element_generic(base, indexed_arg, tuple_type, object_type) + + +class DictionaryLookupFinder(TraverserVisitor): + """Find returns of the form mapping[key].""" + + def __init__(self, func: FuncDef) -> None: + self.arguments = [arg.variable for arg in func.arguments] + self.lookups: list[tuple[int, int] | None] = [] + + def visit_return_stmt(self, stmt: ReturnStmt) -> None: + expr = stmt.expr + + if ( + not isinstance(expr, IndexExpr) + or not isinstance(expr.base, NameExpr) + or not isinstance(expr.index, NameExpr) + ): + self.lookups.append(None) + return + + mapping_index = next( + (i for i, arg in enumerate(self.arguments) if expr.base.node is arg), None + ) + + key_index = next( + (i for i, arg in enumerate(self.arguments) if expr.index.node is arg), None + ) + + if mapping_index is None or key_index is None or mapping_index == key_index: + self.lookups.append(None) + return + + self.lookups.append((mapping_index, key_index)) + + def visit_func_def(self, node: FuncDef) -> None: + # Ignore nested functions. + pass + + +def find_dictionary_lookup(func: FuncDef) -> tuple[int, int] | None: + """Find arguments used in direct mapping[key] returns.""" + + finder = DictionaryLookupFinder(func) + func.body.accept(finder) + + if not finder.lookups: + return None + + first = finder.lookups[0] + + if first is None: + return None + + if any(lookup != first for lookup in finder.lookups): + return None + + if is_argument_reassigned(func, first[0]) or is_argument_reassigned(func, first[1]): + return None + + return first + + +def make_dictionary_generic( + base: CallableType, + mapping_index: int, + key_index: int, + dict_type: Instance, + object_type: Instance, +) -> CallableType | None: + """Build a candidate with dict[K, V], K -> V.""" + + if base.variables: + return None + + if mapping_index == key_index: + return None + + if not (0 <= mapping_index < len(base.arg_types) and 0 <= key_index < len(base.arg_types)): + return None + + if dict_type.type.fullname != "builtins.dict": + return None + + if not all( + isinstance(get_proper_type(tp), AnyType) + for tp in (base.arg_types[mapping_index], base.arg_types[key_index], base.ret_type) + ): + return None + + default = AnyType(TypeOfAny.from_omitted_generics) + + key_var = TypeVarType( + "K", "K", TypeVarId(-1, namespace="dmypy_suggest"), [], object_type, default + ) + + value_var = TypeVarType( + "V", "V", TypeVarId(-2, namespace="dmypy_suggest"), [], object_type, default + ) + + new_arg_types = base.arg_types.copy() + + new_arg_types[mapping_index] = dict_type.copy_modified(args=[key_var, value_var]) + new_arg_types[key_index] = key_var + + return base.copy_modified( + arg_types=new_arg_types, ret_type=value_var, variables=[key_var, value_var] + ) + + +class CallbackApplicationFinder(TraverserVisitor): + """Find returns of the form callback(arg).""" + + def __init__(self, func: FuncDef) -> None: + self.arguments = [arg.variable for arg in func.arguments] + self.applications: list[tuple[int, int] | None] = [] + + def visit_return_stmt(self, stmt: ReturnStmt) -> None: + expr = stmt.expr + + if ( + not isinstance(expr, CallExpr) + or not isinstance(expr.callee, NameExpr) + or len(expr.args) != 1 + or expr.arg_kinds != [ARG_POS] + or not isinstance(expr.args[0], NameExpr) + ): + self.applications.append(None) + return + + callback_index = next( + (i for i, arg in enumerate(self.arguments) if expr.callee.node is arg), None + ) + + arg_index = next( + (i for i, arg in enumerate(self.arguments) if expr.args[0].node is arg), None + ) + + if callback_index is None or arg_index is None or callback_index == arg_index: + self.applications.append(None) + return + + self.applications.append((callback_index, arg_index)) + + def visit_func_def(self, node: FuncDef) -> None: + # Ignore nested functions. + pass + + +def find_callback_application(func: FuncDef) -> tuple[int, int] | None: + """Find arguments used in direct callback(arg) returns.""" + + finder = CallbackApplicationFinder(func) + func.body.accept(finder) + + if not finder.applications: + return None + + first = finder.applications[0] + + if first is None: + return None + + if any(app != first for app in finder.applications): + return None + + if is_argument_reassigned(func, first[0]) or is_argument_reassigned(func, first[1]): + return None + + return first + + +def make_callback_generic( + base: CallableType, + callback_index: int, + arg_index: int, + object_type: Instance, + function_type: Instance | None = None, +) -> CallableType | None: + """Build a candidate with Callable[[T], R], T -> R.""" + + if base.variables: + return None + + if callback_index == arg_index: + return None + + if not (0 <= callback_index < len(base.arg_types) and 0 <= arg_index < len(base.arg_types)): + return None + + if function_type is None: + function_type = base.fallback + + if function_type.type.fullname != "builtins.function": + return None + + if not all( + isinstance(get_proper_type(tp), AnyType) + for tp in (base.arg_types[callback_index], base.arg_types[arg_index], base.ret_type) + ): + return None + + default = AnyType(TypeOfAny.from_omitted_generics) + + arg_var = TypeVarType( + "T", "T", TypeVarId(-1, namespace="dmypy_suggest"), [], object_type, default + ) + + ret_var = TypeVarType( + "R", "R", TypeVarId(-2, namespace="dmypy_suggest"), [], object_type, default + ) + + callback_type = CallableType([arg_var], [ARG_POS], [None], ret_var, function_type) + + new_arg_types = base.arg_types.copy() + new_arg_types[callback_index] = callback_type + new_arg_types[arg_index] = arg_var + + return base.copy_modified( + arg_types=new_arg_types, ret_type=ret_var, variables=[arg_var, ret_var] + ) + + +class ArgumentReassignmentFinder(TraverserVisitor): + """Detect whether a function argument is reassigned.""" + + def __init__(self, variable: Var) -> None: + self.variable = variable + self.reassigned = False + + def check_target(self, target: Expression) -> None: + if isinstance(target, NameExpr): + if target.node is self.variable: + self.reassigned = True + + elif isinstance(target, (TupleExpr, ListExpr)): + for item in target.items: + self.check_target(item) + + elif isinstance(target, StarExpr): + self.check_target(target.expr) + + def visit_assignment_stmt(self, stmt: AssignmentStmt) -> None: + for target in stmt.lvalues: + self.check_target(target) + super().visit_assignment_stmt(stmt) + + def visit_operator_assignment_stmt(self, stmt: OperatorAssignmentStmt) -> None: + self.check_target(stmt.lvalue) + super().visit_operator_assignment_stmt(stmt) + + def visit_assignment_expr(self, expr: AssignmentExpr) -> None: + self.check_target(expr.target) + super().visit_assignment_expr(expr) + + def visit_for_stmt(self, stmt: ForStmt) -> None: + self.check_target(stmt.index) + super().visit_for_stmt(stmt) + + def visit_with_stmt(self, stmt: WithStmt) -> None: + for target in stmt.target: + if target is not None: + self.check_target(target) + super().visit_with_stmt(stmt) + + def visit_del_stmt(self, stmt: DelStmt) -> None: + self.check_target(stmt.expr) + super().visit_del_stmt(stmt) + + def visit_func_def(self, node: FuncDef) -> None: + # Do not traverse nested functions. + pass + + +def is_argument_reassigned(func: FuncDef, index: int) -> bool: + """Check whether an argument is rebound in the function body.""" + + if not 0 <= index < len(func.arguments): + return True + + variable = func.arguments[index].variable + finder = ArgumentReassignmentFinder(variable) + + func.body.accept(finder) + return finder.reassigned + + +def infer_generic_candidates( + func: FuncDef, + base: CallableType, + callsite_arg_types: Sequence[Sequence[Sequence[Type]]], + named_type: Callable[[str], Instance], + min_calls: int = 2, +) -> list[CallableType]: + """Generate generic candidates from a function and its call sites. + + Currently supports ordinary functions with one actual argument + per formal parameter at each call site. + + Candidates must be validated by SuggestionEngine.try_type(). + """ + + if len(base.arg_types) != len(func.arguments): + return [] + + if min_calls < 2 or len(callsite_arg_types) < min_calls: + return [] + + observed: list[list[Type]] = [[] for _ in base.arg_types] + + for call in callsite_arg_types: + if len(call) != len(observed): + return [] + + for i, actual_types in enumerate(call): + if len(actual_types) != 1: + return [] + + typ = get_proper_type(remove_instance_last_known_values(actual_types[0])) + + if isinstance(typ, AnyType): + return [] + + observed[i].append(typ) + + def varies(types: Sequence[Type]) -> bool: + return any(t != types[0] for t in types[1:]) + + object_type = named_type("builtins.object") + + direct = find_direct_returned_argument(func) + + if direct is not None and varies(observed[direct]): + candidate = make_direct_return_generic(base, direct, object_type) + return [candidate] if candidate is not None else [] + + indexed = find_indexed_returned_argument(func) + + if indexed is not None and varies(observed[indexed]): + types = [get_proper_type(t) for t in observed[indexed]] + + if all(isinstance(t, Instance) and t.type.fullname == "builtins.list" for t in types): + candidate = make_list_element_generic( + base, indexed, named_type("builtins.list"), object_type + ) + return [candidate] if candidate is not None else [] + + def tuple_element_type(t: Type) -> Type | None: + t = get_proper_type(t) + + if isinstance(t, TupleType): + if bool(t.items) and all(item == t.items[0] for item in t.items): + return get_proper_type(t.items[0]) + return None + + if ( + isinstance(t, Instance) + and t.type.fullname == "builtins.tuple" + and len(t.args) == 1 + ): + return get_proper_type(t.args[0]) + + return None + + element_types = [tuple_element_type(t) for t in types] + + if all(elem is not None for elem in element_types) and varies( + [elem for elem in element_types if elem is not None] + ): + candidate = make_tuple_element_generic( + base, indexed, named_type("builtins.tuple"), object_type + ) + return [candidate] if candidate is not None else [] + + lookup = find_dictionary_lookup(func) + + if lookup is not None: + mapping_index, key_index = lookup + + mappings = [get_proper_type(t) for t in observed[mapping_index]] + + if all( + isinstance(t, Instance) and t.type.fullname == "builtins.dict" and len(t.args) == 2 + for t in mappings + ): + matching_keys = all( + get_proper_type(mapping.args[0]) == get_proper_type(key_type) + for mapping, key_type in zip(mappings, observed[key_index]) + if isinstance(mapping, Instance) + ) + + if matching_keys and varies(observed[mapping_index]): + candidate = make_dictionary_generic( + base, mapping_index, key_index, named_type("builtins.dict"), object_type + ) + return [candidate] if candidate is not None else [] + + callback = find_callback_application(func) + + if callback is not None: + callback_index, arg_index = callback + + callbacks = [get_proper_type(t) for t in observed[callback_index]] + + if all( + isinstance(t, CallableType) + and not t.is_ellipsis_args + and len(t.arg_types) == 1 + and t.arg_kinds[0] in (ARG_POS, ARG_OPT) + and not isinstance(get_proper_type(t.arg_types[0]), AnyType) + and not isinstance(get_proper_type(t.ret_type), AnyType) + for t in callbacks + ): + matching_args = all( + get_proper_type(cb.arg_types[0]) == get_proper_type(arg_type) + for cb, arg_type in zip(callbacks, observed[arg_index]) + if isinstance(cb, CallableType) + ) + + cb_arg_types = [ + get_proper_type(cb.arg_types[0]) + for cb in callbacks + if isinstance(cb, CallableType) + ] + cb_ret_types = [ + get_proper_type(cb.ret_type) for cb in callbacks if isinstance(cb, CallableType) + ] + + if matching_args and (varies(cb_arg_types) or varies(cb_ret_types)): + candidate = make_callback_generic( + base, callback_index, arg_index, object_type, named_type("builtins.function") + ) + return [candidate] if candidate is not None else [] + + return [] diff --git a/mypy/suggestions.py b/mypy/suggestions.py index 6a66f8345408d..d73f76ef03b75 100644 --- a/mypy/suggestions.py +++ b/mypy/suggestions.py @@ -28,13 +28,16 @@ import json import os import sys +from collections import Counter from collections.abc import Callable, Iterator from contextlib import contextmanager from typing import NamedTuple, TypedDict, TypeVar, cast +from typing_extensions import NotRequired from mypy.argmap import map_actuals_to_formals from mypy.build import Graph, State from mypy.find_sources import InvalidSourceList, SourceFinder +from mypy.generic_suggestions import infer_generic_candidates from mypy.join import join_type_list from mypy.meet import meet_type_list from mypy.modulefinder import PYTHON_EXTENSIONS @@ -87,6 +90,7 @@ class PyAnnotateSignature(TypedDict): return_type: str arg_types: list[str] + type_params: NotRequired[list[str]] class Callsite(NamedTuple): @@ -255,6 +259,7 @@ def __init__( flex_any: float | None = None, use_fixme: str | None = None, max_guesses: int | None = None, + generic_min_calls: int | None = None, ) -> None: self.fgmanager = fgmanager self.manager = fgmanager.manager @@ -270,6 +275,7 @@ def __init__( self.max_guesses = max_guesses or 64 self.use_fixme = use_fixme + self.generic_min_calls = generic_min_calls if generic_min_calls is not None else 2 def suggest(self, function: str) -> str: """Suggest an inferred type for function.""" @@ -475,6 +481,49 @@ def get_guesses_from_parent(self, node: FuncDef) -> list[CallableType]: return [] + def get_generic_callsites( + self, + node: FuncDef, + is_method: bool, + defaults: list[Type | None], + callsites: list[Callsite], + ) -> list[list[list[Type]]]: + """Normalize callsite argument types for generic candidate inference.""" + if any(k in (ARG_STAR, ARG_STAR2) for k in node.arg_kinds): + return [] + + expected_formals = len(node.arguments) - int(is_method) + normalized: list[list[list[Type]]] = [] + + for call in callsites: + if len(call.arg_types) != expected_formals: + continue + if any(k in (ARG_STAR, ARG_STAR2) for kinds in call.arg_kinds for k in kinds): + continue + + call_args: list[list[Type]] = [] + if is_method: + assert node.info is not None + call_args.append([Instance(node.info, [])]) + + valid = True + for i, actual_types in enumerate(call.arg_types): + formal_idx = i + int(is_method) + filtered = [t for t in actual_types if not is_implicit_any(t)] + if not filtered and defaults[formal_idx] is not None: + default_type = defaults[formal_idx] + assert default_type is not None + filtered = [default_type] + if len(filtered) != 1: + valid = False + break + call_args.append(filtered) + + if valid: + normalized.append(call_args) + + return normalized + def get_suggestion(self, mod: str, node: FuncDef) -> PyAnnotateSignature: """Compute a suggestion for a function. @@ -488,15 +537,11 @@ def get_suggestion(self, mod: str, node: FuncDef) -> PyAnnotateSignature: raise SuggestionFailure("Function does not typecheck.") is_method = bool(node.info) and node.has_self_or_cls_argument + starting_type = self.get_starting_type(node) + defaults = self.get_default_arg_types(node) with state.strict_optional_set(graph[mod].options.strict_optional): - guesses = self.get_guesses( - is_method, - self.get_starting_type(node), - self.get_default_arg_types(node), - callsites, - uses, - ) + guesses = self.get_guesses(is_method, starting_type, defaults, callsites, uses) guesses += self.get_guesses_from_parent(node) guesses = self.filter_options(guesses, is_method, ignore_return=True) best, _ = self.find_best(node, guesses) @@ -514,6 +559,42 @@ def get_suggestion(self, mod: str, node: FuncDef) -> PyAnnotateSignature: guesses = self.filter_options(guesses, is_method, ignore_return=False) best, errors = self.find_best(node, guesses) + generic_candidates = infer_generic_candidates( + node, + starting_type, + self.get_generic_callsites(node, is_method, defaults, callsites), + self.named_type, + min_calls=self.generic_min_calls, + ) + + if generic_candidates: + generic_guesses = [refine_callable(cand, best) for cand in generic_candidates] + generic_guesses = self.filter_options(generic_guesses, is_method, ignore_return=False) + + safe_generics = [] + + for candidate in generic_guesses: + candidate_errors = self.try_type(node, candidate) + + if not has_new_errors(orig_errors, candidate_errors): + safe_generics.append((candidate, candidate_errors)) + + if safe_generics: + best_generic, messages = min( + safe_generics, + key=lambda item: (count_errors(item[1]), self.score_callable(item[0])), + ) + + generic_errors = count_errors(messages) + + if ( + generic_errors <= count_errors(orig_errors) + and generic_errors <= errors + and self.score_callable(best_generic) <= self.score_callable(best) + ): + best = best_generic + errors = generic_errors + if self.no_errors and errors: raise SuggestionFailure("No annotation without errors") @@ -693,7 +774,10 @@ def try_type(self, func: FuncDef, typ: ProperType) -> list[str]: # We set type to None to ensure that the type always changes during # reprocessing. func.type = None - func.unanalyzed_type = typ + if isinstance(typ, CallableType): + func.unanalyzed_type = typ.copy_modified() + else: + func.unanalyzed_type = typ try: res = self.fgmanager.trigger(func.fullname) # if res: @@ -746,14 +830,19 @@ def pyannotate_signature( ) -> PyAnnotateSignature: """Format a callable type as a pyannotate dict""" start = int(is_method) - return { + sig: PyAnnotateSignature = { "arg_types": [self.format_type(cur_module, t) for t in typ.arg_types[start:]], "return_type": self.format_type(cur_module, typ.ret_type), } + if typ.variables: + sig["type_params"] = [var.name for var in typ.variables] + return sig def format_signature(self, sig: PyAnnotateSignature) -> str: """Format a callable type in a way suitable as an annotation... kind of""" - return f"({', '.join(sig['arg_types'])}) -> {sig['return_type']}" + type_params = sig.get("type_params") + prefix = f"[{', '.join(type_params)}]" if type_params else "" + return f"{prefix}({', '.join(sig['arg_types'])}) -> {sig['return_type']}" def format_type(self, cur_module: str | None, typ: Type) -> str: if self.use_fixme and isinstance(get_proper_type(typ), AnyType): @@ -862,7 +951,7 @@ def visit_instance(self, t: Instance) -> str: mod = self.module if (mod, obj) == ("builtins", "tuple"): - mod, obj = "typing", "Tuple[" + t.args[0].accept(self) + ", ...]" + obj = f"tuple[{t.args[0].accept(self)}, ...]" elif t.args: obj += f"[{self.list_str(t.args)}]" @@ -882,6 +971,9 @@ def visit_tuple_type(self, t: TupleType) -> str: s = self.list_str(t.items) return f"Tuple[{s}]" + def visit_type_var(self, t: TypeVarType) -> str: + return t.name + def visit_uninhabited_type(self, t: UninhabitedType) -> str: return "Any" @@ -951,6 +1043,15 @@ def count_errors(msgs: list[str]) -> int: return len([x for x in msgs if " error: " in x]) +def has_new_errors(original: list[str], proposed: list[str]) -> bool: + """Check whether a proposed signature introduces new errors.""" + + original_errors = Counter(msg for msg in original if " error: " in msg) + proposed_errors = Counter(msg for msg in proposed if " error: " in msg) + + return bool(proposed_errors - original_errors) + + def refine_type(ti: Type, si: Type) -> Type: """Refine `ti` by replacing Anys in it with information taken from `si` diff --git a/mypy/test/testfinegrained.py b/mypy/test/testfinegrained.py index 26531c0b25f29..5257ea9ee4b4d 100644 --- a/mypy/test/testfinegrained.py +++ b/mypy/test/testfinegrained.py @@ -317,6 +317,8 @@ def maybe_suggest(self, step: int, server: Server, src: str, tmp_dir: str) -> li use_fixme = m.group(1) if m else None m = re.match("--max-guesses=([0-9]+)", flags) max_guesses = int(m.group(1)) if m else None + m = re.match("--generic-min-calls=([0-9]+)", flags) + generic_min_calls = int(m.group(1)) if m else None res: dict[str, Any] = server.cmd_suggest( target.strip(), json=json, @@ -326,6 +328,7 @@ def maybe_suggest(self, step: int, server: Server, src: str, tmp_dir: str) -> li use_fixme=use_fixme, callsites=callsites, max_guesses=max_guesses, + generic_min_calls=generic_min_calls, ) val = res["error"] if "error" in res else res["out"] + res["err"] if json: diff --git a/mypy/test/testgeneric_suggestions.py b/mypy/test/testgeneric_suggestions.py new file mode 100644 index 0000000000000..03008ee8c34d2 --- /dev/null +++ b/mypy/test/testgeneric_suggestions.py @@ -0,0 +1,779 @@ +import os +import tempfile +from unittest import TestCase + +from mypy.dmypy_server import Server +from mypy.dmypy_util import DEFAULT_STATUS_FILE +from mypy.find_sources import create_source_list +from mypy.generic_suggestions import ( + find_callback_application, + find_dictionary_lookup, + find_direct_returned_argument, + find_indexed_returned_argument, + infer_generic_candidates, + make_callback_generic, + make_dictionary_generic, + make_direct_return_generic, + make_list_element_generic, + make_tuple_element_generic, +) +from mypy.nodes import ( + ARG_POS, + Argument, + AssignmentStmt, + Block, + CallExpr, + ClassDef, + FuncDef, + IndexExpr, + IntExpr, + NameExpr, + OperatorAssignmentStmt, + ReturnStmt, + SymbolTable, + TypeInfo, + Var, +) +from mypy.options import Options +from mypy.suggestions import has_new_errors +from mypy.types import ( + AnyType, + CallableType, + Instance, + TupleType, + Type, + TypeOfAny, + TypeVarType, + get_proper_type, +) + + +def make_function(argument_names: list[str], returned_name: str | None) -> FuncDef: + variables = [Var(name) for name in argument_names] + + arguments = [Argument(variable, None, None, ARG_POS) for variable in variables] + + if returned_name is None: + expression = IntExpr(42) + else: + expression = NameExpr(returned_name) + expression.node = next(var for var in variables if var.name == returned_name) + + return FuncDef("example", arguments, Block([ReturnStmt(expression)])) + + +def make_instance(fullname: str) -> Instance: + module, _, name = fullname.rpartition(".") + + class_def = ClassDef(name, Block([]), None, []) + class_def.fullname = fullname + + info = TypeInfo(SymbolTable(), class_def, module) + info.mro = [info] + + return Instance(info, []) + + +def make_indexed_function( + names: list[str], collection_index: int, key_index: int | None = None +) -> FuncDef: + func = make_function(names, names[collection_index]) + + collection = NameExpr(names[collection_index]) + collection.node = func.arguments[collection_index].variable + + if key_index is None: + index = IntExpr(0) + else: + index = NameExpr(names[key_index]) + index.node = func.arguments[key_index].variable + + func.body.body = [ReturnStmt(IndexExpr(collection, index))] + + return func + + +def make_callback_function(names: list[str], callback_index: int, arg_index: int) -> FuncDef: + func = make_function(names, names[callback_index]) + + callee = NameExpr(names[callback_index]) + callee.node = func.arguments[callback_index].variable + + arg = NameExpr(names[arg_index]) + arg.node = func.arguments[arg_index].variable + + func.body.body = [ReturnStmt(CallExpr(callee, [arg], [ARG_POS], [None]))] + + return func + + +class GenericSuggestionsTests(TestCase): + def test_identity(self) -> None: + func = make_function(["x"], "x") + assert find_direct_returned_argument(func) == 0 + + def test_first_argument(self) -> None: + func = make_function(["x", "y"], "x") + assert find_direct_returned_argument(func) == 0 + + def test_second_argument(self) -> None: + func = make_function(["x", "y"], "y") + assert find_direct_returned_argument(func) == 1 + + def test_constant_return(self) -> None: + func = make_function(["x"], None) + assert find_direct_returned_argument(func) is None + + def test_multiple_same_returns(self) -> None: + func = make_function(["x", "y"], "x") + + expr = NameExpr("x") + expr.node = func.arguments[0].variable + func.body.body.append(ReturnStmt(expr)) + + assert find_direct_returned_argument(func) == 0 + + def test_multiple_different_returns(self) -> None: + func = make_function(["x", "y"], "x") + + expr = NameExpr("y") + expr.node = func.arguments[1].variable + func.body.body.append(ReturnStmt(expr)) + + assert find_direct_returned_argument(func) is None + + def test_no_return(self) -> None: + func = make_function(["x"], "x") + func.body.body.clear() + + assert find_direct_returned_argument(func) is None + + def test_make_generic_identity(self) -> None: + object_type = make_instance("builtins.object") + function_type = make_instance("builtins.function") + + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType([any_type], [ARG_POS], ["value"], any_type, function_type) + + candidate = make_direct_return_generic(base, 0, object_type) + + assert candidate is not None + assert len(candidate.variables) == 1 + + type_var = candidate.variables[0] + assert isinstance(type_var, TypeVarType) + assert type_var.name == "T" + + assert candidate.arg_types[0] is type_var + assert candidate.ret_type is type_var + + def test_indexed_return(self) -> None: + func = make_function(["items"], "items") + + expr = NameExpr("items") + expr.node = func.arguments[0].variable + + func.body.body = [ReturnStmt(IndexExpr(expr, IntExpr(0)))] + + assert find_indexed_returned_argument(func) == 0 + + def test_make_list_generic(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type], [ARG_POS], ["items"], any_type, make_instance("builtins.function") + ) + + candidate = make_list_element_generic( + base, 0, make_instance("builtins.list"), make_instance("builtins.object") + ) + + assert candidate is not None + assert len(candidate.variables) == 1 + + type_var = candidate.variables[0] + assert isinstance(type_var, TypeVarType) + + list_arg = get_proper_type(candidate.arg_types[0]) + assert isinstance(list_arg, Instance) + assert list_arg.type.fullname == "builtins.list" + + assert list_arg.args[0] is type_var + assert candidate.ret_type is type_var + + def test_dictionary_lookup(self) -> None: + func = make_function(["mapping", "key"], "mapping") + + mapping = NameExpr("mapping") + mapping.node = func.arguments[0].variable + + key = NameExpr("key") + key.node = func.arguments[1].variable + + func.body.body = [ReturnStmt(IndexExpr(mapping, key))] + + assert find_dictionary_lookup(func) == (0, 1) + + def test_dictionary_lookup_with_other_return(self) -> None: + func = make_function(["mapping", "key"], "mapping") + + mapping = NameExpr("mapping") + mapping.node = func.arguments[0].variable + + key = NameExpr("key") + key.node = func.arguments[1].variable + + func.body.body = [ReturnStmt(IndexExpr(mapping, key)), ReturnStmt(IntExpr(42))] + + assert find_dictionary_lookup(func) is None + + def test_make_dictionary_generic(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type, any_type], + [ARG_POS, ARG_POS], + ["mapping", "key"], + any_type, + make_instance("builtins.function"), + ) + + candidate = make_dictionary_generic( + base, 0, 1, make_instance("builtins.dict"), make_instance("builtins.object") + ) + + assert candidate is not None + assert len(candidate.variables) == 2 + + key_var, value_var = candidate.variables + + assert isinstance(key_var, TypeVarType) + assert isinstance(value_var, TypeVarType) + + mapping_type = get_proper_type(candidate.arg_types[0]) + assert isinstance(mapping_type, Instance) + + assert mapping_type.type.fullname == "builtins.dict" + assert mapping_type.args == (key_var, value_var) + + assert candidate.arg_types[1] is key_var + assert candidate.ret_type is value_var + + def test_make_tuple_generic(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type], [ARG_POS], ["items"], any_type, make_instance("builtins.function") + ) + + candidate = make_tuple_element_generic( + base, 0, make_instance("builtins.tuple"), make_instance("builtins.object") + ) + + assert candidate is not None + assert len(candidate.variables) == 1 + + type_var = candidate.variables[0] + assert isinstance(type_var, TypeVarType) + + tuple_arg = get_proper_type(candidate.arg_types[0]) + assert isinstance(tuple_arg, Instance) + assert tuple_arg.type.fullname == "builtins.tuple" + + assert tuple_arg.args[0] is type_var + assert candidate.ret_type is type_var + + def test_reject_wrong_collection_type(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type], [ARG_POS], ["items"], any_type, make_instance("builtins.function") + ) + + candidate = make_tuple_element_generic( + base, 0, make_instance("builtins.list"), make_instance("builtins.object") + ) + + assert candidate is None + + def test_reassigned_argument(self) -> None: + func = make_function(["x"], "x") + + target = NameExpr("x") + target.node = func.arguments[0].variable + + assignment = AssignmentStmt([target], IntExpr(42)) + + func.body.body.insert(0, assignment) + + assert find_direct_returned_argument(func) is None + + def test_other_argument_reassigned(self) -> None: + func = make_function(["x", "y"], "x") + + target = NameExpr("y") + target.node = func.arguments[1].variable + + assignment = AssignmentStmt([target], IntExpr(42)) + + func.body.body.insert(0, assignment) + + assert find_direct_returned_argument(func) == 0 + + def test_augmented_assignment(self) -> None: + func = make_function(["x"], "x") + + target = NameExpr("x") + target.node = func.arguments[0].variable + + assignment = OperatorAssignmentStmt("+=", target, IntExpr(1)) + + func.body.body.insert(0, assignment) + + assert find_direct_returned_argument(func) is None + + def test_infer_generic_identity(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type], [ARG_POS], ["value"], any_type, make_instance("builtins.function") + ) + + func = make_function(["value"], "value") + + int_type = make_instance("builtins.int") + str_type = make_instance("builtins.str") + + callsites = [[[int_type]], [[str_type]]] + + candidates = infer_generic_candidates(func, base, callsites, make_instance) + + assert len(candidates) == 1 + + candidate = candidates[0] + + assert len(candidate.variables) == 1 + assert candidate.arg_types[0] == candidate.ret_type + + def test_no_generic_for_single_call(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type], [ARG_POS], ["value"], any_type, make_instance("builtins.function") + ) + + func = make_function(["value"], "value") + + candidates = infer_generic_candidates( + func, base, [[[make_instance("builtins.int")]]], make_instance + ) + + assert candidates == [] + + def test_infer_list_generic(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type], [ARG_POS], ["items"], any_type, make_instance("builtins.function") + ) + + func = make_indexed_function(["items"], 0) + + list_type = make_instance("builtins.list") + int_type = make_instance("builtins.int") + str_type = make_instance("builtins.str") + + callsites = [ + [[list_type.copy_modified(args=[int_type])]], + [[list_type.copy_modified(args=[str_type])]], + ] + + candidates = infer_generic_candidates(func, base, callsites, make_instance) + + assert len(candidates) == 1 + + candidate = candidates[0] + collection = get_proper_type(candidate.arg_types[0]) + + assert isinstance(collection, Instance) + assert collection.type.fullname == "builtins.list" + assert collection.args[0] is candidate.ret_type + + def test_infer_tuple_generic(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type], [ARG_POS], ["items"], any_type, make_instance("builtins.function") + ) + + func = make_indexed_function(["items"], 0) + + tuple_type = make_instance("builtins.tuple") + int_type = make_instance("builtins.int") + str_type = make_instance("builtins.str") + + callsites = [ + [[TupleType([int_type, int_type], tuple_type)]], + [[TupleType([str_type, str_type], tuple_type)]], + ] + + candidates = infer_generic_candidates(func, base, callsites, make_instance) + + assert len(candidates) == 1 + + candidate = candidates[0] + collection = get_proper_type(candidate.arg_types[0]) + + assert isinstance(collection, Instance) + assert collection.type.fullname == "builtins.tuple" + assert collection.args[0] is candidate.ret_type + + def test_infer_dictionary_generic(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type, any_type], + [ARG_POS, ARG_POS], + ["mapping", "key"], + any_type, + make_instance("builtins.function"), + ) + + func = make_indexed_function(["mapping", "key"], 0, 1) + + dict_type = make_instance("builtins.dict") + str_type = make_instance("builtins.str") + int_type = make_instance("builtins.int") + bool_type = make_instance("builtins.bool") + + callsites = [ + [[dict_type.copy_modified(args=[str_type, int_type])], [str_type]], + [[dict_type.copy_modified(args=[int_type, bool_type])], [int_type]], + ] + + candidates = infer_generic_candidates(func, base, callsites, make_instance) + + assert len(candidates) == 1 + + candidate = candidates[0] + + assert len(candidate.variables) == 2 + key_var, value_var = candidate.variables + + mapping = get_proper_type(candidate.arg_types[0]) + + assert isinstance(mapping, Instance) + assert mapping.args == (key_var, value_var) + assert candidate.arg_types[1] is key_var + assert candidate.ret_type is value_var + + def test_reject_mismatched_dictionary_key(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type, any_type], + [ARG_POS, ARG_POS], + ["mapping", "key"], + any_type, + make_instance("builtins.function"), + ) + + func = make_indexed_function(["mapping", "key"], 0, 1) + + dict_type = make_instance("builtins.dict") + str_type = make_instance("builtins.str") + int_type = make_instance("builtins.int") + + callsites = [ + [[dict_type.copy_modified(args=[str_type, int_type])], [int_type]], + [[dict_type.copy_modified(args=[str_type, int_type])], [str_type]], + ] + + candidates = infer_generic_candidates(func, base, callsites, make_instance) + + assert candidates == [] + + def test_callback_application(self) -> None: + func = make_callback_function(["f", "x"], 0, 1) + assert find_callback_application(func) == (0, 1) + + def test_callback_application_reversed_params(self) -> None: + func = make_callback_function(["x", "f"], 1, 0) + assert find_callback_application(func) == (1, 0) + + def test_make_callback_generic(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + function_type = make_instance("builtins.function") + + base = CallableType( + [any_type, any_type], [ARG_POS, ARG_POS], ["f", "x"], any_type, function_type + ) + + candidate = make_callback_generic( + base, 0, 1, make_instance("builtins.object"), function_type + ) + + assert candidate is not None + assert len(candidate.variables) == 2 + + arg_var, ret_var = candidate.variables + assert isinstance(arg_var, TypeVarType) + assert isinstance(ret_var, TypeVarType) + assert arg_var.name == "T" + assert ret_var.name == "R" + + cb_type = get_proper_type(candidate.arg_types[0]) + assert isinstance(cb_type, CallableType) + assert cb_type.arg_types == [arg_var] + assert cb_type.ret_type is ret_var + assert candidate.arg_types[1] is arg_var + assert candidate.ret_type is ret_var + + def test_infer_callback_generic(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + function_type = make_instance("builtins.function") + + base = CallableType( + [any_type, any_type], [ARG_POS, ARG_POS], ["f", "x"], any_type, function_type + ) + + func = make_callback_function(["f", "x"], 0, 1) + + int_type = make_instance("builtins.int") + str_type = make_instance("builtins.str") + bool_type = make_instance("builtins.bool") + + cb1 = CallableType([int_type], [ARG_POS], [None], str_type, function_type) + cb2 = CallableType([str_type], [ARG_POS], [None], bool_type, function_type) + + callsites: list[list[list[Type]]] = [[[cb1], [int_type]], [[cb2], [str_type]]] + + candidates = infer_generic_candidates(func, base, callsites, make_instance) + + assert len(candidates) == 1 + candidate = candidates[0] + assert len(candidate.variables) == 2 + arg_var, ret_var = candidate.variables + cb_arg = get_proper_type(candidate.arg_types[0]) + assert isinstance(cb_arg, CallableType) + assert cb_arg.arg_types == [arg_var] + assert cb_arg.ret_type is ret_var + assert candidate.arg_types[1] is arg_var + assert candidate.ret_type is ret_var + + def test_reject_mismatched_callback_arg(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + function_type = make_instance("builtins.function") + + base = CallableType( + [any_type, any_type], [ARG_POS, ARG_POS], ["f", "x"], any_type, function_type + ) + + func = make_callback_function(["f", "x"], 0, 1) + + int_type = make_instance("builtins.int") + str_type = make_instance("builtins.str") + + cb1 = CallableType([int_type], [ARG_POS], [None], str_type, function_type) + cb2 = CallableType([str_type], [ARG_POS], [None], int_type, function_type) + + callsites: list[list[list[Type]]] = [[[cb1], [str_type]], [[cb2], [int_type]]] + + candidates = infer_generic_candidates(func, base, callsites, make_instance) + + assert candidates == [] + + def test_infer_generic_min_calls(self) -> None: + any_type = AnyType(TypeOfAny.suggestion_engine) + + base = CallableType( + [any_type], [ARG_POS], ["value"], any_type, make_instance("builtins.function") + ) + + func = make_function(["value"], "value") + + int_type = make_instance("builtins.int") + str_type = make_instance("builtins.str") + bool_type = make_instance("builtins.bool") + + two_calls = [[[int_type]], [[str_type]]] + three_calls = [[[int_type]], [[str_type]], [[bool_type]]] + + assert infer_generic_candidates(func, base, two_calls, make_instance, min_calls=3) == [] + assert ( + len(infer_generic_candidates(func, base, three_calls, make_instance, min_calls=3)) == 1 + ) + + def test_no_new_errors(self) -> None: + original = ["foo.py:5: error: Error A"] + proposed = ["foo.py:5: error: Error A"] + + assert not has_new_errors(original, proposed) + + def test_reject_different_error(self) -> None: + original = ["foo.py:5: error: Error A"] + proposed = ["foo.py:8: error: Error B"] + + assert has_new_errors(original, proposed) + + def test_allow_removed_errors(self) -> None: + original = ["foo.py:5: error: Error A"] + proposed: list[str] = [] + + assert not has_new_errors(original, proposed) + + def test_reject_additional_error(self) -> None: + original = ["foo.py:5: error: Error A"] + proposed = ["foo.py:5: error: Error A", "foo.py:8: error: Error B"] + + assert has_new_errors(original, proposed) + + +class DmypySuggestGenericIntegrationTests(TestCase): + """End-to-end integration tests running dmypy suggest on real Python files.""" + + tmpdir: tempfile.TemporaryDirectory[str] + server: Server + + @classmethod + def setUpClass(cls) -> None: + cls.tmpdir = tempfile.TemporaryDirectory() + mod_path = os.path.join(cls.tmpdir.name, "sample.py") + source = """ +def identity(value): + return value + +def first_list(items): + return items[0] + +def first_tuple(items): + return items[0] + +def lookup(mapping, key): + return mapping[key] + +def apply(f, x): + return f(x) + +def to_str(n: int) -> str: + return str(n) + +def str_len(s: str) -> int: + return len(s) + +def add_self(x): + y = x + x + return x + +def same_type_identity(value): + return value + +def single_call_identity(value): + return value + +class Box: + def echo(self, item): + return item + +a_id: int = identity(1) +b_id: str = identity(value="hello") + +a_list: int = first_list([1, 2, 3]) +b_list: str = first_list(["a", "b"]) + +a_tuple: int = first_tuple((1, 2)) +b_tuple: str = first_tuple(("a", "b")) + +a_dict: int = lookup({"x": 1}, "x") +b_dict: str = lookup({10: "ten"}, 10) + +a_cb: str = apply(to_str, 42) +b_cb: int = apply(str_len, "abc") + +add_self(1) +add_self("abc") + +same_type_identity(1) +same_type_identity(2) + +single_call_identity(1) + +box = Box() +a_box: int = box.echo(1) +b_box: str = box.echo("hello") +""" + with open(mod_path, "w", encoding="utf-8") as f: + f.write(source) + + options = Options() + options.incremental = True + options.fine_grained_incremental = True + options.local_partial_types = True + options.show_traceback = True + options.error_summary = False + + status_file = os.path.join(cls.tmpdir.name, DEFAULT_STATUS_FILE) + cls.server = Server(options, status_file) + sources = create_source_list([mod_path], options) + cls.server.check(sources, export_types=False, is_tty=False, terminal_width=-1) + + @classmethod + def tearDownClass(cls) -> None: + cls.tmpdir.cleanup() + + def suggest( + self, + target: str, + *, + generic_min_calls: int | None = None, + json: bool = False, + no_errors: bool = False, + no_any: bool = False, + ) -> str: + res = self.server.cmd_suggest( + f"sample.{target}", + callsites=False, + json=json, + no_errors=no_errors, + no_any=no_any, + flex_any=None, + use_fixme=None, + max_guesses=None, + generic_min_calls=generic_min_calls, + ) + assert "out" in res, f"Unexpected error: {res}" + out = res["out"] + assert isinstance(out, str) + return out.strip() + + def test_dmypy_suggest_identity_generic(self) -> None: + assert self.suggest("identity", no_errors=True) == "[T](T) -> T" + + def test_dmypy_suggest_list_generic(self) -> None: + assert self.suggest("first_list", no_errors=True) == "[T](list[T]) -> T" + + def test_dmypy_suggest_tuple_generic(self) -> None: + assert self.suggest("first_tuple", no_errors=True) == "[T](tuple[T, ...]) -> T" + + def test_dmypy_suggest_dict_generic(self) -> None: + assert self.suggest("lookup", no_errors=True) == "[K, V](dict[K, V], K) -> V" + + def test_dmypy_suggest_callback_generic(self) -> None: + assert self.suggest("apply", no_errors=True) == "[T, R](Callable[[T], R], T) -> R" + + def test_dmypy_suggest_method_generic(self) -> None: + assert self.suggest("Box.echo", no_errors=True) == "[T](T) -> T" + + def test_dmypy_suggest_min_calls_option(self) -> None: + assert self.suggest("identity", generic_min_calls=3) == "(object) -> object" + assert self.suggest("identity", generic_min_calls=2) == "[T](T) -> T" + + def test_dmypy_suggest_rejects_generic_when_body_constrains_type(self) -> None: + assert self.suggest("add_self") == "(object) -> object" + + def test_dmypy_suggest_rejects_generic_for_same_type_calls(self) -> None: + assert self.suggest("same_type_identity") == "(int) -> int" + + def test_dmypy_suggest_rejects_generic_for_single_call(self) -> None: + assert self.suggest("single_call_identity") == "(int) -> int" diff --git a/mypy/typeanal.py b/mypy/typeanal.py index bd6f97cc94cef..9c45af6c005e9 100644 --- a/mypy/typeanal.py +++ b/mypy/typeanal.py @@ -1929,6 +1929,8 @@ def bind_function_type_variables( """Find the type variables of the function type and bind them in our tvar_scope""" has_self_type = False if fun_type.variables: + if all(var.id.namespace == "dmypy_suggest" for var in fun_type.variables): + return tuple(fun_type.variables), has_self_type defs = [] for var in fun_type.variables: if self.api.type and self.api.type.self_type and var == self.api.type.self_type: @@ -1942,6 +1944,14 @@ def bind_function_type_variables( defs.append(binding) return tuple(defs), has_self_type typevars, has_self_type = self.infer_type_variables(fun_type) + if not typevars and isinstance(defn, nodes.FuncItem): + visitor = FindTypeVarVisitor(self.api, self.tvar_scope) + for arg in fun_type.arg_types: + arg.accept(visitor) + visitor.include_callables = False + fun_type.ret_type.accept(visitor) + if visitor.existing_type_vars: + return tuple(visitor.existing_type_vars), has_self_type # Do not define a new type variable if already defined in scope. typevars = [ (name, tvar) for name, tvar in typevars if not self.is_defined_type_var(name, defn) @@ -2763,6 +2773,7 @@ def __init__(self, api: SemanticAnalyzerCoreInterface, scope: TypeVarLikeScope) self.api = api self.scope = scope self.type_var_likes: list[tuple[str, TypeVarLikeExpr]] = [] + self.existing_type_vars: list[TypeVarLikeType] = [] self.has_self_type = False self.include_callables = True @@ -2819,6 +2830,8 @@ def visit_deleted_type(self, t: DeletedType) -> None: pass def visit_type_var(self, t: TypeVarType) -> None: + if t.id.namespace == "dmypy_suggest" and t not in self.existing_type_vars: + self.existing_type_vars.append(t) self.process_types([t.upper_bound, t.default] + t.values) def visit_param_spec(self, t: ParamSpecType) -> None: diff --git a/test-data/unit/fine-grained-suggest.test b/test-data/unit/fine-grained-suggest.test index 5f83e3f35d3c8..188e0f890cd24 100644 --- a/test-data/unit/fine-grained-suggest.test +++ b/test-data/unit/fine-grained-suggest.test @@ -1268,3 +1268,91 @@ optional5(None) [out] (int | None) -> None == + +[case testSuggestGenericFunctions] +# suggest: foo.identity +# suggest: foo.first_list +# suggest: foo.lookup +# suggest: foo.apply +# suggest: --generic-min-calls=3 foo.identity +[file foo.py] +def identity(value): + return value + +def first_list(items): + return items[0] + +def lookup(mapping, key): + return mapping[key] + +def apply(f, x): + return f(x) + +def to_str(x: int) -> str: + return "a" + +def to_int(x: str) -> int: + return 1 + +def caller() -> None: + a: int = identity(1) + b: str = identity("x") + + c: int = first_list([1, 2]) + d: str = first_list(["a", "b"]) + + g: int = lookup({"k": 1}, "k") + h: str = lookup({1: "v"}, 1) + + i: str = apply(to_str, 1) + j: int = apply(to_int, "x") +[builtins fixtures/dict.pyi] +[out] +[T](T) -> T +[T](list[T]) -> T +[K, V](dict[K, V], K) -> V +[T, R](Callable[[T], R], T) -> R +(object) -> object +== + +[case testSuggestGenericTuple] +# suggest: foo.first_tuple +[file foo.py] +def first_tuple(items): + return items[0] + +def caller() -> None: + e: int = first_tuple((1, 2)) + f: str = first_tuple(("a", "b")) +[builtins fixtures/tuple.pyi] +[out] +[T](tuple[T, ...]) -> T +== + +[case testSuggestGenericNegativeCases] +# suggest: foo.single_call +# suggest: foo.same_type +# suggest: foo.reassigned +[file foo.py] +def single_call(x): + return x + +def same_type(x): + return x + +def reassigned(x): + x = 1 + return x + +def caller() -> None: + single_call(1) + same_type(1) + same_type(2) + reassigned(1) + reassigned("a") +[builtins fixtures/isinstancelist.pyi] +[out] +(int) -> int +(int) -> int +(object) -> int +==