From 835e1150d06207d66d560b8fe16715ad20334b96 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Fri, 11 Sep 2026 10:35:28 -0400 Subject: [PATCH 01/12] ci: fork automation (upstream sync + OCR AI review) Squashes the four CI commits this fork carried on top of upstream into one, and cuts the carry from 3776 lines to ~1000. Removed: - .github/{README,QUICKSTART}.md and .github/docs/* (2273 lines). Beyond being bulk we rebase hourly, they had drifted into being wrong: they described a daily (not hourly) sync, Claude 3.5 Sonnet via the direct Anthropic API (we run Opus on Bedrock), a static IAM user (we use OIDC), and a Windows dependency-builder workflow that the same commit which documented it had already deleted. 16 referenced paths did not exist. An AI reviewer reading this repo ingests that as truth. - sync-upstream-manual.yml (252 lines). sync-upstream.yml already has workflow_dispatch; the sole behavioral difference was a force_push toggle whose "false" setting cannot work after a rebase anyway. - .github/.gitignore, which only ignored scripts/ai-review/, a directory that does not exist here. Rewritten: - sync-upstream.yml -> fork-sync-upstream.yml, 260 lines to 47. git rebase already handles the fast-forward, no-op and diverged cases that the ahead/behind arithmetic hand-rolled, so all that goes away, as do the issue open/comment/close machinery (a scheduled-run failure is already emailed to the one consumer, and the run log carries what the issue body would) and the "dev setup|dev v[0-9]" commit allowlist, which matched zero commits on master and had since it was written. Sync drops to every three hours: hourly meant 24 full-history clones of an 840MB repo per day, each force-push also re-triggering pg-ci.yml on master, to collect a handful of upstream commits. Renamed: - ocr-review.yml, ocr-model-check.yml -> fork-*.yml. Upstream owns .github/ too (it has touched pg-ci.yml three times this year), so fork-owned workflows now live under a name prefix upstream will not collide with, making the rebase structurally conflict-free and letting the sync guard match fork-owned paths exactly rather than allowing all of .github/. Unchanged: the OCR review workflows' logic and .github/ocr/* (rule.json, context.md, litellm.yaml, pg-history.py) - the part that carries value. --- .github/ocr/context.md | 126 ++++++ .github/ocr/litellm.yaml | 41 ++ .github/ocr/pg-history.py | 225 +++++++++++ .github/ocr/rule.json | 65 ++++ .github/workflows/fork-ocr-model-check.yml | 89 +++++ .github/workflows/fork-ocr-review.yml | 427 +++++++++++++++++++++ .github/workflows/fork-sync-upstream.yml | 47 +++ 7 files changed, 1020 insertions(+) create mode 100644 .github/ocr/context.md create mode 100644 .github/ocr/litellm.yaml create mode 100644 .github/ocr/pg-history.py create mode 100644 .github/ocr/rule.json create mode 100644 .github/workflows/fork-ocr-model-check.yml create mode 100644 .github/workflows/fork-ocr-review.yml create mode 100644 .github/workflows/fork-sync-upstream.yml diff --git a/.github/ocr/context.md b/.github/ocr/context.md new file mode 100644 index 0000000000000..c4a83b85b124e --- /dev/null +++ b/.github/ocr/context.md @@ -0,0 +1,126 @@ +# OCR review context — PostgreSQL contribution standards + +You are reviewing a change to a **PostgreSQL** fork. Every PR here is destined to +become a patch posted to the **pgsql-hackers** mailing list and tracked in a +**commitfest**. Review with the combined rigor, taste, and attention to detail of +the PostgreSQL committers. This context applies to the *whole* change, on top of +the per-file rules. + +## Review discipline +- Be precise and blunt; lead with the most serious problem. No praise, no + validation of the author, no disclaimers — accuracy is the only metric. +- Verify every claim against the actual diff. Confirm names, signatures, line + numbers, and APIs before asserting. Never invent behavior or cite code not in + the change. If unsure, say so, and tag each finding **high / moderate / low** + confidence. +- Judge the change on its merits regardless of how the PR frames it. A draft PR + is WIP: weight design/approach feedback over style nits. + +## Patch hygiene (top rejection reasons on -hackers) +1. **Minimal diff.** The fastest way to get a patch rejected is unrelated + changes: reformatting untouched lines, rewording unrelated comments, touching + code not required by the change. Flag any hunk not needed for the stated + purpose. After the patch, the code should read as if it had always been + written that way. +2. **Atomic, bisectable commits.** Each commit must build and pass tests on its + own — a broken intermediate commit breaks `git bisect`, revert, and + cherry-pick. Flag a commit that only compiles once a later commit lands. + Prefer one focused patch, or a clearly-ordered series of + independently-committable pieces. +3. **Tests + docs are mandatory.** A user-visible change without regression/TAP + tests **and** documentation is WIP, not commit-ready. New behavior needs + tests that cover edge and error paths, not just the happy path. +4. **DRY / reuse.** Prefer existing infrastructure (`List` in `pg_list.h`, + `StringInfo`, `dynahash`/`simplehash`, `palloc`/`MemoryContext`, `foreach`) + over reinventing it. Flag copy-paste and speculative abstraction alike — the + community wants minimal, targeted changes that fit the subsystem's existing + patterns. +5. **Whitespace.** No trailing whitespace; tabs (width 4) for C indentation; + `git diff --check` must be clean. Whitespace-only churn on untouched lines is + a defect. + +## Committer-owned files — do NOT touch in a patch (flag if present) +These are the committer's job at push time; including them causes needless +merge conflicts and is a mistake: +- **`src/include/catalog/catversion.h`** — the `CATALOG_VERSION_NO` bump is done + by the **committer** when pushing. A catversion bump in the PR is **wrong** — + flag it. (This is the single most common author mistake in catalog patches.) +- **Release notes** (`doc/src/sgml/release-*.sgml`) and version strings + (`configure.ac` `AC_INIT` version, `meson.build` `version`, `PG_VERSION`). + +## Generated files — never hand-edit; edit the source +Flag direct edits to generated output; point the author at the source instead: +- Catalog headers `src/include/catalog/*_d.h`, `postgres.bki`, `schemapg.h`, + `system_constraints.sql` → edit the `pg_*.dat` files. +- `src/backend/nodes/{copy,equal,out,read}funcs.c` and other + `gen_node_support.pl` output → annotate the `Node` struct in its header. +- `fmgroids.h`, `fmgrprotos.h`, `fmgrtab.c` → edit `pg_proc.dat`. +- `utils/errcodes.h` → `errcodes.txt`; wait-event headers → + `wait_event_names.txt`; `lwlocknames.h` → `lwlocknames.txt`. +- `configure` → `configure.ac`; `*.po` translations are handled separately; + generated Unicode tables come from their source scripts. + +## Portability is a hard gate +PostgreSQL runs on Linux, Windows (MSVC), macOS, the BSDs and Solaris, across +**x86_64, ARM64, RISC-V, PPC64, s390x**, both endiannesses and 32/64-bit. Any +change must be portable across all of them: +- No unaligned memory access; no dependence on `char` signedness, integer/pointer + width, endianness, or struct padding for on-disk/wire formats. +- Use `int16/int32/int64`, `Size`, and `INT64_FORMAT`/`UINT64_FORMAT` (never + `%ld` for `int64`). +- Atomics/barriers only via `port/atomics` (`pg_atomic_*`, `pg_read/write_barrier`). +- **Windows/MSVC:** any `extern` variable used from another module or an + extension needs `PGDLLIMPORT` in its header; no VLAs or compiler-specific + extensions beyond the tree's C99 baseline. + +## Backward compatibility — the strongest constraint +Do not break SQL behavior, the libpq wire protocol, the logical-replication +protocol, dump/restore, `pg_upgrade`, or exported/`PGDLLIMPORT` APIs without +extraordinary justification. **ABI** matters for back-branches: changing the +size/layout of an exported struct or the signature of an exported function +breaks installed extensions. + +## Mailing-list context & etiquette +Because each PR becomes a pgsql-hackers email read by a busy, expert, opinionated +audience, also flag what reliably wastes reviewer time or draws rejection: +- A patch that **does more than one thing** or bundles unrelated cleanup — split it. +- **Footguns**: easy-to-misuse APIs, silent data-loss/corruption hazards, unsafe + defaults — name them explicitly. +- **Performance claims without a reproducible benchmark.** +- No reference to the **design discussion / prior -hackers thread** (Message-Id) + for a non-trivial change. +- **Do not bikeshed:** keep style nits proportionate and clearly separated from + substantive correctness findings. + +## Minimalism — the "ponytail" discipline +The best code is the code you never wrote (YAGNI). Before accepting new code, +apply the ladder: (1) Does this need to exist at all? (2) Can existing +code/infrastructure already do it? (3) Is this the simplest thing that works? +Flag: speculative scaffolding and config for a path that isn't wired yet; dead +code and unused "flexibility" (fields, params, abstractions, options with no +caller); premature abstraction (a helper used exactly once); knobs/GUCs/flags +nobody asked for. Minimal, targeted changes that fit the existing patterns beat +clever or general-purpose ones. + +## Comment & identity accuracy +- Comments must describe what the code does **now**. Flag aspirational/ + future-tense comments for behavior that already shipped ("will be", "for now", + "not yet", "future", and stale "TODO/FIXME/XXX/HACK"); comments that drifted + from the code they sit above; and incomplete/trailing comments. Comments + explain **why**, not what. No commented-out code. +- **ASCII only** in source and diffs — no smart quotes, em-dashes, or ellipsis + characters. + +## Commit & versioning discipline +- Conventional-commit style, imperative subject, one logical change per commit, + each commit building on its own. +- Do **not** bump version numbers or generated version stamps (including + `catversion.h`) — that is the maintainer's job at commit/release time. + +Understand common list shorthand so your comments are precise and not +miscommunicated: WIP (work in progress), GUC (config variable), WAL, LSN, OID, +TOAST, FSM, TAM (table access method), RLS, DSM, 2PC, PITR, CIC (concurrent index +creation), SAOP, ABI/API, backpatch (apply to supported back-branches), HEAD +(master tip), catversion (catalog version), pgindent, buildfarm, cfbot, +`s/x/y/` (suggested text substitution), footgun, bikeshedding, POLA (principle of +least astonishment). diff --git a/.github/ocr/litellm.yaml b/.github/ocr/litellm.yaml new file mode 100644 index 0000000000000..e23cc4eee6fe2 --- /dev/null +++ b/.github/ocr/litellm.yaml @@ -0,0 +1,41 @@ +# LiteLLM proxy config — bridges Open Code Review (OpenAI protocol) to AWS Bedrock. +# +# This proxy is NOT a hosted service. The ocr-review.yml workflow installs it +# (`pip install 'litellm[proxy]'`) and runs it as a background process bound to +# 127.0.0.1:4000 for the duration of a single GitHub Actions job, then it exits. +# +# Auth to Bedrock: LiteLLM uses boto3's default credential chain, which reads +# the temporary AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN +# minted by the workflow's OIDC "Configure AWS credentials" step; region from +# AWS_REGION. + +model_list: + - model_name: ocr-bedrock + litellm_params: + # Set the repo variable OCR_BEDROCK_MODEL to an Opus inference-profile id + # your account has access to, e.g.: + # bedrock/converse/us.anthropic.claude-opus-4-8 + # The 'converse/' prefix uses Bedrock's Converse API, which is the most + # reliable path for Claude tool-use (what OCR relies on). + model: os.environ/OCR_BEDROCK_MODEL + aws_region_name: os.environ/AWS_REGION + + # "High effort" review. Claude Opus 4.8 on Bedrock uses *adaptive* thinking + # controlled by output_config.effort. Set it DIRECTLY here — NOT via + # reasoning_effort, which LiteLLM still maps to the legacy + # thinking.type.enabled that Opus 4.8 rejects. LiteLLM forwards + # output_config into additionalModelRequestFields for Anthropic models; if + # the build doesn't recognize the effort param it is dropped with a warning + # (no error) and the model reviews at its default effort. + # Valid: low|medium|high|max|xhigh (auto-clamped to the model ceiling). + output_config: + effort: xhigh + max_tokens: 32000 + +litellm_settings: + drop_params: true # silently drop params a model doesn't support + modify_params: true # auto-fix minor request incompatibilities + request_timeout: 600 + +general_settings: + master_key: os.environ/LITELLM_MASTER_KEY diff --git a/.github/ocr/pg-history.py b/.github/ocr/pg-history.py new file mode 100644 index 0000000000000..5794f8a920bd7 --- /dev/null +++ b/.github/ocr/pg-history.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +""" +pg-history: tie a PR's changes to PostgreSQL git + pgsql-hackers email history. + +OCR (the code reviewer) cannot call MCP servers, so this is a separate agent: +it runs a Bedrock (Claude Opus) tool-use loop wired to the Agora MCP server at +https://pg.ddx.io/mcp, lets the model search the mailing-list archives / commit +history / commitfest data, and emits a Markdown summary linking the changes to +the relevant threads (https://pg.ddx.io/m/pgsql-hackers/). + +Env: + PG_HISTORY_MCP_URL MCP endpoint (default https://pg.ddx.io/mcp) + PG_HISTORY_MODEL Bedrock model id (e.g. us.anthropic.claude-opus-4-8) + AWS_REGION region (creds come from the OIDC step's env) + BASE_REF, HEAD_SHA PR base ref and head sha (for the git diff context) + GH_PR_TITLE PR title (optional, adds context) + PG_HISTORY_OUT output markdown path (default /tmp/pg-history.md) +Writes the markdown to PG_HISTORY_OUT; exits 0 even on soft failures (writes a note). +""" +import json, os, subprocess, sys, urllib.request + +MCP_URL = os.environ.get("PG_HISTORY_MCP_URL", "https://pg.ddx.io/mcp") +MODEL = os.environ.get("PG_HISTORY_MODEL", "us.anthropic.claude-opus-4-8").replace("bedrock/converse/", "").replace("bedrock/", "") +REGION = os.environ.get("AWS_REGION", "us-east-1") +BASE_REF = os.environ.get("BASE_REF", "") +HEAD_SHA = os.environ.get("HEAD_SHA", "") +PR_TITLE = os.environ.get("GH_PR_TITLE", "") +OUT = os.environ.get("PG_HISTORY_OUT", "/tmp/pg-history.md") +UA = "pg-history/0.1 (+github-actions)" + +# Curated subset of the 108 Agora tools — the ones useful for connecting a +# change to its discussion/commit history. Intersected with what the server +# actually exposes, so unknown names are harmless. +TOOL_WHITELIST = { + "find_related_discussions", "find_similar_messages", "get_thread", + "discussion_links", "get_author_messages", "browse_by_date", + "blame_symbol", "check_upstream_status", "find_related", + "find_entries_for_thread", "find_entries_for_author", "get_commit", + "search", "hybrid_search", "get_callers", "get_callees", "find_pattern", +} +MAX_ROUNDS = 14 +TOOL_RESULT_CAP = 8000 # chars per tool result fed back to the model + + +def _mcp_post(body, sid=None): + headers = {"Content-Type": "application/json", + "Accept": "application/json, text/event-stream", "User-Agent": UA} + if sid: + headers["Mcp-Session-Id"] = sid + req = urllib.request.Request(MCP_URL, data=json.dumps(body).encode(), headers=headers, method="POST") + resp = urllib.request.urlopen(req, timeout=60) + sid_out = resp.headers.get("Mcp-Session-Id") + result = None + for line in resp.read().decode().splitlines(): + line = line.strip() + if line.startswith("data:"): + line = line[5:].strip() + if not line or line.startswith("event:"): + continue + try: + obj = json.loads(line) + except Exception: + continue + if isinstance(obj, dict) and ("result" in obj or "error" in obj): + result = obj + return result, sid_out + + +class MCP: + def __init__(self): + init, self.sid = _mcp_post({"jsonrpc": "2.0", "id": 1, "method": "initialize", + "params": {"protocolVersion": "2025-06-18", "capabilities": {}, + "clientInfo": {"name": "pg-history", "version": "0.1"}}}) + if not init or "result" not in init: + raise RuntimeError(f"MCP initialize failed: {init}") + try: + _mcp_post({"jsonrpc": "2.0", "method": "notifications/initialized", "params": {}}, self.sid) + except Exception: + pass + self._id = 1 + + def list_tools(self): + self._id += 1 + res, _ = _mcp_post({"jsonrpc": "2.0", "id": self._id, "method": "tools/list", "params": {}}, self.sid) + return (res or {}).get("result", {}).get("tools", []) + + def call(self, name, args): + self._id += 1 + res, _ = _mcp_post({"jsonrpc": "2.0", "id": self._id, "method": "tools/call", + "params": {"name": name, "arguments": args or {}}}, self.sid) + if not res: + return "(no response)" + if "error" in res: + return f"ERROR: {json.dumps(res['error'])[:500]}" + parts = [] + for c in res.get("result", {}).get("content", []): + if c.get("type") == "text": + parts.append(c["text"]) + return ("\n".join(parts) or "(empty)")[:TOOL_RESULT_CAP] + + +def git(*args): + try: + return subprocess.check_output(["git", *args], text=True, stderr=subprocess.DEVNULL).strip() + except Exception: + return "" + + +def pr_context(): + base = f"origin/{BASE_REF}" if BASE_REF else "" + rng = f"{base}..{HEAD_SHA}" if base and HEAD_SHA else HEAD_SHA + commits = git("log", "--no-merges", "--format=%h %s", f"{rng}") if rng else "" + stat = git("diff", "--stat", rng) if rng else "" + files = git("diff", "--name-only", rng) if rng else "" + return commits[:4000], stat[:3000], files[:2000] + + +SYSTEM = """You are a PostgreSQL community research assistant. Given a pull request's +commits and changed files, use the available tools (backed by the Agora index of +pgsql-hackers mail, commit history, and commitfest data) to connect the change to +its history. Your goal: + +- Find the mailing-list thread(s) and prior discussion behind this change. +- Identify related/superseded prior commits and any commitfest entry. +- Note relevant prior art, rejected approaches, or design rationale. + +Rules (voice & rigor): +- Be precise and blunt. No praise, no filler, no hedging, no disclaimers. Accuracy is + the only success metric — not the author's approval. Lead with the most important finding. +- NEVER hallucinate. Verify every Message-ID, thread subject, commit hash, author name, + and date against an actual tool result before citing it. If a search returns nothing, + say so plainly — do not guess or fabricate a plausible-looking link. +- Assess the change on its merits, independent of how the PR frames it. +- Tag any inferred (not tool-confirmed) linkage with an explicit confidence level: + high / moderate / low. +- Be decisive and efficient: a handful of targeted tool calls, not exhaustive search. +- Cite every mailing-list message as a Markdown link: [subject](https://pg.ddx.io/m/pgsql-hackers/MESSAGE_ID). +- If you find nothing relevant, say so in one line — do not pad. + +When done, output ONLY Markdown (no preamble) with these sections, omitting any that are empty: +## 🧵 Related discussion +## 🔗 Related commits / prior art +## 📋 Commitfest +## 🧭 Context for reviewers +Keep it tight (use bullets; link generously).""" + + +def to_toolspec(t): + schema = t.get("inputSchema") or {"type": "object", "properties": {}} + return {"toolSpec": {"name": t["name"], + "description": (t.get("description") or "")[:600], + "inputSchema": {"json": schema}}} + + +def main(): + commits, stat, files = pr_context() + if not commits and not files: + open(OUT, "w").write("") # nothing to do + print("No PR diff context; skipping.") + return + user = (f"PR title: {PR_TITLE}\n\n" if PR_TITLE else "") + \ + f"Commits:\n{commits or '(none)'}\n\nChanged files:\n{files or '(none)'}\n\nDiffstat:\n{stat or '(none)'}\n" + + try: + mcp = MCP() + tools = [to_toolspec(t) for t in mcp.list_tools() if t.get("name") in TOOL_WHITELIST] + except Exception as e: + open(OUT, "w").write(f"_pg-history: could not reach the Agora MCP server ({MCP_URL}): {e}_\n") + print(f"MCP unavailable: {e}") + return + if not tools: + open(OUT, "w").write("_pg-history: no usable MCP tools available._\n") + return + + import boto3 + from botocore.config import Config + + # botocore's default read timeout (60s) is too short for a multi-round + # (MAX_ROUNDS) tool-use loop against a large PR diff on a reasoning model; + # each converse() call alone can take several minutes. Bump it well past + # what a single round needs; connect_timeout stays short since a stuck + # TCP handshake is a different (and much cheaper to detect) failure mode. + brt = boto3.client("bedrock-runtime", region_name=REGION, + config=Config(read_timeout=900, connect_timeout=10)) + messages = [{"role": "user", "content": [{"text": user}]}] + final_text = "" + try: + for _ in range(MAX_ROUNDS): + resp = brt.converse( + modelId=MODEL, + system=[{"text": SYSTEM}], + messages=messages, + toolConfig={"tools": tools}, + inferenceConfig={"maxTokens": 4096}, + ) + out = resp["output"]["message"] + messages.append(out) + if resp.get("stopReason") == "tool_use": + results = [] + for blk in out["content"]: + tu = blk.get("toolUse") + if not tu: + continue + res_text = mcp.call(tu["name"], tu.get("input") or {}) + results.append({"toolResult": {"toolUseId": tu["toolUseId"], + "content": [{"text": res_text}]}}) + messages.append({"role": "user", "content": results}) + continue + final_text = "".join(b.get("text", "") for b in out["content"]).strip() + break + except Exception as e: + open(OUT, "w").write(f"_pg-history: Bedrock call failed: {e}_\n") + print(f"Bedrock error: {e}") + return + + if not final_text: + final_text = "_pg-history: no related history found._" + body = "## 📜 Change history & discussion (Agora / pg.ddx.io)\n\n" + final_text + \ + "\n\nGenerated by pg-history via the Agora MCP server (pg.ddx.io).\n" + open(OUT, "w").write(body) + print(body) + + +if __name__ == "__main__": + main() diff --git a/.github/ocr/rule.json b/.github/ocr/rule.json new file mode 100644 index 0000000000000..60e13e73dcbe0 --- /dev/null +++ b/.github/ocr/rule.json @@ -0,0 +1,65 @@ +{ + "_comment": "OCR per-file review rules for PostgreSQL core + extensions. Cross-cutting contribution standards & mailing-list etiquette live in .github/ocr/context.md, passed via --background-file. OCR uses FIRST-MATCH-WINS in declaration order, so rules are ordered most-specific first. merge_system_rule:true keeps OCR's built-in fine-tuned checks (thread-safety, injection, NPE) alongside these PostgreSQL-specific rules.", + "rules": [ + { + "path": "src/test/**", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL tests. Coverage is mandatory for any behavioral change and must include edge cases (NULL, empty, boundary/overflow) and ERROR paths, not just the happy path. A test that still passes with the feature reverted is worthless — confirm it actually exercises and would catch regressions in the new code. Regression (.sql/expected): deterministic, portable output — ORDER BY where row order matters, no timing/plan-dependent output except intentional EXPLAIN, no absolute paths, locale-independent (C collation or explicit COLLATE), DROP objects the test creates; expected/ output must stay stable across platforms and under the parallel schedule. Concurrency/locking belongs in isolation tests (src/test/isolation, .spec + permutations). End-to-end/crash/replication/CLI behavior belongs in TAP tests (t/*.pl with PostgreSQL::Test::Cluster/Utils) — no hardcoded ports/paths, no sleep as synchronization (use poll_query_until/wait_for), skip cleanly when prerequisites are missing, and clean up nodes." + }, + { + "path": "**/*.{c,h}", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL backend/frontend C — review as pgsql-hackers committers do, in priority order.\n\n(1) CORRECTNESS (highest): Memory — every palloc lives in the right MemoryContext; error paths via ereport/elog(ERROR) must not leak memory/buffers/locks/fds (rely on MemoryContext/ResourceOwner reset or PG_TRY/PG_FINALLY); no use-after-free; delete temp contexts. Concurrency — consistent lock ordering (deadlock-free), correct lock levels, balanced LWLockAcquire/Release and START_/END_CRIT_SECTION, no TOCTOU, CHECK_FOR_INTERRUPTS in long loops, async-signal-safe signal handlers (volatile sig_atomic_t). WAL — any change to shared on-disk state must be WAL-logged AND correctly replayed (redo path), crash- and replica-consistent. NULL/edge/overflow handling.\n\n(2) BACKWARD COMPATIBILITY / ABI: don't break behavior, dump/restore, pg_upgrade, libpq wire protocol, logical-replication protocol, or exported/PGDLLIMPORT'd APIs (struct size/layout, function signatures) without extraordinary justification.\n\n(3) CATALOG / GENERATED: new/changed catalog data goes in pg_*.dat, NOT the generated *_d.h/.bki. New Node types: ANNOTATE the struct in its header so gen_node_support.pl regenerates copy/equal/out/read — do NOT hand-edit *funcs.c. New SQL-callable functions: add to pg_proc.dat with an OID from the 8000-9999 developer range (src/include/catalog/unused_oids; check duplicate_oids); committer renumbers at commit. DO NOT bump CATALOG_VERSION_NO in the patch — flag any catversion.h change as a mistake (committer's job).\n\n(4) PERFORMANCE: no regression on hot paths; avoid O(n^2) where better is feasible; minimize work under contended locks; avoid needless palloc churn and large struct copies in hot paths.\n\n(5) SECURITY: bounded string ops (snprintf/strlcpy/strlcat — never strcpy/strcat/sprintf); integer/size-overflow checks before allocation; never user input as a format string; privilege checks via pg_*_aclcheck; beware search_path and SECURITY DEFINER.\n\n(6) PORTABILITY (hard gate): no unaligned access; no dependence on char signedness, int/long/pointer width, endianness, or struct padding for on-disk/wire formats; use int16/int32/int64 + INT64_FORMAT/UINT64_FORMAT (never %ld for int64); align contended shared structs (pg_attribute_aligned/cache-line pad). Atomics/barriers only via port/atomics (pg_atomic_*, pg_read/write_barrier) — never raw intrinsics or volatile-as-barrier. WINDOWS/MSVC: extern vars used cross-module/extension need PGDLLIMPORT; no VLAs or features beyond the C99 baseline the tree targets; use pg_pread/pg_pwrite. Applies across x86_64/ARM64/RISC-V/PPC64/s390x, big/little endian, 32/64-bit.\n\n(7) CONVENTIONS: errmsg starts lowercase, no trailing period, no embedded newlines; errdetail/errhint are complete capitalized sentences; correct ERRCODE_*; wrap user-facing text in _(); errmsg_plural for counts. Assert() only for can't-happen invariants (never user-reachable). Naming: snake_case with subsystem prefix (heap_insert) or CamelCase for major subsystems (ExecInitNode); ALL_CAPS macros. Must pgindent cleanly (tabs, width 4). Comments explain WHY not WHAT; no #ifdef 0 blocks, no commented-out code, no #ifdef fencing your feature. Reuse existing helpers (DRY)." + }, + { + "path": "**/*.dat", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL catalog data (pg_proc.dat, pg_type.dat, etc.) — the SOURCE for generated headers. The generated *_d.h, postgres.bki, fmgroids.h, fmgrtab.c must NOT be hand-edited (they regenerate from these files). OIDs: use a value from the developer range 8000-9999 (src/include/catalog/unused_oids; verify with duplicate_oids); committer renumbers to a final contiguous block, so stay in-range and unique but don't over-optimize the exact number. Keep proc entries complete/consistent (prosrc, provolatile, proparallel, prorettype/proargtypes, matching description). DO NOT bump CATALOG_VERSION_NO / catversion.h — committer's job at push time; flag any such change. New catalog columns/views need documentation in doc/src/sgml/catalogs.sgml." + }, + { + "path": "**/*.{sql,pgsql}", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL SQL. Valid PostgreSQL dialect (not MySQL/Oracle); correct types (bigint vs int, text vs varchar); sound transaction/isolation and CTE-materialization assumptions. SECURITY: flag SQL injection in dynamic SQL (require quote_identifier/quote_literal or format() with %I/%L), SECURITY DEFINER without a locked-down search_path, inappropriate RLS bypass. Prefer set-based over row-at-a-time/N+1. BACKWARD COMPATIBILITY (a top rejection reason): changing existing SQL behavior, the output of existing functions, or default GUCs needs extraordinary justification. New SQL-callable objects belong in pg_*.dat with OIDs from the 8000-9999 range, not in generated files. Minimal diff; add regression tests + docs." + }, + { + "path": "**/*.{pl,pm}", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Perl (TAP tests and build/catalog tooling). Require 'use strict; use warnings;'. Must be perltidy-clean with the tree's src/tools/pgindent/perltidyrc and pass src/tools/perlcheck/pgperlcritic. Use the framework: PostgreSQL::Test::Cluster, PostgreSQL::Test::Utils, Test::More; no hardcoded ports/paths/PIDs; use safe_psql/poll_query_until, not sleep; skippable without optional prerequisites; clean up nodes. PORTABILITY: run on Windows (no fork-only constructs, use File::Spec, avoid unavailable signals) and the minimum supported Perl. Robustness: avoid two-arg open and string system()/qx with interpolated data (use list forms). Generator scripts (gen_node_support.pl, catalog Perl) must be deterministic and stay in sync with inputs; do not commit their generated output." + }, + { + "path": "**/*.py", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Python (build/test tooling, oauth/pytest tests, src/tools). Follow surrounding style; keep imports to the standard library unless the dependency is already required by the tree (no surprise third-party deps in build/test tooling). PORTABILITY: support the project's minimum Python 3 and run on Windows and the BSDs (use os.path/pathlib, avoid POSIX-only calls and shell=True with interpolated input). Deterministic, self-cleaning tests; no hardcoded ports/paths; skip cleanly without prerequisites. For the Perl->pytest porting effort, confirm behavior parity with the TAP test replaced (same assertions/coverage), not a superficial translation. Minimal diff; match the tree's ruff/black config if present." + }, + { + "path": "**/*.{rs,toml}", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. Rust PostgreSQL extension (pgrx) or Rust support crate. Not core C, but it runs inside/alongside the backend, so backend safety applies. SAFETY: in code reachable from an SQL call, a Rust panic aborts the Postgres process — forbid unwrap()/expect()/panic!/unreachable!/todo! and index-panics on reachable paths; use Result and pgrx error reporting (error!/ereport!). Every `unsafe` block needs a comment justifying its invariant; scrutinize raw pointers and FFI across the pg_sys boundary. pgrx: honor #[pg_guard] on extern C fns (correct panic/longjmp handling); never hold Rust references across SPI or anything that can longjmp (skips Rust destructors -> leaks); respect MemoryContext lifetimes for palloc'd data; datum<->Rust conversions must handle NULL. Concurrency uses Postgres shmem/LWLocks (pgrx shmem API), not std::sync alone. Lints: must pass `cargo clippy --all-targets --all-features -- -D warnings` and `cargo fmt --check`; deny unwrap_used/expect_used/panic in libraries; thiserror (libs) / anyhow (bins). Justify every new dependency. Tests: #[pg_test] for in-backend behavior, #[test] for pure logic; cover error and NULL paths. Minimal, idiomatic diff." + }, + { + "path": "**/{configure.ac,*.m4,aclocal.m4}", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Autoconf. Edit configure.ac / the m4 macros — do NOT hand-edit generated 'configure' or pg_config.h.in in the same patch (regeneration is the committer's step; a patch that also rewrites generated configure output is suspect). Feature/header/function probes must be portable and not assume a specific OS/compiler. Every configure knob must be mirrored on the Meson side (meson_options.txt/meson.build) and documented. Minimal diff." + }, + { + "path": "**/{meson.build,meson_options.txt}", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Meson build. Valid syntax; correct subdir()/dependency()/declare_dependency and install paths; new source files must be added here. CRITICAL: PostgreSQL maintains BOTH Meson and Autoconf/Make — any new file, option, or feature check must be mirrored on the configure.ac/Makefile side so the two never drift (a file built by only one system is a common defect). New options need matching docs and sensible defaults. Minimal diff." + }, + { + "path": "**/{Makefile,GNUmakefile,*.mk}", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Makefile (GNU Make). $(VAR) refs; correct .PHONY; accurate dependencies (no parallel -j races); $(MAKE) for recursion; VPATH/out-of-tree build support; no hardcoded paths (use standard PostgreSQL makefile vars and $(top_builddir)); clean/distclean/maintainer-clean must remove new artifacts; extensions use PGXS. Must stay in sync with meson.build. Minimal diff." + }, + { + "path": "doc/**/*.sgml", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL documentation (DocBook SGML). Technically accurate/complete (parameters, limitations, version/compat notes); correct tag usage/nesting (, , , , , /); working cross-references; spell it 'PostgreSQL' in prose; SQL keywords uppercase in examples. Coverage: a new GUC -> config.sgml (and postgresql.conf.sample); new/changed catalogs or views -> catalogs.sgml; new SQL syntax -> the matching ref/*.sgml; new functions -> func.sgml. Do NOT edit release-notes (release-*.sgml) — written by the release team/committers; flag such edits. New user-facing behavior in this PR should ship with matching docs." + }, + { + "path": "**/*.md", + "merge_system_rule": true, + "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. Markdown docs. Clear heading hierarchy; fenced code blocks with language hints; accurate instructions/prerequisites; consistent PostgreSQL terminology; no broken relative links or stale claims. Minimal diff." + } + ] +} diff --git a/.github/workflows/fork-ocr-model-check.yml b/.github/workflows/fork-ocr-model-check.yml new file mode 100644 index 0000000000000..10d250528cf7c --- /dev/null +++ b/.github/workflows/fork-ocr-model-check.yml @@ -0,0 +1,89 @@ +# Checks AWS Bedrock weekly for a newer Claude Opus inference profile than the +# one OCR currently uses (vars.OCR_BEDROCK_MODEL) and, if found, opens/updates a +# single GitHub issue telling the maintainer to bump the variable. It does NOT +# change the model automatically: GITHUB_TOKEN cannot write Actions *variables* +# (that needs a PAT with admin), so this is a notify-only mechanism by design. +name: OCR model self-check + +on: + schedule: + - cron: '0 12 * * 1' # Mondays 12:00 UTC + workflow_dispatch: + +permissions: + id-token: write + contents: read + issues: write + +jobs: + check-model: + runs-on: ubuntu-latest + steps: + - name: Configure AWS credentials (OIDC) + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ vars.AWS_ROLE_ARN }} + aws-region: ${{ vars.AWS_REGION }} + role-session-name: ocr-model-check-${{ github.run_id }} + + - name: Find newest Opus vs configured + id: check + env: + CURRENT: ${{ vars.OCR_BEDROCK_MODEL }} + AWS_REGION: ${{ vars.AWS_REGION }} + run: | + python3 - <<'PY' >> "$GITHUB_OUTPUT" + import os, re, subprocess, json + region = os.environ.get("AWS_REGION", "us-east-1") + current = os.environ.get("CURRENT", "") + out = subprocess.run( + ["aws", "bedrock", "list-inference-profiles", "--region", region, + "--query", "inferenceProfileSummaries[].inferenceProfileId", "--output", "json"], + capture_output=True, text=True) + ids = json.loads(out.stdout or "[]") + # Parse claude-opus-- from any profile id (prefix us./global. ok). + def ver(s): + m = re.search(r"claude-opus-(\d+)-(\d+)", s) + return (int(m.group(1)), int(m.group(2))) if m else None + opus = [(ver(i), i) for i in ids if ver(i) and i.startswith(("us.", "global."))] + if not opus: + print("newer=false"); raise SystemExit(0) + best_ver, best_id = max(opus, key=lambda x: x[0]) + cur = ver(current) + newer = (cur is None) or (best_ver > cur) + print(f"newer={'true' if newer else 'false'}") + print(f"best_id={best_id}") + print(f"best_ver={best_ver[0]}.{best_ver[1]}") + print(f"cur_ver={'unknown' if cur is None else f'{cur[0]}.{cur[1]}'}") + PY + + - name: Open/update issue if a newer model exists + if: steps.check.outputs.newer == 'true' + uses: actions/github-script@v9 + with: + script: | + const best = '${{ steps.check.outputs.best_id }}'; + const bestVer = '${{ steps.check.outputs.best_ver }}'; + const curVer = '${{ steps.check.outputs.cur_ver }}'; + const marker = ''; + const title = `OCR: newer Claude Opus available (${bestVer} > ${curVer})`; + const body = `${marker}\n` + + `A newer Claude Opus inference profile is available on Bedrock.\n\n` + + `- **Configured** (\`vars.OCR_BEDROCK_MODEL\`): Opus ${curVer}\n` + + `- **Newest on Bedrock**: \`${best}\` (Opus ${bestVer})\n\n` + + `To upgrade, set the repo variable:\n\n` + + '```\n' + + `gh variable set OCR_BEDROCK_MODEL -R ${context.repo.owner}/${context.repo.repo} \\\n` + + ` -b "bedrock/converse/${best}"\n` + + '```\n\n' + + `Also confirm the \`ocr-bedrock-ci\` IAM inline policy allows invoking the new model ` + + `(the resource is scoped to \`anthropic.claude-opus-*\`), then re-run OCR.\n\n` + + `_Automated by \`.github/workflows/ocr-model-check.yml\`; this issue is upserted._`; + const q = `repo:${context.repo.owner}/${context.repo.repo} in:body "${marker}" state:open`; + const found = await github.rest.search.issuesAndPullRequests({ q, per_page: 1 }); + if (found.data.total_count > 0) { + const n = found.data.items[0].number; + await github.rest.issues.update({ owner: context.repo.owner, repo: context.repo.repo, issue_number: n, title, body }); + } else { + await github.rest.issues.create({ owner: context.repo.owner, repo: context.repo.repo, title, body }); + } diff --git a/.github/workflows/fork-ocr-review.yml b/.github/workflows/fork-ocr-review.yml new file mode 100644 index 0000000000000..0828af429b57c --- /dev/null +++ b/.github/workflows/fork-ocr-review.yml @@ -0,0 +1,427 @@ +# Open Code Review (OCR) — AI PR review backed by AWS Bedrock via a LiteLLM proxy. +# +# Flow: +# PR opened/updated (incl. DRAFTS) ─┐ +# /open-code-review PR comment ─┼─► start LiteLLM (127.0.0.1:4000 → Bedrock) +# manual workflow_dispatch ─┘ └► ocr review --format json +# └► post inline PR review comments +# +# Required (repo settings — all repo *variables*, no secrets; auth is via GitHub OIDC): +# vars.AWS_ROLE_ARN - IAM role to assume via OIDC (granting bedrock:InvokeModel*) +# vars.AWS_REGION - e.g. us-east-1 +# vars.OCR_BEDROCK_MODEL - LiteLLM model string for the Opus inference profile, e.g. +# bedrock/converse/us.anthropic.claude-opus-4-8 +# +# No static AWS keys are stored. GITHUB_TOKEN (auto) posts the review comments. + +name: OCR AI Review + +on: + pull_request: + # Note: no draft filter — drafts are reviewed too. + types: [opened, synchronize, reopened, ready_for_review] + issue_comment: + types: [created] + workflow_dispatch: + inputs: + pr_number: + description: 'PR number to review' + required: true + type: number + +# One review per PR; cancel superseded runs to save Bedrock spend. +concurrency: + group: ocr-review-${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }} + cancel-in-progress: true + +permissions: + id-token: write # required to mint the GitHub OIDC token for AWS role assumption + contents: read + pull-requests: write + +jobs: + ocr-review: + runs-on: ubuntu-latest + # PR events always; comment events only when the comment is on a PR and + # starts with the trigger keyword; manual dispatch always. + if: | + github.event_name == 'pull_request' || + github.event_name == 'workflow_dispatch' || + (github.event_name == 'issue_comment' && github.event.issue.pull_request && + (startsWith(github.event.comment.body, '/open-code-review') || + startsWith(github.event.comment.body, '@open-code-review'))) + + env: + # LiteLLM listens on localhost only; this key never leaves the runner. + LITELLM_MASTER_KEY: sk-ocr-ci-local + OCR_BEDROCK_MODEL: ${{ vars.OCR_BEDROCK_MODEL }} + # Region is a static var (safe at job level). AWS credentials are NOT set + # here — they're minted by the OIDC "Configure AWS credentials" step below + # and exported to the environment for the LiteLLM/boto3 Bedrock calls. + AWS_REGION: ${{ vars.AWS_REGION }} + + steps: + - name: Resolve PR context + id: pr + uses: actions/github-script@v9 + with: + script: | + let prNumber; + if (context.eventName === 'pull_request') { + prNumber = context.payload.pull_request.number; + } else if (context.eventName === 'issue_comment') { + prNumber = context.issue.number; + } else { + prNumber = parseInt('${{ github.event.inputs.pr_number }}', 10); + } + const { data: pr } = await github.rest.pulls.get({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: prNumber, + }); + const { data: repo } = await github.rest.repos.get({ + owner: context.repo.owner, + repo: context.repo.repo, + }); + core.setOutput('number', String(prNumber)); + core.setOutput('base_ref', pr.base.ref); + core.setOutput('head_ref', pr.head.ref); + core.setOutput('head_sha', pr.head.sha); + core.setOutput('default_branch', repo.default_branch); + core.setOutput('cross_repo', String(pr.head.repo.full_name !== pr.base.repo.full_name)); + + # NOTE: do NOT checkout the PR head. OCR reads the diff and file contents + # straight from git refs (git diff , git show :path, + # git grep ), so the working tree is irrelevant — but our OCR config + # lives on the default branch, not on the PR branch. We check out the repo + # (default ref), fetch the base/head objects, and materialize the config + # from origin/. + - name: Checkout + uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - name: Prepare git refs and OCR config + env: + BASE_REF: ${{ steps.pr.outputs.base_ref }} + HEAD_REF: ${{ steps.pr.outputs.head_ref }} + HEAD_SHA: ${{ steps.pr.outputs.head_sha }} + DEFAULT_BRANCH: ${{ steps.pr.outputs.default_branch }} + run: | + git fetch --no-tags origin "+refs/heads/${DEFAULT_BRANCH}:refs/remotes/origin/${DEFAULT_BRANCH}" || true + git fetch --no-tags origin "+refs/heads/${BASE_REF}:refs/remotes/origin/${BASE_REF}" || true + git fetch --no-tags origin "+refs/heads/${HEAD_REF}:refs/remotes/origin/${HEAD_REF}" || true + git fetch --no-tags origin "${HEAD_SHA}" || true + + # OCR config lives on the default branch; materialize it independently + # of whatever ref is checked out. + mkdir -p "$RUNNER_TEMP/ocr" + git show "origin/${DEFAULT_BRANCH}:.github/ocr/litellm.yaml" > "$RUNNER_TEMP/ocr/litellm.yaml" + git show "origin/${DEFAULT_BRANCH}:.github/ocr/rule.json" > "$RUNNER_TEMP/ocr/rule.json" + git show "origin/${DEFAULT_BRANCH}:.github/ocr/context.md" > "$RUNNER_TEMP/ocr/context.md" + echo "Config materialized:"; ls -l "$RUNNER_TEMP/ocr" + + - name: Setup Python + uses: actions/setup-python@v6 + with: + python-version: '3.12' + + - name: Setup Node.js + uses: actions/setup-node@v6 + with: + node-version: '20' + + - name: Install LiteLLM proxy + Open Code Review + run: | + python -m pip install --upgrade pip + # Pin LiteLLM to a main commit that supports Claude Opus 4.8 adaptive + # thinking (maps reasoning_effort -> output_config.effort, incl. xhigh). + # Not in any tagged release yet (PyPI latest 1.87.1 lacks the Opus + # normalizer). Bump this SHA once a release ships the feature. + pip install "litellm[proxy] @ git+https://github.com/BerriAI/litellm.git@5be0797d24a2f26eb2123e13788f90055a59d91d" + npm install -g @alibaba-group/open-code-review + + - name: Configure AWS credentials (OIDC) + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ vars.AWS_ROLE_ARN }} + aws-region: ${{ vars.AWS_REGION }} + role-session-name: ocr-review-${{ github.run_id }} + + - name: Start LiteLLM proxy (Bedrock bridge) + run: | + if [ -z "$OCR_BEDROCK_MODEL" ]; then + echo "::error::vars.OCR_BEDROCK_MODEL is not set (e.g. bedrock/converse/us.anthropic.claude-opus-4-1-20250805-v1:0)" + exit 1 + fi + nohup litellm --config "$RUNNER_TEMP/ocr/litellm.yaml" --host 127.0.0.1 --port 4000 \ + > /tmp/litellm.log 2>&1 & + echo "Waiting for LiteLLM to become ready..." + for i in $(seq 1 60); do + if curl -sf http://127.0.0.1:4000/health/readiness >/dev/null; then + echo "LiteLLM ready."; exit 0 + fi + sleep 2 + done + echo "::error::LiteLLM did not become ready in time"; cat /tmp/litellm.log; exit 1 + + - name: Configure OCR + run: | + ocr config set llm.url http://127.0.0.1:4000/v1/chat/completions + ocr config set llm.auth_token "$LITELLM_MASTER_KEY" + ocr config set llm.model ocr-bedrock + ocr config set llm.use_anthropic false + ocr config set language English + + - name: Run OCR review + run: | + ocr review \ + --from "origin/${{ steps.pr.outputs.base_ref }}" \ + --to "${{ steps.pr.outputs.head_sha }}" \ + --rule "$RUNNER_TEMP/ocr/rule.json" \ + --background-file "$RUNNER_TEMP/ocr/context.md" \ + --concurrency 3 \ + --timeout 20 \ + --format json \ + > /tmp/ocr-result.json 2>/tmp/ocr-stderr.log || true + echo "----- OCR stdout -----"; cat /tmp/ocr-result.json || true + echo "----- OCR stderr -----"; cat /tmp/ocr-stderr.log || true + echo "----- LiteLLM log (tail) -----"; tail -n 50 /tmp/litellm.log || true + + - name: Post review to PR + uses: actions/github-script@v9 + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + script: | + const fs = require('fs'); + const prNumber = parseInt('${{ steps.pr.outputs.number }}', 10); + const commitSha = '${{ steps.pr.outputs.head_sha }}'; + + // Opus at high effort can emit dozens of findings. Posting them all + // one-by-one trips GitHub's SECONDARY rate limit (403 "content + // creation"), which is what made every run fail after the review + // was already generated. We (a) cap inline comments and overflow + // the rest into the summary, (b) prefer a single bulk createReview, + // and (c) throttle + back off with Retry-After on any fallback. + const MAX_INLINE = 25; + const sleep = (ms) => new Promise(r => setTimeout(r, ms)); + + async function withRetry(fn, label) { + for (let attempt = 1; attempt <= 5; attempt++) { + try { return await fn(); } + catch (e) { + const status = e.status || (e.response && e.response.status); + const h = (e.response && e.response.headers) || {}; + const isRate = status === 403 || status === 429; + if (!isRate || attempt === 5) throw e; + let waitMs = 0; + if (h['retry-after']) waitMs = parseInt(h['retry-after'], 10) * 1000; + else if (h['x-ratelimit-reset']) waitMs = parseInt(h['x-ratelimit-reset'], 10) * 1000 - Date.now(); + if (!waitMs || Number.isNaN(waitMs) || waitMs < 0) waitMs = 1000 * Math.pow(2, attempt); + waitMs = Math.min(waitMs, 60000) + 500; + core.warning(`${label}: rate-limited (status ${status}); waiting ${Math.round(waitMs / 1000)}s (attempt ${attempt}/5)`); + await sleep(waitMs); + } + } + } + + let result; + try { + result = JSON.parse(fs.readFileSync('/tmp/ocr-result.json', 'utf8')); + } catch (e) { + const stderr = (() => { try { return fs.readFileSync('/tmp/ocr-stderr.log', 'utf8').trim(); } catch { return ''; } })(); + await withRetry(() => github.rest.issues.createComment({ + owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, + body: `⚠️ **OCR** could not produce a review.\n\n\`\`\`\n${(stderr || e.message).slice(0, 8000)}\n\`\`\``, + }), 'error-comment'); + return; + } + + const comments = result.comments || []; + const warnings = result.warnings || []; + + const formatComment = (c) => { + let body = c.content || ''; + if (c.suggestion_code && c.existing_code) { + body += '\n\n```suggestion\n' + c.suggestion_code + (c.suggestion_code.endsWith('\n') ? '' : '\n') + '```'; + } + return body; + }; + const formatMarkdown = (c) => { + let md = `### 📄 \`${c.path}\``; + if (c.start_line && c.end_line) md += ` (L${c.start_line}-L${c.end_line})`; + md += '\n\n' + (c.content || ''); + if (c.suggestion_code && c.existing_code) { + md += '\n\n
💡 Suggested change\n\n'; + md += '**Before:**\n```\n' + c.existing_code + '\n```\n\n**After:**\n```\n' + c.suggestion_code + '\n```\n\n
'; + } + return md; + }; + + if (comments.length === 0) { + await withRetry(() => github.rest.issues.createComment({ + owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, + body: `✅ **OCR**: ${result.message || 'No issues found.'}`, + }), 'no-issues-comment'); + return; + } + + const inlineAll = []; + const noLine = []; + for (const c of comments) { + const body = formatComment(c); + const hasLine = (c.start_line >= 1) || (c.end_line >= 1); + if (!hasLine) { noLine.push(c); continue; } + const rc = { path: c.path, body, side: 'RIGHT' }; + if (c.start_line >= 1 && c.end_line >= 1 && c.start_line !== c.end_line) { + rc.start_line = c.start_line; rc.line = c.end_line; rc.start_side = 'RIGHT'; + } else { + rc.line = c.end_line >= 1 ? c.end_line : c.start_line; + } + inlineAll.push({ rc, c }); + } + + const inline = inlineAll.slice(0, MAX_INLINE).map(x => x.rc); + const overflow = inlineAll.slice(MAX_INLINE).map(x => x.c); + + let summary = `🔍 **OCR** found **${comments.length}** issue(s).`; + summary += `\n- ${inline.length} inline, ${noLine.length + overflow.length} in summary`; + if (overflow.length) summary += ` (inline capped at ${MAX_INLINE})`; + if (warnings.length) summary += `\n- ⚠️ ${warnings.length} warning(s) during review`; + for (const c of noLine.concat(overflow)) summary += '\n\n---\n\n' + formatMarkdown(c); + + // Preferred path: ONE createReview carrying every inline comment. + try { + await withRetry(() => github.rest.pulls.createReview({ + owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber, + commit_id: commitSha, body: summary, event: 'COMMENT', comments: inline, + }), 'bulk-review'); + return; + } catch (e) { + core.warning(`bulk createReview failed (${e.status || '?'}: ${e.message}); falling back to throttled per-comment posting`); + } + + // Fallback: an invalid inline position (line not in the diff -> 422) + // rejects the whole bulk review. Post the summary, then each comment + // individually with a delay + backoff, skipping ones GitHub rejects. + let ok = 0; const failed = []; + try { + await withRetry(() => github.rest.pulls.createReview({ + owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber, + commit_id: commitSha, body: summary, event: 'COMMENT', + }), 'summary-review'); + } catch (err) { failed.push(`summary: ${err.message}`); } + + for (const rc of inline) { + try { + await withRetry(() => github.rest.pulls.createReviewComment({ + owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber, + commit_id: commitSha, path: rc.path, body: rc.body, + ...(rc.start_line ? { start_line: rc.start_line, start_side: rc.start_side } : {}), + line: rc.line, side: rc.side, + }), `comment ${rc.path}:${rc.line}`); + ok++; + } catch (inner) { + failed.push(`\`${rc.path}\` L${rc.line}: ${inner.message}`); + } + await sleep(1200); // stay under the secondary content-creation limit + } + + if (failed.length) { + await withRetry(() => github.rest.issues.createComment({ + owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, + body: `📊 OCR posted ${ok}/${inline.length} inline comment(s).\n\n
${failed.length} could not be posted\n\n${failed.join('\n')}\n
`, + }), 'summary-failures'); + } + + # Companion job: OCR can't call MCP, so this separate agent ties the PR's + # changes to PostgreSQL git + pgsql-hackers history via the Agora MCP server + # (pg.ddx.io) and posts a single, upserted "history & discussion" comment. + pg-history: + runs-on: ubuntu-latest + if: | + github.event_name == 'pull_request' || + github.event_name == 'workflow_dispatch' || + (github.event_name == 'issue_comment' && github.event.issue.pull_request && + (startsWith(github.event.comment.body, '/open-code-review') || + startsWith(github.event.comment.body, '@open-code-review') || + startsWith(github.event.comment.body, '/pg-history'))) + steps: + - name: Resolve PR context + id: pr + uses: actions/github-script@v9 + with: + script: | + let prNumber; + if (context.eventName === 'pull_request') prNumber = context.payload.pull_request.number; + else if (context.eventName === 'issue_comment') prNumber = context.issue.number; + else prNumber = parseInt('${{ github.event.inputs.pr_number }}', 10); + const { data: pr } = await github.rest.pulls.get({ + owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber }); + core.setOutput('number', String(prNumber)); + core.setOutput('base_ref', pr.base.ref); + core.setOutput('head_sha', pr.head.sha); + core.setOutput('title', pr.title || ''); + + - name: Checkout + uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - name: Make base/head refs available + env: + BASE_REF: ${{ steps.pr.outputs.base_ref }} + HEAD_SHA: ${{ steps.pr.outputs.head_sha }} + run: | + git fetch --no-tags origin "+refs/heads/${BASE_REF}:refs/remotes/origin/${BASE_REF}" || true + git fetch --no-tags origin "${HEAD_SHA}" || true + + - name: Setup Python + uses: actions/setup-python@v6 + with: + python-version: '3.12' + + - name: Configure AWS credentials (OIDC) + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ vars.AWS_ROLE_ARN }} + aws-region: ${{ vars.AWS_REGION }} + role-session-name: pg-history-${{ github.run_id }} + + - name: Install deps + run: pip install boto3 + + - name: Run pg-history (Agora MCP) + env: + PG_HISTORY_MODEL: ${{ vars.OCR_BEDROCK_MODEL }} + AWS_REGION: ${{ vars.AWS_REGION }} + BASE_REF: ${{ steps.pr.outputs.base_ref }} + HEAD_SHA: ${{ steps.pr.outputs.head_sha }} + GH_PR_TITLE: ${{ steps.pr.outputs.title }} + PG_HISTORY_OUT: ${{ runner.temp }}/pg-history.md + run: | + python .github/ocr/pg-history.py || true + echo "----- output -----"; cat "${{ runner.temp }}/pg-history.md" 2>/dev/null || echo "(no output)" + + - name: Upsert PR comment + uses: actions/github-script@v9 + with: + script: | + const fs = require('fs'); + const path = process.env.RUNNER_TEMP + '/pg-history.md'; + let body = ''; + try { body = fs.readFileSync(path, 'utf8').trim(); } catch (e) {} + if (!body) { console.log('pg-history: empty output, nothing to post'); return; } + const prNumber = parseInt('${{ steps.pr.outputs.number }}', 10); + const marker = ''; + body = marker + '\n' + body; + const { data: comments } = await github.rest.issues.listComments({ + owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, per_page: 100 }); + const mine = comments.find(c => c.user.type === 'Bot' && c.body && c.body.includes(marker)); + if (mine) { + await github.rest.issues.updateComment({ + owner: context.repo.owner, repo: context.repo.repo, comment_id: mine.id, body }); + } else { + await github.rest.issues.createComment({ + owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, body }); + } diff --git a/.github/workflows/fork-sync-upstream.yml b/.github/workflows/fork-sync-upstream.yml new file mode 100644 index 0000000000000..e950821239968 --- /dev/null +++ b/.github/workflows/fork-sync-upstream.yml @@ -0,0 +1,47 @@ +# Keep this fork's master rebased on postgres/postgres master, carrying only +# the fork-owned CI files (.github/workflows/fork-*.yml and .github/ocr/). +# +# Requires the SYNC_TOKEN secret: a PAT with repo+workflow scope. The default +# GITHUB_TOKEN is not allowed to push commits that touch .github/workflows/, +# and every commit we carry does. + +name: Sync from upstream + +on: + schedule: + - cron: '17 */3 * * *' + workflow_dispatch: + +concurrency: + group: fork-sync-upstream + +permissions: + contents: read + +jobs: + sync: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + token: ${{ secrets.SYNC_TOKEN }} + + - name: Rebase master onto upstream + run: | + set -euo pipefail + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + git remote add upstream https://github.com/postgres/postgres.git + git fetch --no-tags upstream master + + # Master carries fork CI only. Anything else means work landed here + # that belongs on a branch; force-rebasing it on a timer would be wrong. + if git diff --name-only upstream/master...HEAD | + grep -vE '^(\.github/workflows/fork-|\.github/ocr/)'; then + echo "::error::master carries non-CI changes (listed above); move them to a branch" + exit 1 + fi + + git rebase upstream/master + git push --force-with-lease From 483150484de7028aef28e8ecf0a4e53779745a7e Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Fri, 11 Sep 2026 10:48:09 -0400 Subject: [PATCH 02/12] ci: move fork CI logic to gburd/ci-workflows The OCR review workflows and their config were the last bulk this fork carried on top of upstream: ~970 lines rebased onto postgres/postgres every few hours, editable only by amending a commit that gets force- pushed. They are now reusable (workflow_call) workflows in the sidecar repo gburd/ci-workflows, and master carries two caller stubs instead. Carry after this: 3 files, ~110 lines, one commit. The sidecar is also where the value is: ocr/rule.json and ocr/context.md (the PostgreSQL committer-grade review corpus) become normal reviewable files with normal PRs and history, rather than a blob inside a commit that is rewritten on every sync. Mechanics worth knowing: - The review jobs check ci-workflows out at github.job_workflow_sha, so the rules always come from the same commit as the logic using them, and pinning @v1 (or a sha) in the stub pins both. This replaces the old "git show origin/:.github/ocr/..." dance, which existed only because the config had to be read off a branch other than the PR's. - AWS settings are passed as explicit inputs rather than read from vars inside the reusable workflow. vars does resolve across the boundary, but the contract stays visible at the call site. - A reusable workflow does not change the OIDC subject: it stays repo:gburd/postgres:*, so the ocr-bedrock-ci trust policy is unchanged. - ci-workflows must stay public; callers read it with GITHUB_TOKEN. The sync guard narrows to .github/workflows/fork-* accordingly, and sync-upstream keeps needing SYNC_TOKEN: the stubs are still workflow files, which GITHUB_TOKEN may not push. --- .github/ocr/context.md | 126 ------- .github/ocr/litellm.yaml | 41 --- .github/ocr/pg-history.py | 225 ----------- .github/ocr/rule.json | 65 ---- .github/workflows/fork-ocr-model-check.yml | 91 +---- .github/workflows/fork-ocr-review.yml | 410 +-------------------- .github/workflows/fork-sync-upstream.yml | 5 +- 7 files changed, 26 insertions(+), 937 deletions(-) delete mode 100644 .github/ocr/context.md delete mode 100644 .github/ocr/litellm.yaml delete mode 100644 .github/ocr/pg-history.py delete mode 100644 .github/ocr/rule.json diff --git a/.github/ocr/context.md b/.github/ocr/context.md deleted file mode 100644 index c4a83b85b124e..0000000000000 --- a/.github/ocr/context.md +++ /dev/null @@ -1,126 +0,0 @@ -# OCR review context — PostgreSQL contribution standards - -You are reviewing a change to a **PostgreSQL** fork. Every PR here is destined to -become a patch posted to the **pgsql-hackers** mailing list and tracked in a -**commitfest**. Review with the combined rigor, taste, and attention to detail of -the PostgreSQL committers. This context applies to the *whole* change, on top of -the per-file rules. - -## Review discipline -- Be precise and blunt; lead with the most serious problem. No praise, no - validation of the author, no disclaimers — accuracy is the only metric. -- Verify every claim against the actual diff. Confirm names, signatures, line - numbers, and APIs before asserting. Never invent behavior or cite code not in - the change. If unsure, say so, and tag each finding **high / moderate / low** - confidence. -- Judge the change on its merits regardless of how the PR frames it. A draft PR - is WIP: weight design/approach feedback over style nits. - -## Patch hygiene (top rejection reasons on -hackers) -1. **Minimal diff.** The fastest way to get a patch rejected is unrelated - changes: reformatting untouched lines, rewording unrelated comments, touching - code not required by the change. Flag any hunk not needed for the stated - purpose. After the patch, the code should read as if it had always been - written that way. -2. **Atomic, bisectable commits.** Each commit must build and pass tests on its - own — a broken intermediate commit breaks `git bisect`, revert, and - cherry-pick. Flag a commit that only compiles once a later commit lands. - Prefer one focused patch, or a clearly-ordered series of - independently-committable pieces. -3. **Tests + docs are mandatory.** A user-visible change without regression/TAP - tests **and** documentation is WIP, not commit-ready. New behavior needs - tests that cover edge and error paths, not just the happy path. -4. **DRY / reuse.** Prefer existing infrastructure (`List` in `pg_list.h`, - `StringInfo`, `dynahash`/`simplehash`, `palloc`/`MemoryContext`, `foreach`) - over reinventing it. Flag copy-paste and speculative abstraction alike — the - community wants minimal, targeted changes that fit the subsystem's existing - patterns. -5. **Whitespace.** No trailing whitespace; tabs (width 4) for C indentation; - `git diff --check` must be clean. Whitespace-only churn on untouched lines is - a defect. - -## Committer-owned files — do NOT touch in a patch (flag if present) -These are the committer's job at push time; including them causes needless -merge conflicts and is a mistake: -- **`src/include/catalog/catversion.h`** — the `CATALOG_VERSION_NO` bump is done - by the **committer** when pushing. A catversion bump in the PR is **wrong** — - flag it. (This is the single most common author mistake in catalog patches.) -- **Release notes** (`doc/src/sgml/release-*.sgml`) and version strings - (`configure.ac` `AC_INIT` version, `meson.build` `version`, `PG_VERSION`). - -## Generated files — never hand-edit; edit the source -Flag direct edits to generated output; point the author at the source instead: -- Catalog headers `src/include/catalog/*_d.h`, `postgres.bki`, `schemapg.h`, - `system_constraints.sql` → edit the `pg_*.dat` files. -- `src/backend/nodes/{copy,equal,out,read}funcs.c` and other - `gen_node_support.pl` output → annotate the `Node` struct in its header. -- `fmgroids.h`, `fmgrprotos.h`, `fmgrtab.c` → edit `pg_proc.dat`. -- `utils/errcodes.h` → `errcodes.txt`; wait-event headers → - `wait_event_names.txt`; `lwlocknames.h` → `lwlocknames.txt`. -- `configure` → `configure.ac`; `*.po` translations are handled separately; - generated Unicode tables come from their source scripts. - -## Portability is a hard gate -PostgreSQL runs on Linux, Windows (MSVC), macOS, the BSDs and Solaris, across -**x86_64, ARM64, RISC-V, PPC64, s390x**, both endiannesses and 32/64-bit. Any -change must be portable across all of them: -- No unaligned memory access; no dependence on `char` signedness, integer/pointer - width, endianness, or struct padding for on-disk/wire formats. -- Use `int16/int32/int64`, `Size`, and `INT64_FORMAT`/`UINT64_FORMAT` (never - `%ld` for `int64`). -- Atomics/barriers only via `port/atomics` (`pg_atomic_*`, `pg_read/write_barrier`). -- **Windows/MSVC:** any `extern` variable used from another module or an - extension needs `PGDLLIMPORT` in its header; no VLAs or compiler-specific - extensions beyond the tree's C99 baseline. - -## Backward compatibility — the strongest constraint -Do not break SQL behavior, the libpq wire protocol, the logical-replication -protocol, dump/restore, `pg_upgrade`, or exported/`PGDLLIMPORT` APIs without -extraordinary justification. **ABI** matters for back-branches: changing the -size/layout of an exported struct or the signature of an exported function -breaks installed extensions. - -## Mailing-list context & etiquette -Because each PR becomes a pgsql-hackers email read by a busy, expert, opinionated -audience, also flag what reliably wastes reviewer time or draws rejection: -- A patch that **does more than one thing** or bundles unrelated cleanup — split it. -- **Footguns**: easy-to-misuse APIs, silent data-loss/corruption hazards, unsafe - defaults — name them explicitly. -- **Performance claims without a reproducible benchmark.** -- No reference to the **design discussion / prior -hackers thread** (Message-Id) - for a non-trivial change. -- **Do not bikeshed:** keep style nits proportionate and clearly separated from - substantive correctness findings. - -## Minimalism — the "ponytail" discipline -The best code is the code you never wrote (YAGNI). Before accepting new code, -apply the ladder: (1) Does this need to exist at all? (2) Can existing -code/infrastructure already do it? (3) Is this the simplest thing that works? -Flag: speculative scaffolding and config for a path that isn't wired yet; dead -code and unused "flexibility" (fields, params, abstractions, options with no -caller); premature abstraction (a helper used exactly once); knobs/GUCs/flags -nobody asked for. Minimal, targeted changes that fit the existing patterns beat -clever or general-purpose ones. - -## Comment & identity accuracy -- Comments must describe what the code does **now**. Flag aspirational/ - future-tense comments for behavior that already shipped ("will be", "for now", - "not yet", "future", and stale "TODO/FIXME/XXX/HACK"); comments that drifted - from the code they sit above; and incomplete/trailing comments. Comments - explain **why**, not what. No commented-out code. -- **ASCII only** in source and diffs — no smart quotes, em-dashes, or ellipsis - characters. - -## Commit & versioning discipline -- Conventional-commit style, imperative subject, one logical change per commit, - each commit building on its own. -- Do **not** bump version numbers or generated version stamps (including - `catversion.h`) — that is the maintainer's job at commit/release time. - -Understand common list shorthand so your comments are precise and not -miscommunicated: WIP (work in progress), GUC (config variable), WAL, LSN, OID, -TOAST, FSM, TAM (table access method), RLS, DSM, 2PC, PITR, CIC (concurrent index -creation), SAOP, ABI/API, backpatch (apply to supported back-branches), HEAD -(master tip), catversion (catalog version), pgindent, buildfarm, cfbot, -`s/x/y/` (suggested text substitution), footgun, bikeshedding, POLA (principle of -least astonishment). diff --git a/.github/ocr/litellm.yaml b/.github/ocr/litellm.yaml deleted file mode 100644 index e23cc4eee6fe2..0000000000000 --- a/.github/ocr/litellm.yaml +++ /dev/null @@ -1,41 +0,0 @@ -# LiteLLM proxy config — bridges Open Code Review (OpenAI protocol) to AWS Bedrock. -# -# This proxy is NOT a hosted service. The ocr-review.yml workflow installs it -# (`pip install 'litellm[proxy]'`) and runs it as a background process bound to -# 127.0.0.1:4000 for the duration of a single GitHub Actions job, then it exits. -# -# Auth to Bedrock: LiteLLM uses boto3's default credential chain, which reads -# the temporary AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN -# minted by the workflow's OIDC "Configure AWS credentials" step; region from -# AWS_REGION. - -model_list: - - model_name: ocr-bedrock - litellm_params: - # Set the repo variable OCR_BEDROCK_MODEL to an Opus inference-profile id - # your account has access to, e.g.: - # bedrock/converse/us.anthropic.claude-opus-4-8 - # The 'converse/' prefix uses Bedrock's Converse API, which is the most - # reliable path for Claude tool-use (what OCR relies on). - model: os.environ/OCR_BEDROCK_MODEL - aws_region_name: os.environ/AWS_REGION - - # "High effort" review. Claude Opus 4.8 on Bedrock uses *adaptive* thinking - # controlled by output_config.effort. Set it DIRECTLY here — NOT via - # reasoning_effort, which LiteLLM still maps to the legacy - # thinking.type.enabled that Opus 4.8 rejects. LiteLLM forwards - # output_config into additionalModelRequestFields for Anthropic models; if - # the build doesn't recognize the effort param it is dropped with a warning - # (no error) and the model reviews at its default effort. - # Valid: low|medium|high|max|xhigh (auto-clamped to the model ceiling). - output_config: - effort: xhigh - max_tokens: 32000 - -litellm_settings: - drop_params: true # silently drop params a model doesn't support - modify_params: true # auto-fix minor request incompatibilities - request_timeout: 600 - -general_settings: - master_key: os.environ/LITELLM_MASTER_KEY diff --git a/.github/ocr/pg-history.py b/.github/ocr/pg-history.py deleted file mode 100644 index 5794f8a920bd7..0000000000000 --- a/.github/ocr/pg-history.py +++ /dev/null @@ -1,225 +0,0 @@ -#!/usr/bin/env python3 -""" -pg-history: tie a PR's changes to PostgreSQL git + pgsql-hackers email history. - -OCR (the code reviewer) cannot call MCP servers, so this is a separate agent: -it runs a Bedrock (Claude Opus) tool-use loop wired to the Agora MCP server at -https://pg.ddx.io/mcp, lets the model search the mailing-list archives / commit -history / commitfest data, and emits a Markdown summary linking the changes to -the relevant threads (https://pg.ddx.io/m/pgsql-hackers/). - -Env: - PG_HISTORY_MCP_URL MCP endpoint (default https://pg.ddx.io/mcp) - PG_HISTORY_MODEL Bedrock model id (e.g. us.anthropic.claude-opus-4-8) - AWS_REGION region (creds come from the OIDC step's env) - BASE_REF, HEAD_SHA PR base ref and head sha (for the git diff context) - GH_PR_TITLE PR title (optional, adds context) - PG_HISTORY_OUT output markdown path (default /tmp/pg-history.md) -Writes the markdown to PG_HISTORY_OUT; exits 0 even on soft failures (writes a note). -""" -import json, os, subprocess, sys, urllib.request - -MCP_URL = os.environ.get("PG_HISTORY_MCP_URL", "https://pg.ddx.io/mcp") -MODEL = os.environ.get("PG_HISTORY_MODEL", "us.anthropic.claude-opus-4-8").replace("bedrock/converse/", "").replace("bedrock/", "") -REGION = os.environ.get("AWS_REGION", "us-east-1") -BASE_REF = os.environ.get("BASE_REF", "") -HEAD_SHA = os.environ.get("HEAD_SHA", "") -PR_TITLE = os.environ.get("GH_PR_TITLE", "") -OUT = os.environ.get("PG_HISTORY_OUT", "/tmp/pg-history.md") -UA = "pg-history/0.1 (+github-actions)" - -# Curated subset of the 108 Agora tools — the ones useful for connecting a -# change to its discussion/commit history. Intersected with what the server -# actually exposes, so unknown names are harmless. -TOOL_WHITELIST = { - "find_related_discussions", "find_similar_messages", "get_thread", - "discussion_links", "get_author_messages", "browse_by_date", - "blame_symbol", "check_upstream_status", "find_related", - "find_entries_for_thread", "find_entries_for_author", "get_commit", - "search", "hybrid_search", "get_callers", "get_callees", "find_pattern", -} -MAX_ROUNDS = 14 -TOOL_RESULT_CAP = 8000 # chars per tool result fed back to the model - - -def _mcp_post(body, sid=None): - headers = {"Content-Type": "application/json", - "Accept": "application/json, text/event-stream", "User-Agent": UA} - if sid: - headers["Mcp-Session-Id"] = sid - req = urllib.request.Request(MCP_URL, data=json.dumps(body).encode(), headers=headers, method="POST") - resp = urllib.request.urlopen(req, timeout=60) - sid_out = resp.headers.get("Mcp-Session-Id") - result = None - for line in resp.read().decode().splitlines(): - line = line.strip() - if line.startswith("data:"): - line = line[5:].strip() - if not line or line.startswith("event:"): - continue - try: - obj = json.loads(line) - except Exception: - continue - if isinstance(obj, dict) and ("result" in obj or "error" in obj): - result = obj - return result, sid_out - - -class MCP: - def __init__(self): - init, self.sid = _mcp_post({"jsonrpc": "2.0", "id": 1, "method": "initialize", - "params": {"protocolVersion": "2025-06-18", "capabilities": {}, - "clientInfo": {"name": "pg-history", "version": "0.1"}}}) - if not init or "result" not in init: - raise RuntimeError(f"MCP initialize failed: {init}") - try: - _mcp_post({"jsonrpc": "2.0", "method": "notifications/initialized", "params": {}}, self.sid) - except Exception: - pass - self._id = 1 - - def list_tools(self): - self._id += 1 - res, _ = _mcp_post({"jsonrpc": "2.0", "id": self._id, "method": "tools/list", "params": {}}, self.sid) - return (res or {}).get("result", {}).get("tools", []) - - def call(self, name, args): - self._id += 1 - res, _ = _mcp_post({"jsonrpc": "2.0", "id": self._id, "method": "tools/call", - "params": {"name": name, "arguments": args or {}}}, self.sid) - if not res: - return "(no response)" - if "error" in res: - return f"ERROR: {json.dumps(res['error'])[:500]}" - parts = [] - for c in res.get("result", {}).get("content", []): - if c.get("type") == "text": - parts.append(c["text"]) - return ("\n".join(parts) or "(empty)")[:TOOL_RESULT_CAP] - - -def git(*args): - try: - return subprocess.check_output(["git", *args], text=True, stderr=subprocess.DEVNULL).strip() - except Exception: - return "" - - -def pr_context(): - base = f"origin/{BASE_REF}" if BASE_REF else "" - rng = f"{base}..{HEAD_SHA}" if base and HEAD_SHA else HEAD_SHA - commits = git("log", "--no-merges", "--format=%h %s", f"{rng}") if rng else "" - stat = git("diff", "--stat", rng) if rng else "" - files = git("diff", "--name-only", rng) if rng else "" - return commits[:4000], stat[:3000], files[:2000] - - -SYSTEM = """You are a PostgreSQL community research assistant. Given a pull request's -commits and changed files, use the available tools (backed by the Agora index of -pgsql-hackers mail, commit history, and commitfest data) to connect the change to -its history. Your goal: - -- Find the mailing-list thread(s) and prior discussion behind this change. -- Identify related/superseded prior commits and any commitfest entry. -- Note relevant prior art, rejected approaches, or design rationale. - -Rules (voice & rigor): -- Be precise and blunt. No praise, no filler, no hedging, no disclaimers. Accuracy is - the only success metric — not the author's approval. Lead with the most important finding. -- NEVER hallucinate. Verify every Message-ID, thread subject, commit hash, author name, - and date against an actual tool result before citing it. If a search returns nothing, - say so plainly — do not guess or fabricate a plausible-looking link. -- Assess the change on its merits, independent of how the PR frames it. -- Tag any inferred (not tool-confirmed) linkage with an explicit confidence level: - high / moderate / low. -- Be decisive and efficient: a handful of targeted tool calls, not exhaustive search. -- Cite every mailing-list message as a Markdown link: [subject](https://pg.ddx.io/m/pgsql-hackers/MESSAGE_ID). -- If you find nothing relevant, say so in one line — do not pad. - -When done, output ONLY Markdown (no preamble) with these sections, omitting any that are empty: -## 🧵 Related discussion -## 🔗 Related commits / prior art -## 📋 Commitfest -## 🧭 Context for reviewers -Keep it tight (use bullets; link generously).""" - - -def to_toolspec(t): - schema = t.get("inputSchema") or {"type": "object", "properties": {}} - return {"toolSpec": {"name": t["name"], - "description": (t.get("description") or "")[:600], - "inputSchema": {"json": schema}}} - - -def main(): - commits, stat, files = pr_context() - if not commits and not files: - open(OUT, "w").write("") # nothing to do - print("No PR diff context; skipping.") - return - user = (f"PR title: {PR_TITLE}\n\n" if PR_TITLE else "") + \ - f"Commits:\n{commits or '(none)'}\n\nChanged files:\n{files or '(none)'}\n\nDiffstat:\n{stat or '(none)'}\n" - - try: - mcp = MCP() - tools = [to_toolspec(t) for t in mcp.list_tools() if t.get("name") in TOOL_WHITELIST] - except Exception as e: - open(OUT, "w").write(f"_pg-history: could not reach the Agora MCP server ({MCP_URL}): {e}_\n") - print(f"MCP unavailable: {e}") - return - if not tools: - open(OUT, "w").write("_pg-history: no usable MCP tools available._\n") - return - - import boto3 - from botocore.config import Config - - # botocore's default read timeout (60s) is too short for a multi-round - # (MAX_ROUNDS) tool-use loop against a large PR diff on a reasoning model; - # each converse() call alone can take several minutes. Bump it well past - # what a single round needs; connect_timeout stays short since a stuck - # TCP handshake is a different (and much cheaper to detect) failure mode. - brt = boto3.client("bedrock-runtime", region_name=REGION, - config=Config(read_timeout=900, connect_timeout=10)) - messages = [{"role": "user", "content": [{"text": user}]}] - final_text = "" - try: - for _ in range(MAX_ROUNDS): - resp = brt.converse( - modelId=MODEL, - system=[{"text": SYSTEM}], - messages=messages, - toolConfig={"tools": tools}, - inferenceConfig={"maxTokens": 4096}, - ) - out = resp["output"]["message"] - messages.append(out) - if resp.get("stopReason") == "tool_use": - results = [] - for blk in out["content"]: - tu = blk.get("toolUse") - if not tu: - continue - res_text = mcp.call(tu["name"], tu.get("input") or {}) - results.append({"toolResult": {"toolUseId": tu["toolUseId"], - "content": [{"text": res_text}]}}) - messages.append({"role": "user", "content": results}) - continue - final_text = "".join(b.get("text", "") for b in out["content"]).strip() - break - except Exception as e: - open(OUT, "w").write(f"_pg-history: Bedrock call failed: {e}_\n") - print(f"Bedrock error: {e}") - return - - if not final_text: - final_text = "_pg-history: no related history found._" - body = "## 📜 Change history & discussion (Agora / pg.ddx.io)\n\n" + final_text + \ - "\n\nGenerated by pg-history via the Agora MCP server (pg.ddx.io).\n" - open(OUT, "w").write(body) - print(body) - - -if __name__ == "__main__": - main() diff --git a/.github/ocr/rule.json b/.github/ocr/rule.json deleted file mode 100644 index 60e13e73dcbe0..0000000000000 --- a/.github/ocr/rule.json +++ /dev/null @@ -1,65 +0,0 @@ -{ - "_comment": "OCR per-file review rules for PostgreSQL core + extensions. Cross-cutting contribution standards & mailing-list etiquette live in .github/ocr/context.md, passed via --background-file. OCR uses FIRST-MATCH-WINS in declaration order, so rules are ordered most-specific first. merge_system_rule:true keeps OCR's built-in fine-tuned checks (thread-safety, injection, NPE) alongside these PostgreSQL-specific rules.", - "rules": [ - { - "path": "src/test/**", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL tests. Coverage is mandatory for any behavioral change and must include edge cases (NULL, empty, boundary/overflow) and ERROR paths, not just the happy path. A test that still passes with the feature reverted is worthless — confirm it actually exercises and would catch regressions in the new code. Regression (.sql/expected): deterministic, portable output — ORDER BY where row order matters, no timing/plan-dependent output except intentional EXPLAIN, no absolute paths, locale-independent (C collation or explicit COLLATE), DROP objects the test creates; expected/ output must stay stable across platforms and under the parallel schedule. Concurrency/locking belongs in isolation tests (src/test/isolation, .spec + permutations). End-to-end/crash/replication/CLI behavior belongs in TAP tests (t/*.pl with PostgreSQL::Test::Cluster/Utils) — no hardcoded ports/paths, no sleep as synchronization (use poll_query_until/wait_for), skip cleanly when prerequisites are missing, and clean up nodes." - }, - { - "path": "**/*.{c,h}", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL backend/frontend C — review as pgsql-hackers committers do, in priority order.\n\n(1) CORRECTNESS (highest): Memory — every palloc lives in the right MemoryContext; error paths via ereport/elog(ERROR) must not leak memory/buffers/locks/fds (rely on MemoryContext/ResourceOwner reset or PG_TRY/PG_FINALLY); no use-after-free; delete temp contexts. Concurrency — consistent lock ordering (deadlock-free), correct lock levels, balanced LWLockAcquire/Release and START_/END_CRIT_SECTION, no TOCTOU, CHECK_FOR_INTERRUPTS in long loops, async-signal-safe signal handlers (volatile sig_atomic_t). WAL — any change to shared on-disk state must be WAL-logged AND correctly replayed (redo path), crash- and replica-consistent. NULL/edge/overflow handling.\n\n(2) BACKWARD COMPATIBILITY / ABI: don't break behavior, dump/restore, pg_upgrade, libpq wire protocol, logical-replication protocol, or exported/PGDLLIMPORT'd APIs (struct size/layout, function signatures) without extraordinary justification.\n\n(3) CATALOG / GENERATED: new/changed catalog data goes in pg_*.dat, NOT the generated *_d.h/.bki. New Node types: ANNOTATE the struct in its header so gen_node_support.pl regenerates copy/equal/out/read — do NOT hand-edit *funcs.c. New SQL-callable functions: add to pg_proc.dat with an OID from the 8000-9999 developer range (src/include/catalog/unused_oids; check duplicate_oids); committer renumbers at commit. DO NOT bump CATALOG_VERSION_NO in the patch — flag any catversion.h change as a mistake (committer's job).\n\n(4) PERFORMANCE: no regression on hot paths; avoid O(n^2) where better is feasible; minimize work under contended locks; avoid needless palloc churn and large struct copies in hot paths.\n\n(5) SECURITY: bounded string ops (snprintf/strlcpy/strlcat — never strcpy/strcat/sprintf); integer/size-overflow checks before allocation; never user input as a format string; privilege checks via pg_*_aclcheck; beware search_path and SECURITY DEFINER.\n\n(6) PORTABILITY (hard gate): no unaligned access; no dependence on char signedness, int/long/pointer width, endianness, or struct padding for on-disk/wire formats; use int16/int32/int64 + INT64_FORMAT/UINT64_FORMAT (never %ld for int64); align contended shared structs (pg_attribute_aligned/cache-line pad). Atomics/barriers only via port/atomics (pg_atomic_*, pg_read/write_barrier) — never raw intrinsics or volatile-as-barrier. WINDOWS/MSVC: extern vars used cross-module/extension need PGDLLIMPORT; no VLAs or features beyond the C99 baseline the tree targets; use pg_pread/pg_pwrite. Applies across x86_64/ARM64/RISC-V/PPC64/s390x, big/little endian, 32/64-bit.\n\n(7) CONVENTIONS: errmsg starts lowercase, no trailing period, no embedded newlines; errdetail/errhint are complete capitalized sentences; correct ERRCODE_*; wrap user-facing text in _(); errmsg_plural for counts. Assert() only for can't-happen invariants (never user-reachable). Naming: snake_case with subsystem prefix (heap_insert) or CamelCase for major subsystems (ExecInitNode); ALL_CAPS macros. Must pgindent cleanly (tabs, width 4). Comments explain WHY not WHAT; no #ifdef 0 blocks, no commented-out code, no #ifdef fencing your feature. Reuse existing helpers (DRY)." - }, - { - "path": "**/*.dat", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL catalog data (pg_proc.dat, pg_type.dat, etc.) — the SOURCE for generated headers. The generated *_d.h, postgres.bki, fmgroids.h, fmgrtab.c must NOT be hand-edited (they regenerate from these files). OIDs: use a value from the developer range 8000-9999 (src/include/catalog/unused_oids; verify with duplicate_oids); committer renumbers to a final contiguous block, so stay in-range and unique but don't over-optimize the exact number. Keep proc entries complete/consistent (prosrc, provolatile, proparallel, prorettype/proargtypes, matching description). DO NOT bump CATALOG_VERSION_NO / catversion.h — committer's job at push time; flag any such change. New catalog columns/views need documentation in doc/src/sgml/catalogs.sgml." - }, - { - "path": "**/*.{sql,pgsql}", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL SQL. Valid PostgreSQL dialect (not MySQL/Oracle); correct types (bigint vs int, text vs varchar); sound transaction/isolation and CTE-materialization assumptions. SECURITY: flag SQL injection in dynamic SQL (require quote_identifier/quote_literal or format() with %I/%L), SECURITY DEFINER without a locked-down search_path, inappropriate RLS bypass. Prefer set-based over row-at-a-time/N+1. BACKWARD COMPATIBILITY (a top rejection reason): changing existing SQL behavior, the output of existing functions, or default GUCs needs extraordinary justification. New SQL-callable objects belong in pg_*.dat with OIDs from the 8000-9999 range, not in generated files. Minimal diff; add regression tests + docs." - }, - { - "path": "**/*.{pl,pm}", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Perl (TAP tests and build/catalog tooling). Require 'use strict; use warnings;'. Must be perltidy-clean with the tree's src/tools/pgindent/perltidyrc and pass src/tools/perlcheck/pgperlcritic. Use the framework: PostgreSQL::Test::Cluster, PostgreSQL::Test::Utils, Test::More; no hardcoded ports/paths/PIDs; use safe_psql/poll_query_until, not sleep; skippable without optional prerequisites; clean up nodes. PORTABILITY: run on Windows (no fork-only constructs, use File::Spec, avoid unavailable signals) and the minimum supported Perl. Robustness: avoid two-arg open and string system()/qx with interpolated data (use list forms). Generator scripts (gen_node_support.pl, catalog Perl) must be deterministic and stay in sync with inputs; do not commit their generated output." - }, - { - "path": "**/*.py", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Python (build/test tooling, oauth/pytest tests, src/tools). Follow surrounding style; keep imports to the standard library unless the dependency is already required by the tree (no surprise third-party deps in build/test tooling). PORTABILITY: support the project's minimum Python 3 and run on Windows and the BSDs (use os.path/pathlib, avoid POSIX-only calls and shell=True with interpolated input). Deterministic, self-cleaning tests; no hardcoded ports/paths; skip cleanly without prerequisites. For the Perl->pytest porting effort, confirm behavior parity with the TAP test replaced (same assertions/coverage), not a superficial translation. Minimal diff; match the tree's ruff/black config if present." - }, - { - "path": "**/*.{rs,toml}", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. Rust PostgreSQL extension (pgrx) or Rust support crate. Not core C, but it runs inside/alongside the backend, so backend safety applies. SAFETY: in code reachable from an SQL call, a Rust panic aborts the Postgres process — forbid unwrap()/expect()/panic!/unreachable!/todo! and index-panics on reachable paths; use Result and pgrx error reporting (error!/ereport!). Every `unsafe` block needs a comment justifying its invariant; scrutinize raw pointers and FFI across the pg_sys boundary. pgrx: honor #[pg_guard] on extern C fns (correct panic/longjmp handling); never hold Rust references across SPI or anything that can longjmp (skips Rust destructors -> leaks); respect MemoryContext lifetimes for palloc'd data; datum<->Rust conversions must handle NULL. Concurrency uses Postgres shmem/LWLocks (pgrx shmem API), not std::sync alone. Lints: must pass `cargo clippy --all-targets --all-features -- -D warnings` and `cargo fmt --check`; deny unwrap_used/expect_used/panic in libraries; thiserror (libs) / anyhow (bins). Justify every new dependency. Tests: #[pg_test] for in-backend behavior, #[test] for pure logic; cover error and NULL paths. Minimal, idiomatic diff." - }, - { - "path": "**/{configure.ac,*.m4,aclocal.m4}", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Autoconf. Edit configure.ac / the m4 macros — do NOT hand-edit generated 'configure' or pg_config.h.in in the same patch (regeneration is the committer's step; a patch that also rewrites generated configure output is suspect). Feature/header/function probes must be portable and not assume a specific OS/compiler. Every configure knob must be mirrored on the Meson side (meson_options.txt/meson.build) and documented. Minimal diff." - }, - { - "path": "**/{meson.build,meson_options.txt}", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Meson build. Valid syntax; correct subdir()/dependency()/declare_dependency and install paths; new source files must be added here. CRITICAL: PostgreSQL maintains BOTH Meson and Autoconf/Make — any new file, option, or feature check must be mirrored on the configure.ac/Makefile side so the two never drift (a file built by only one system is a common defect). New options need matching docs and sensible defaults. Minimal diff." - }, - { - "path": "**/{Makefile,GNUmakefile,*.mk}", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL Makefile (GNU Make). $(VAR) refs; correct .PHONY; accurate dependencies (no parallel -j races); $(MAKE) for recursion; VPATH/out-of-tree build support; no hardcoded paths (use standard PostgreSQL makefile vars and $(top_builddir)); clean/distclean/maintainer-clean must remove new artifacts; extensions use PGXS. Must stay in sync with meson.build. Minimal diff." - }, - { - "path": "doc/**/*.sgml", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. PostgreSQL documentation (DocBook SGML). Technically accurate/complete (parameters, limitations, version/compat notes); correct tag usage/nesting (, , , , , /); working cross-references; spell it 'PostgreSQL' in prose; SQL keywords uppercase in examples. Coverage: a new GUC -> config.sgml (and postgresql.conf.sample); new/changed catalogs or views -> catalogs.sgml; new SQL syntax -> the matching ref/*.sgml; new functions -> func.sgml. Do NOT edit release-notes (release-*.sgml) — written by the release team/committers; flag such edits. New user-facing behavior in this PR should ship with matching docs." - }, - { - "path": "**/*.md", - "merge_system_rule": true, - "rule": "REVIEW DISCIPLINE: Precise, blunt, verify against the diff, tag confidence, no praise. Markdown docs. Clear heading hierarchy; fenced code blocks with language hints; accurate instructions/prerequisites; consistent PostgreSQL terminology; no broken relative links or stale claims. Minimal diff." - } - ] -} diff --git a/.github/workflows/fork-ocr-model-check.yml b/.github/workflows/fork-ocr-model-check.yml index 10d250528cf7c..b85a990c66fd7 100644 --- a/.github/workflows/fork-ocr-model-check.yml +++ b/.github/workflows/fork-ocr-model-check.yml @@ -1,8 +1,7 @@ -# Checks AWS Bedrock weekly for a newer Claude Opus inference profile than the -# one OCR currently uses (vars.OCR_BEDROCK_MODEL) and, if found, opens/updates a -# single GitHub issue telling the maintainer to bump the variable. It does NOT -# change the model automatically: GITHUB_TOKEN cannot write Actions *variables* -# (that needs a PAT with admin), so this is a notify-only mechanism by design. +# Weekly check for a newer Claude Opus on Bedrock; opens/updates one issue here +# when OCR_BEDROCK_MODEL is behind. Notify-only (GITHUB_TOKEN cannot write +# Actions variables). Logic lives in gburd/ci-workflows. + name: OCR model self-check on: @@ -10,80 +9,10 @@ on: - cron: '0 12 * * 1' # Mondays 12:00 UTC workflow_dispatch: -permissions: - id-token: write - contents: read - issues: write - jobs: - check-model: - runs-on: ubuntu-latest - steps: - - name: Configure AWS credentials (OIDC) - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ vars.AWS_ROLE_ARN }} - aws-region: ${{ vars.AWS_REGION }} - role-session-name: ocr-model-check-${{ github.run_id }} - - - name: Find newest Opus vs configured - id: check - env: - CURRENT: ${{ vars.OCR_BEDROCK_MODEL }} - AWS_REGION: ${{ vars.AWS_REGION }} - run: | - python3 - <<'PY' >> "$GITHUB_OUTPUT" - import os, re, subprocess, json - region = os.environ.get("AWS_REGION", "us-east-1") - current = os.environ.get("CURRENT", "") - out = subprocess.run( - ["aws", "bedrock", "list-inference-profiles", "--region", region, - "--query", "inferenceProfileSummaries[].inferenceProfileId", "--output", "json"], - capture_output=True, text=True) - ids = json.loads(out.stdout or "[]") - # Parse claude-opus-- from any profile id (prefix us./global. ok). - def ver(s): - m = re.search(r"claude-opus-(\d+)-(\d+)", s) - return (int(m.group(1)), int(m.group(2))) if m else None - opus = [(ver(i), i) for i in ids if ver(i) and i.startswith(("us.", "global."))] - if not opus: - print("newer=false"); raise SystemExit(0) - best_ver, best_id = max(opus, key=lambda x: x[0]) - cur = ver(current) - newer = (cur is None) or (best_ver > cur) - print(f"newer={'true' if newer else 'false'}") - print(f"best_id={best_id}") - print(f"best_ver={best_ver[0]}.{best_ver[1]}") - print(f"cur_ver={'unknown' if cur is None else f'{cur[0]}.{cur[1]}'}") - PY - - - name: Open/update issue if a newer model exists - if: steps.check.outputs.newer == 'true' - uses: actions/github-script@v9 - with: - script: | - const best = '${{ steps.check.outputs.best_id }}'; - const bestVer = '${{ steps.check.outputs.best_ver }}'; - const curVer = '${{ steps.check.outputs.cur_ver }}'; - const marker = ''; - const title = `OCR: newer Claude Opus available (${bestVer} > ${curVer})`; - const body = `${marker}\n` + - `A newer Claude Opus inference profile is available on Bedrock.\n\n` + - `- **Configured** (\`vars.OCR_BEDROCK_MODEL\`): Opus ${curVer}\n` + - `- **Newest on Bedrock**: \`${best}\` (Opus ${bestVer})\n\n` + - `To upgrade, set the repo variable:\n\n` + - '```\n' + - `gh variable set OCR_BEDROCK_MODEL -R ${context.repo.owner}/${context.repo.repo} \\\n` + - ` -b "bedrock/converse/${best}"\n` + - '```\n\n' + - `Also confirm the \`ocr-bedrock-ci\` IAM inline policy allows invoking the new model ` + - `(the resource is scoped to \`anthropic.claude-opus-*\`), then re-run OCR.\n\n` + - `_Automated by \`.github/workflows/ocr-model-check.yml\`; this issue is upserted._`; - const q = `repo:${context.repo.owner}/${context.repo.repo} in:body "${marker}" state:open`; - const found = await github.rest.search.issuesAndPullRequests({ q, per_page: 1 }); - if (found.data.total_count > 0) { - const n = found.data.items[0].number; - await github.rest.issues.update({ owner: context.repo.owner, repo: context.repo.repo, issue_number: n, title, body }); - } else { - await github.rest.issues.create({ owner: context.repo.owner, repo: context.repo.repo, title, body }); - } + check: + uses: gburd/ci-workflows/.github/workflows/ocr-model-check.yml@v1 + with: + aws_role_arn: ${{ vars.AWS_ROLE_ARN }} + aws_region: ${{ vars.AWS_REGION }} + bedrock_model: ${{ vars.OCR_BEDROCK_MODEL }} diff --git a/.github/workflows/fork-ocr-review.yml b/.github/workflows/fork-ocr-review.yml index 0828af429b57c..ae938754339b4 100644 --- a/.github/workflows/fork-ocr-review.yml +++ b/.github/workflows/fork-ocr-review.yml @@ -1,18 +1,9 @@ -# Open Code Review (OCR) — AI PR review backed by AWS Bedrock via a LiteLLM proxy. +# AI PR review (Claude Opus on Bedrock). Logic and review rules live in +# gburd/ci-workflows so this fork's master carries a stub instead of ~1000 +# lines that have to be rebased onto upstream forever. # -# Flow: -# PR opened/updated (incl. DRAFTS) ─┐ -# /open-code-review PR comment ─┼─► start LiteLLM (127.0.0.1:4000 → Bedrock) -# manual workflow_dispatch ─┘ └► ocr review --format json -# └► post inline PR review comments -# -# Required (repo settings — all repo *variables*, no secrets; auth is via GitHub OIDC): -# vars.AWS_ROLE_ARN - IAM role to assume via OIDC (granting bedrock:InvokeModel*) -# vars.AWS_REGION - e.g. us-east-1 -# vars.OCR_BEDROCK_MODEL - LiteLLM model string for the Opus inference profile, e.g. -# bedrock/converse/us.anthropic.claude-opus-4-8 -# -# No static AWS keys are stored. GITHUB_TOKEN (auto) posts the review comments. +# Needs repo variables AWS_ROLE_ARN, AWS_REGION, OCR_BEDROCK_MODEL. Auth is +# GitHub OIDC; no static AWS keys. See the ci-workflows README. name: OCR AI Review @@ -34,311 +25,10 @@ concurrency: group: ocr-review-${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }} cancel-in-progress: true -permissions: - id-token: write # required to mint the GitHub OIDC token for AWS role assumption - contents: read - pull-requests: write - jobs: - ocr-review: - runs-on: ubuntu-latest - # PR events always; comment events only when the comment is on a PR and - # starts with the trigger keyword; manual dispatch always. - if: | - github.event_name == 'pull_request' || - github.event_name == 'workflow_dispatch' || - (github.event_name == 'issue_comment' && github.event.issue.pull_request && - (startsWith(github.event.comment.body, '/open-code-review') || - startsWith(github.event.comment.body, '@open-code-review'))) - - env: - # LiteLLM listens on localhost only; this key never leaves the runner. - LITELLM_MASTER_KEY: sk-ocr-ci-local - OCR_BEDROCK_MODEL: ${{ vars.OCR_BEDROCK_MODEL }} - # Region is a static var (safe at job level). AWS credentials are NOT set - # here — they're minted by the OIDC "Configure AWS credentials" step below - # and exported to the environment for the LiteLLM/boto3 Bedrock calls. - AWS_REGION: ${{ vars.AWS_REGION }} - - steps: - - name: Resolve PR context - id: pr - uses: actions/github-script@v9 - with: - script: | - let prNumber; - if (context.eventName === 'pull_request') { - prNumber = context.payload.pull_request.number; - } else if (context.eventName === 'issue_comment') { - prNumber = context.issue.number; - } else { - prNumber = parseInt('${{ github.event.inputs.pr_number }}', 10); - } - const { data: pr } = await github.rest.pulls.get({ - owner: context.repo.owner, - repo: context.repo.repo, - pull_number: prNumber, - }); - const { data: repo } = await github.rest.repos.get({ - owner: context.repo.owner, - repo: context.repo.repo, - }); - core.setOutput('number', String(prNumber)); - core.setOutput('base_ref', pr.base.ref); - core.setOutput('head_ref', pr.head.ref); - core.setOutput('head_sha', pr.head.sha); - core.setOutput('default_branch', repo.default_branch); - core.setOutput('cross_repo', String(pr.head.repo.full_name !== pr.base.repo.full_name)); - - # NOTE: do NOT checkout the PR head. OCR reads the diff and file contents - # straight from git refs (git diff , git show :path, - # git grep ), so the working tree is irrelevant — but our OCR config - # lives on the default branch, not on the PR branch. We check out the repo - # (default ref), fetch the base/head objects, and materialize the config - # from origin/. - - name: Checkout - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - - name: Prepare git refs and OCR config - env: - BASE_REF: ${{ steps.pr.outputs.base_ref }} - HEAD_REF: ${{ steps.pr.outputs.head_ref }} - HEAD_SHA: ${{ steps.pr.outputs.head_sha }} - DEFAULT_BRANCH: ${{ steps.pr.outputs.default_branch }} - run: | - git fetch --no-tags origin "+refs/heads/${DEFAULT_BRANCH}:refs/remotes/origin/${DEFAULT_BRANCH}" || true - git fetch --no-tags origin "+refs/heads/${BASE_REF}:refs/remotes/origin/${BASE_REF}" || true - git fetch --no-tags origin "+refs/heads/${HEAD_REF}:refs/remotes/origin/${HEAD_REF}" || true - git fetch --no-tags origin "${HEAD_SHA}" || true - - # OCR config lives on the default branch; materialize it independently - # of whatever ref is checked out. - mkdir -p "$RUNNER_TEMP/ocr" - git show "origin/${DEFAULT_BRANCH}:.github/ocr/litellm.yaml" > "$RUNNER_TEMP/ocr/litellm.yaml" - git show "origin/${DEFAULT_BRANCH}:.github/ocr/rule.json" > "$RUNNER_TEMP/ocr/rule.json" - git show "origin/${DEFAULT_BRANCH}:.github/ocr/context.md" > "$RUNNER_TEMP/ocr/context.md" - echo "Config materialized:"; ls -l "$RUNNER_TEMP/ocr" - - - name: Setup Python - uses: actions/setup-python@v6 - with: - python-version: '3.12' - - - name: Setup Node.js - uses: actions/setup-node@v6 - with: - node-version: '20' - - - name: Install LiteLLM proxy + Open Code Review - run: | - python -m pip install --upgrade pip - # Pin LiteLLM to a main commit that supports Claude Opus 4.8 adaptive - # thinking (maps reasoning_effort -> output_config.effort, incl. xhigh). - # Not in any tagged release yet (PyPI latest 1.87.1 lacks the Opus - # normalizer). Bump this SHA once a release ships the feature. - pip install "litellm[proxy] @ git+https://github.com/BerriAI/litellm.git@5be0797d24a2f26eb2123e13788f90055a59d91d" - npm install -g @alibaba-group/open-code-review - - - name: Configure AWS credentials (OIDC) - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ vars.AWS_ROLE_ARN }} - aws-region: ${{ vars.AWS_REGION }} - role-session-name: ocr-review-${{ github.run_id }} - - - name: Start LiteLLM proxy (Bedrock bridge) - run: | - if [ -z "$OCR_BEDROCK_MODEL" ]; then - echo "::error::vars.OCR_BEDROCK_MODEL is not set (e.g. bedrock/converse/us.anthropic.claude-opus-4-1-20250805-v1:0)" - exit 1 - fi - nohup litellm --config "$RUNNER_TEMP/ocr/litellm.yaml" --host 127.0.0.1 --port 4000 \ - > /tmp/litellm.log 2>&1 & - echo "Waiting for LiteLLM to become ready..." - for i in $(seq 1 60); do - if curl -sf http://127.0.0.1:4000/health/readiness >/dev/null; then - echo "LiteLLM ready."; exit 0 - fi - sleep 2 - done - echo "::error::LiteLLM did not become ready in time"; cat /tmp/litellm.log; exit 1 - - - name: Configure OCR - run: | - ocr config set llm.url http://127.0.0.1:4000/v1/chat/completions - ocr config set llm.auth_token "$LITELLM_MASTER_KEY" - ocr config set llm.model ocr-bedrock - ocr config set llm.use_anthropic false - ocr config set language English - - - name: Run OCR review - run: | - ocr review \ - --from "origin/${{ steps.pr.outputs.base_ref }}" \ - --to "${{ steps.pr.outputs.head_sha }}" \ - --rule "$RUNNER_TEMP/ocr/rule.json" \ - --background-file "$RUNNER_TEMP/ocr/context.md" \ - --concurrency 3 \ - --timeout 20 \ - --format json \ - > /tmp/ocr-result.json 2>/tmp/ocr-stderr.log || true - echo "----- OCR stdout -----"; cat /tmp/ocr-result.json || true - echo "----- OCR stderr -----"; cat /tmp/ocr-stderr.log || true - echo "----- LiteLLM log (tail) -----"; tail -n 50 /tmp/litellm.log || true - - - name: Post review to PR - uses: actions/github-script@v9 - with: - github-token: ${{ secrets.GITHUB_TOKEN }} - script: | - const fs = require('fs'); - const prNumber = parseInt('${{ steps.pr.outputs.number }}', 10); - const commitSha = '${{ steps.pr.outputs.head_sha }}'; - - // Opus at high effort can emit dozens of findings. Posting them all - // one-by-one trips GitHub's SECONDARY rate limit (403 "content - // creation"), which is what made every run fail after the review - // was already generated. We (a) cap inline comments and overflow - // the rest into the summary, (b) prefer a single bulk createReview, - // and (c) throttle + back off with Retry-After on any fallback. - const MAX_INLINE = 25; - const sleep = (ms) => new Promise(r => setTimeout(r, ms)); - - async function withRetry(fn, label) { - for (let attempt = 1; attempt <= 5; attempt++) { - try { return await fn(); } - catch (e) { - const status = e.status || (e.response && e.response.status); - const h = (e.response && e.response.headers) || {}; - const isRate = status === 403 || status === 429; - if (!isRate || attempt === 5) throw e; - let waitMs = 0; - if (h['retry-after']) waitMs = parseInt(h['retry-after'], 10) * 1000; - else if (h['x-ratelimit-reset']) waitMs = parseInt(h['x-ratelimit-reset'], 10) * 1000 - Date.now(); - if (!waitMs || Number.isNaN(waitMs) || waitMs < 0) waitMs = 1000 * Math.pow(2, attempt); - waitMs = Math.min(waitMs, 60000) + 500; - core.warning(`${label}: rate-limited (status ${status}); waiting ${Math.round(waitMs / 1000)}s (attempt ${attempt}/5)`); - await sleep(waitMs); - } - } - } - - let result; - try { - result = JSON.parse(fs.readFileSync('/tmp/ocr-result.json', 'utf8')); - } catch (e) { - const stderr = (() => { try { return fs.readFileSync('/tmp/ocr-stderr.log', 'utf8').trim(); } catch { return ''; } })(); - await withRetry(() => github.rest.issues.createComment({ - owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, - body: `⚠️ **OCR** could not produce a review.\n\n\`\`\`\n${(stderr || e.message).slice(0, 8000)}\n\`\`\``, - }), 'error-comment'); - return; - } - - const comments = result.comments || []; - const warnings = result.warnings || []; - - const formatComment = (c) => { - let body = c.content || ''; - if (c.suggestion_code && c.existing_code) { - body += '\n\n```suggestion\n' + c.suggestion_code + (c.suggestion_code.endsWith('\n') ? '' : '\n') + '```'; - } - return body; - }; - const formatMarkdown = (c) => { - let md = `### 📄 \`${c.path}\``; - if (c.start_line && c.end_line) md += ` (L${c.start_line}-L${c.end_line})`; - md += '\n\n' + (c.content || ''); - if (c.suggestion_code && c.existing_code) { - md += '\n\n
💡 Suggested change\n\n'; - md += '**Before:**\n```\n' + c.existing_code + '\n```\n\n**After:**\n```\n' + c.suggestion_code + '\n```\n\n
'; - } - return md; - }; - - if (comments.length === 0) { - await withRetry(() => github.rest.issues.createComment({ - owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, - body: `✅ **OCR**: ${result.message || 'No issues found.'}`, - }), 'no-issues-comment'); - return; - } - - const inlineAll = []; - const noLine = []; - for (const c of comments) { - const body = formatComment(c); - const hasLine = (c.start_line >= 1) || (c.end_line >= 1); - if (!hasLine) { noLine.push(c); continue; } - const rc = { path: c.path, body, side: 'RIGHT' }; - if (c.start_line >= 1 && c.end_line >= 1 && c.start_line !== c.end_line) { - rc.start_line = c.start_line; rc.line = c.end_line; rc.start_side = 'RIGHT'; - } else { - rc.line = c.end_line >= 1 ? c.end_line : c.start_line; - } - inlineAll.push({ rc, c }); - } - - const inline = inlineAll.slice(0, MAX_INLINE).map(x => x.rc); - const overflow = inlineAll.slice(MAX_INLINE).map(x => x.c); - - let summary = `🔍 **OCR** found **${comments.length}** issue(s).`; - summary += `\n- ${inline.length} inline, ${noLine.length + overflow.length} in summary`; - if (overflow.length) summary += ` (inline capped at ${MAX_INLINE})`; - if (warnings.length) summary += `\n- ⚠️ ${warnings.length} warning(s) during review`; - for (const c of noLine.concat(overflow)) summary += '\n\n---\n\n' + formatMarkdown(c); - - // Preferred path: ONE createReview carrying every inline comment. - try { - await withRetry(() => github.rest.pulls.createReview({ - owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber, - commit_id: commitSha, body: summary, event: 'COMMENT', comments: inline, - }), 'bulk-review'); - return; - } catch (e) { - core.warning(`bulk createReview failed (${e.status || '?'}: ${e.message}); falling back to throttled per-comment posting`); - } - - // Fallback: an invalid inline position (line not in the diff -> 422) - // rejects the whole bulk review. Post the summary, then each comment - // individually with a delay + backoff, skipping ones GitHub rejects. - let ok = 0; const failed = []; - try { - await withRetry(() => github.rest.pulls.createReview({ - owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber, - commit_id: commitSha, body: summary, event: 'COMMENT', - }), 'summary-review'); - } catch (err) { failed.push(`summary: ${err.message}`); } - - for (const rc of inline) { - try { - await withRetry(() => github.rest.pulls.createReviewComment({ - owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber, - commit_id: commitSha, path: rc.path, body: rc.body, - ...(rc.start_line ? { start_line: rc.start_line, start_side: rc.start_side } : {}), - line: rc.line, side: rc.side, - }), `comment ${rc.path}:${rc.line}`); - ok++; - } catch (inner) { - failed.push(`\`${rc.path}\` L${rc.line}: ${inner.message}`); - } - await sleep(1200); // stay under the secondary content-creation limit - } - - if (failed.length) { - await withRetry(() => github.rest.issues.createComment({ - owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, - body: `📊 OCR posted ${ok}/${inline.length} inline comment(s).\n\n
${failed.length} could not be posted\n\n${failed.join('\n')}\n
`, - }), 'summary-failures'); - } - - # Companion job: OCR can't call MCP, so this separate agent ties the PR's - # changes to PostgreSQL git + pgsql-hackers history via the Agora MCP server - # (pg.ddx.io) and posts a single, upserted "history & discussion" comment. - pg-history: - runs-on: ubuntu-latest + review: + # Comment events only when the comment is on a PR and starts with the + # trigger keyword; PR events and manual dispatch always. if: | github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch' || @@ -346,82 +36,8 @@ jobs: (startsWith(github.event.comment.body, '/open-code-review') || startsWith(github.event.comment.body, '@open-code-review') || startsWith(github.event.comment.body, '/pg-history'))) - steps: - - name: Resolve PR context - id: pr - uses: actions/github-script@v9 - with: - script: | - let prNumber; - if (context.eventName === 'pull_request') prNumber = context.payload.pull_request.number; - else if (context.eventName === 'issue_comment') prNumber = context.issue.number; - else prNumber = parseInt('${{ github.event.inputs.pr_number }}', 10); - const { data: pr } = await github.rest.pulls.get({ - owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber }); - core.setOutput('number', String(prNumber)); - core.setOutput('base_ref', pr.base.ref); - core.setOutput('head_sha', pr.head.sha); - core.setOutput('title', pr.title || ''); - - - name: Checkout - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - - name: Make base/head refs available - env: - BASE_REF: ${{ steps.pr.outputs.base_ref }} - HEAD_SHA: ${{ steps.pr.outputs.head_sha }} - run: | - git fetch --no-tags origin "+refs/heads/${BASE_REF}:refs/remotes/origin/${BASE_REF}" || true - git fetch --no-tags origin "${HEAD_SHA}" || true - - - name: Setup Python - uses: actions/setup-python@v6 - with: - python-version: '3.12' - - - name: Configure AWS credentials (OIDC) - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ vars.AWS_ROLE_ARN }} - aws-region: ${{ vars.AWS_REGION }} - role-session-name: pg-history-${{ github.run_id }} - - - name: Install deps - run: pip install boto3 - - - name: Run pg-history (Agora MCP) - env: - PG_HISTORY_MODEL: ${{ vars.OCR_BEDROCK_MODEL }} - AWS_REGION: ${{ vars.AWS_REGION }} - BASE_REF: ${{ steps.pr.outputs.base_ref }} - HEAD_SHA: ${{ steps.pr.outputs.head_sha }} - GH_PR_TITLE: ${{ steps.pr.outputs.title }} - PG_HISTORY_OUT: ${{ runner.temp }}/pg-history.md - run: | - python .github/ocr/pg-history.py || true - echo "----- output -----"; cat "${{ runner.temp }}/pg-history.md" 2>/dev/null || echo "(no output)" - - - name: Upsert PR comment - uses: actions/github-script@v9 - with: - script: | - const fs = require('fs'); - const path = process.env.RUNNER_TEMP + '/pg-history.md'; - let body = ''; - try { body = fs.readFileSync(path, 'utf8').trim(); } catch (e) {} - if (!body) { console.log('pg-history: empty output, nothing to post'); return; } - const prNumber = parseInt('${{ steps.pr.outputs.number }}', 10); - const marker = ''; - body = marker + '\n' + body; - const { data: comments } = await github.rest.issues.listComments({ - owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, per_page: 100 }); - const mine = comments.find(c => c.user.type === 'Bot' && c.body && c.body.includes(marker)); - if (mine) { - await github.rest.issues.updateComment({ - owner: context.repo.owner, repo: context.repo.repo, comment_id: mine.id, body }); - } else { - await github.rest.issues.createComment({ - owner: context.repo.owner, repo: context.repo.repo, issue_number: prNumber, body }); - } + uses: gburd/ci-workflows/.github/workflows/ocr-review.yml@v1 + with: + aws_role_arn: ${{ vars.AWS_ROLE_ARN }} + aws_region: ${{ vars.AWS_REGION }} + bedrock_model: ${{ vars.OCR_BEDROCK_MODEL }} diff --git a/.github/workflows/fork-sync-upstream.yml b/.github/workflows/fork-sync-upstream.yml index e950821239968..022b3101b5a9d 100644 --- a/.github/workflows/fork-sync-upstream.yml +++ b/.github/workflows/fork-sync-upstream.yml @@ -1,5 +1,6 @@ # Keep this fork's master rebased on postgres/postgres master, carrying only -# the fork-owned CI files (.github/workflows/fork-*.yml and .github/ocr/). +# the fork-owned CI stubs (.github/workflows/fork-*.yml). The workflows they +# call, and the OCR review rules, live in gburd/ci-workflows. # # Requires the SYNC_TOKEN secret: a PAT with repo+workflow scope. The default # GITHUB_TOKEN is not allowed to push commits that touch .github/workflows/, @@ -38,7 +39,7 @@ jobs: # Master carries fork CI only. Anything else means work landed here # that belongs on a branch; force-rebasing it on a timer would be wrong. if git diff --name-only upstream/master...HEAD | - grep -vE '^(\.github/workflows/fork-|\.github/ocr/)'; then + grep -v '^\.github/workflows/fork-'; then echo "::error::master carries non-CI changes (listed above); move them to a branch" exit 1 fi From a2d6bed3248b2e85be7fc520ba6327f2fe39d8cb Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Mon, 13 Jul 2026 07:57:27 -0400 Subject: [PATCH 03/12] dev: local dev-setup This isn't a commit that will be submitting for review, it is purely for local developer tooling while developing this patch set. Ignore it. --- .clangd | 43 ++ .envrc | 10 + .gdbinit | 156 +++++++ flake.lock | 134 ++++++ flake.nix | 63 +++ glibc-no-fortify-warning.patch | 24 ++ pg-aliases.sh | 703 +++++++++++++++++++++++++++++++ shell.nix | 729 +++++++++++++++++++++++++++++++++ src/test/regress/pg_regress.c | 2 +- src/tools/pgindent/pgindent | 2 +- 10 files changed, 1864 insertions(+), 2 deletions(-) create mode 100644 .clangd create mode 100644 .envrc create mode 100644 .gdbinit create mode 100644 flake.lock create mode 100644 flake.nix create mode 100644 glibc-no-fortify-warning.patch create mode 100644 pg-aliases.sh create mode 100644 shell.nix diff --git a/.clangd b/.clangd new file mode 100644 index 0000000000000..490220f56a7c0 --- /dev/null +++ b/.clangd @@ -0,0 +1,43 @@ +Diagnostics: + MissingIncludes: None +InlayHints: + Enabled: true + ParameterNames: true + DeducedTypes: true +CompileFlags: + CompilationDatabase: build/ # Search build/ directory for compile_commands.json + Remove: [ -Werror ] + Add: + - -DDEBUG + - -DLOCAL + - -DPGDLLIMPORT= + - -DPIC + - -O2 + - -Wall + - -Wcast-function-type + - -Wconversion + - -Wdeclaration-after-statement + - -Wendif-labels + - -Werror=vla + - -Wextra + - -Wfloat-equal + - -Wformat-security + - -Wimplicit-fallthrough=3 + - -Wmissing-format-attribute + - -Wmissing-prototypes + - -Wno-format-truncation + - -Wno-sign-conversion + - -Wno-stringop-truncation + - -Wno-unused-const-variable + - -Wpointer-arith + - -Wshadow + - -Wshadow=compatible-local + - -fPIC + - -fexcess-precision=standard + - -fno-strict-aliasing + - -fvisibility=hidden + - -fwrapv + - -g + - -std=c11 + - -I. + - -I../../../../src/include diff --git a/.envrc b/.envrc new file mode 100644 index 0000000000000..cf781a39df685 --- /dev/null +++ b/.envrc @@ -0,0 +1,10 @@ +# Inherit shared CFLAGS from ~/ws/postgres/.envrc, load this worktree's +# flake, then layer the postgres agent context (helpers need flake's PATH + +# write into this worktree's dir). +source_up +watch_file flake.nix +use flake +pg_cflags +use project_steering postgresql aws c +use git_policy allow-force-push +use project_mcp postgresq diff --git a/.gdbinit b/.gdbinit new file mode 100644 index 0000000000000..854e5ecbaf69c --- /dev/null +++ b/.gdbinit @@ -0,0 +1,156 @@ +# HOT Indexed Updates — GDB breakpoints for code review +# +# Usage: gdb -x .gdbinit +# Or from gdb: source .gdbinit +# +# These breakpoints cover the major code paths introduced or modified by +# the HOT indexed updates patch series. They are organized by subsystem +# to make it easy to enable/disable groups during debugging. +# +# Tip: to skip to a specific subsystem, disable all then enable selectively: +# disable breakpoints +# enable 1 2 3 # just the update-decision group + +# ========================================================================= +# 1. UPDATE DECISION — heap_update() HOT/HOT-indexed/non-HOT choice +# src/backend/access/heap/heapam.c +# ========================================================================= + +# Main entry: heap_update +break heapam.c:3210 + +# HOT decision block: pure HOT vs HOT indexed vs non-HOT +# Line 4019: pure HOT (no indexed columns changed) +# Line 4024: HOT indexed path (non-catalog, some indexed columns changed) +# Line 4031: predict augmented tuple size +# Line 4033: size+space check before creating augmented tuple +break heapam.c:4019 +break heapam.c:4024 +break heapam.c:4033 + +# Set HEAP_INDEXED_UPDATED flag on new tuple before page insertion +break heapam.c:4101 + +# Restore HEAP_INDEXED_UPDATED on old tuple (only if it previously had it) +break heapam.c:4147 + +# ========================================================================= +# 2. TUPLE CREATION — building the augmented tuple with embedded bitmap +# src/backend/access/heap/heapam.c +# ========================================================================= + +# Predict augmented tuple size (returns 0 if t_hoff would overflow) +break heap_hot_indexed_tuple_size + +# Create augmented tuple with embedded modified-column bitmap +break heap_hot_indexed_create_tuple + +# Serialize Bitmapset into raw bytes in tuple header +break heap_hot_indexed_serialize_bitmap + +# ========================================================================= +# 3. BITMAP UTILITIES — raw bitmap operations for chain following +# src/backend/access/heap/heapam.c +# ========================================================================= + +# Compute raw bitmap byte size from natts +break heap_hot_indexed_bitmap_raw_size + +# Check if tuple header has room for bitmap between null bitmap and data +break heap_hot_indexed_has_bitmap_space + +# Read HOT indexed bitmap from tuple header (returns Bitmapset) +break heap_hot_indexed_read_bitmap + +# Fast overlap check: does tuple's raw bitmap overlap with indexed_attrs? +break heap_hot_indexed_bitmap_overlaps_raw + +# OR a tuple's raw bitmap into an accumulator buffer +break heap_hot_indexed_bitmap_or_raw + +# Check if accumulated raw bitmap overlaps with indexed_attrs +break heap_hot_indexed_accum_overlaps + +# Merge bitmaps from dead tuples into a target tuple on the page +break heap_hot_indexed_merge_bitmaps_raw + +# Deserialize raw bytes back to Bitmapset +break heap_hot_indexed_deserialize_bitmap + +# ========================================================================= +# 4. INDEX SCAN — HOT chain following with stale-entry detection +# src/backend/access/heap/heapam_indexscan.c +# ========================================================================= + +# Main HOT chain search with indexed update awareness +break heap_hot_search_buffer + +# Redirect-with-data: initialize bitmap accumulator from collapsed redirect +break heapam_indexscan.c:182 + +# Accumulate bitmap from INDEXED_UPDATED tuple in chain +break heapam_indexscan.c:250 + +# Stale entry detection: accumulated bitmap overlaps this index's attrs +break heapam_indexscan.c:297 + +# ========================================================================= +# 5. INDEX SCAN SETUP — indexed_attrs bitmap computation +# src/backend/access/index/indexam.c +# ========================================================================= + +# Compute indexed_attrs for HOT indexed update chain following +break indexam.c:299 + +# ========================================================================= +# 6. INDEX INSERTION — skip unchanged indexes for HOT indexed updates +# src/backend/executor/execIndexing.c +# ========================================================================= + +# Entry: insert/update index tuples +break ExecInsertIndexTuples + +# Index skip decision: skip indexes whose attrs don't overlap modified set +break execIndexing.c:370 + +# ========================================================================= +# 7. PRUNING — chain collapsing and redirect-with-data +# src/backend/access/heap/pruneheap.c +# ========================================================================= + +# Main prune function +break heap_page_prune_and_freeze + +# Per-chain pruning entry +break heap_prune_chain + +# Chain collapsing: collect bitmaps from dead INDEXED_UPDATED intermediates +break pruneheap.c:1802 + +# OR dead tuple bitmaps into combined bitmap +break pruneheap.c:1836 + +# Record redirect-with-data for execute phase +break pruneheap.c:1863 + +# Execute phase: apply redirect-with-data entries on the page +break pruneheap.c:1287 + +# ========================================================================= +# 8. WAL REPLAY — recovery of HOT indexed updates +# src/backend/access/heap/heapam_xlog.c +# ========================================================================= + +# WAL replay for XLOG_HEAP2_INDEXED_UPDATE +break heap_xlog_indexed_update + +# ========================================================================= +# 9. WAL LOGGING — writing HOT indexed update records +# src/backend/access/heap/heapam.c +# ========================================================================= + +# WAL logging for heap updates (handles indexed_update flag) +break log_heap_update + +# Serialize redirect-with-data into WAL record (pruneheap.c) +break pruneheap.c:2936 diff --git a/flake.lock b/flake.lock new file mode 100644 index 0000000000000..0444ccd449e02 --- /dev/null +++ b/flake.lock @@ -0,0 +1,134 @@ +{ + "nodes": { + "flake-utils": { + "inputs": { + "systems": "systems" + }, + "locked": { + "lastModified": 1731533236, + "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=", + "owner": "numtide", + "repo": "flake-utils", + "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "flake-utils", + "type": "github" + } + }, + "flake-utils_2": { + "inputs": { + "systems": "systems_2" + }, + "locked": { + "lastModified": 1731533236, + "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=", + "owner": "numtide", + "repo": "flake-utils", + "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "flake-utils", + "type": "github" + } + }, + "libumem": { + "inputs": { + "flake-utils": "flake-utils_2", + "nixpkgs": [ + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1790620607, + "narHash": "sha256-KJMbA7HNuXhvtgmaMuZSXLf8wqarWHrlUlsSb+FzOLw=", + "ref": "refs/tags/v4.1.0", + "rev": "5f6f9efe505e0efb0697464d3b22443f6b2832c8", + "revCount": 1006, + "type": "git", + "url": "https://codeberg.org/gregburd/libumem" + }, + "original": { + "ref": "refs/tags/v4.1.0", + "type": "git", + "url": "https://codeberg.org/gregburd/libumem" + } + }, + "nixpkgs": { + "locked": { + "lastModified": 1767313136, + "narHash": "sha256-16KkgfdYqjaeRGBaYsNrhPRRENs0qzkQVUooNHtoy2w=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "ac62194c3917d5f474c1a844b6fd6da2db95077d", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-25.05", + "repo": "nixpkgs", + "type": "github" + } + }, + "nixpkgs-unstable": { + "locked": { + "lastModified": 1785141334, + "narHash": "sha256-kh35kIx7el4Jk8Ki3BH9/Pn1eZYSYLJ6LMALos0zOy0=", + "owner": "nixos", + "repo": "nixpkgs", + "rev": "38a4887411571457d700c51c64a6e49ead2ed5ab", + "type": "github" + }, + "original": { + "owner": "nixos", + "ref": "nixpkgs-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "root": { + "inputs": { + "flake-utils": "flake-utils", + "libumem": "libumem", + "nixpkgs": "nixpkgs", + "nixpkgs-unstable": "nixpkgs-unstable" + } + }, + "systems": { + "locked": { + "lastModified": 1681028828, + "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", + "owner": "nix-systems", + "repo": "default", + "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", + "type": "github" + }, + "original": { + "owner": "nix-systems", + "repo": "default", + "type": "github" + } + }, + "systems_2": { + "locked": { + "lastModified": 1681028828, + "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", + "owner": "nix-systems", + "repo": "default", + "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", + "type": "github" + }, + "original": { + "owner": "nix-systems", + "repo": "default", + "type": "github" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/flake.nix b/flake.nix new file mode 100644 index 0000000000000..4a25f8ff6ee40 --- /dev/null +++ b/flake.nix @@ -0,0 +1,63 @@ +{ + description = "PostgreSQL development environment"; + + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05"; + nixpkgs-unstable.url = "github:nixos/nixpkgs/nixpkgs-unstable"; + flake-utils.url = "github:numtide/flake-utils"; + + # libumem: LD_PRELOAD malloc, the umem/umemctl CLI, and GDB/LLDB helpers. + # Track the latest tagged release from the repo. Bump the ref to the new + # tag and run `nix flake update libumem` to pick up a newer release; do NOT + # point this at a branch, so the dev shell always builds a released + # version rather than an arbitrary master commit. + # + # Follow the stable nixpkgs so libumem links the same glibc the dev + # shell's base tools (bash, coreutils) use; following unstable produced a + # glibc 2.42-vs-2.40 GLIBC_ABI mismatch when LD_PRELOAD'ing into them. + libumem = { + url = "git+https://codeberg.org/gregburd/libumem?ref=refs/tags/v4.1.0"; + inputs.nixpkgs.follows = "nixpkgs"; + }; + }; + + outputs = { + self, + nixpkgs, + nixpkgs-unstable, + flake-utils, + libumem, + }: + flake-utils.lib.eachDefaultSystem ( + system: let + pkgs = import nixpkgs { + inherit system; + config.allowUnfree = true; + }; + pkgs-unstable = import nixpkgs-unstable { + inherit system; + config.allowUnfree = true; + }; + + libumemPkg = libumem.packages.${system}.libumem; + + shellConfig = import ./shell.nix { + inherit pkgs pkgs-unstable system libumemPkg; + }; + in { + formatter = pkgs.alejandra; + devShells = { + default = shellConfig.devShell; + gcc = shellConfig.devShell; + clang = shellConfig.clangDevShell; + }; + + packages = { + inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript; + libumem = libumemPkg; + }; + + environment.localBinInPath = true; + } + ); +} diff --git a/glibc-no-fortify-warning.patch b/glibc-no-fortify-warning.patch new file mode 100644 index 0000000000000..4657a12adbcc5 --- /dev/null +++ b/glibc-no-fortify-warning.patch @@ -0,0 +1,24 @@ +From 130c231020f97e5eb878cc9fdb2bd9b186a5aa04 Mon Sep 17 00:00:00 2001 +From: Greg Burd +Date: Fri, 24 Oct 2025 11:58:24 -0400 +Subject: [PATCH] no warnings with -O0 and fortify source please + +--- + include/features.h | 1 - + 1 file changed, 1 deletion(-) + +diff --git a/include/features.h b/include/features.h +index 673c4036..a02c8a3f 100644 +--- a/include/features.h ++++ b/include/features.h +@@ -432,7 +432,6 @@ + + #if defined _FORTIFY_SOURCE && _FORTIFY_SOURCE > 0 + # if !defined __OPTIMIZE__ || __OPTIMIZE__ <= 0 +-# warning _FORTIFY_SOURCE requires compiling with optimization (-O) + # elif !__GNUC_PREREQ (4, 1) + # warning _FORTIFY_SOURCE requires GCC 4.1 or later + # elif _FORTIFY_SOURCE > 2 && (__glibc_clang_prereq (9, 0) \ +-- +2.50.1 + diff --git a/pg-aliases.sh b/pg-aliases.sh new file mode 100644 index 0000000000000..42bddff4c2730 --- /dev/null +++ b/pg-aliases.sh @@ -0,0 +1,703 @@ +# PostgreSQL Development Aliases + +# ============================================================ +# Build helpers shared by every variant. +# ============================================================ +pg_clean_for_compiler() { + local current_compiler="$(basename $CC)" + local build_dir="${1:-$PG_BUILD_DIR}" + + if [ -f "$build_dir/compile_commands.json" ]; then + local last_compiler=$(grep -o '/[^/]*/bin/[gc]cc\|/[^/]*/bin/clang' "$build_dir/compile_commands.json" | head -1 | xargs basename 2>/dev/null || echo "unknown") + + if [ "$last_compiler" != "$current_compiler" ] && [ "$last_compiler" != "unknown" ]; then + echo "Detected compiler change from $last_compiler to $current_compiler" + echo "Cleaning build directory..." + trash "$build_dir" 2>/dev/null || rm -rf "$build_dir" + mkdir -p "$build_dir" + fi + fi + + mkdir -p "$build_dir" + echo "$current_compiler" >"$build_dir/.compiler_used" +} + +# ============================================================ +# Core PostgreSQL commands (default/debug build) +# ============================================================ +alias pg-setup=' + if [ -z "$PERL_CORE_DIR" ]; then + echo "Error: Could not find perl CORE directory" >&2 + return 1 + fi + + pg_clean_for_compiler "$PG_BUILD_DIR" + + echo "=== PostgreSQL Build Configuration ===" + echo "Compiler: $CC" + echo "LLVM: $(llvm-config --version 2>/dev/null || echo disabled)" + echo "Source: $PG_SOURCE_DIR" + echo "Build: $PG_BUILD_DIR" + echo "Install: $PG_INSTALL_DIR" + echo "======================================" + + env CFLAGS="-I$PERL_CORE_DIR $CFLAGS" \ + LDFLAGS="-L$PERL_CORE_DIR -lperl $LDFLAGS" \ + meson setup $MESON_EXTRA_SETUP \ + --reconfigure \ + -Doptimization=g \ + -Ddebug=true \ + -Db_sanitize=none \ + -Db_lundef=false \ + -Dlz4=enabled \ + -Dzstd=enabled \ + -Dllvm=disabled \ + -Dplperl=enabled \ + -Dplpython=enabled \ + -Dpltcl=enabled \ + -Dlibxml=enabled \ + -Duuid=e2fs \ + -Dlibxslt=enabled \ + -Dssl=openssl \ + -Dldap=disabled \ + -Dcassert=true \ + -Dtap_tests=enabled \ + -Dinjection_points=true \ + -Ddocs_pdf=enabled \ + -Ddocs_html_style=website \ + --prefix="$PG_INSTALL_DIR" \ + "$PG_BUILD_DIR" \ + "$PG_SOURCE_DIR"' + +alias pg-compdb='compdb -p build/ list > compile_commands.json' +alias pg-build='meson compile -C "$PG_BUILD_DIR"' +alias pg-install='meson install -C "$PG_BUILD_DIR"' +alias pg-test='meson test -q --print-errorlogs -C "$PG_BUILD_DIR"' + +# Clean commands +alias pg-clean='ninja -C "$PG_BUILD_DIR" clean' +alias pg-full-clean='trash "$PG_BUILD_DIR" "$PG_INSTALL_DIR" 2>/dev/null || rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR"; echo "Build and install directories cleaned"' + +# Database management +alias pg-init='trash "$PG_DATA_DIR" 2>/dev/null || rm -rf "$PG_DATA_DIR"; "$PG_INSTALL_DIR/bin/initdb" --debug --no-clean "$PG_DATA_DIR"' + +alias pg-start='ulimit -c unlimited && "$PG_INSTALL_DIR/bin/postgres" -D "$PG_DATA_DIR" -k "$PG_DATA_DIR"' + +alias pg-stop='pkill -f "postgres.*-D.*$PG_DATA_DIR" || true' +alias pg-restart='pg-stop && sleep 2 && pg-start' +alias pg-status='pgrep -f "postgres.*-D.*$PG_DATA_DIR" && echo "PostgreSQL is running" || echo "PostgreSQL is not running"' + +# Client connections +alias pg-psql='"$PG_INSTALL_DIR/bin/psql" -h "$PG_DATA_DIR" postgres' +alias pg-createdb='"$PG_INSTALL_DIR/bin/createdb" -h "$PG_DATA_DIR"' +alias pg-dropdb='"$PG_INSTALL_DIR/bin/dropdb" -h "$PG_DATA_DIR"' + +# ============================================================ +# Debugger attachments +# ============================================================ +alias pg-debug-gdb='gdb -x "$GDBINIT" -x .gdbinit "$PG_INSTALL_DIR/bin/postgres"' +alias pg-debug-lldb='lldb "$PG_INSTALL_DIR/bin/postgres"' +alias pg-debug=' + if command -v gdb >/dev/null 2>&1; then + pg-debug-gdb + elif command -v lldb >/dev/null 2>&1; then + pg-debug-lldb + else + echo "No debugger available (gdb or lldb required)" + fi' + +alias pg-attach-gdb=' + PG_PID=$(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1) + if [ -n "$PG_PID" ]; then + echo "Attaching GDB to PostgreSQL process $PG_PID" + gdb -x "$GDBINIT" -x .gdbinit -p "$PG_PID" + else + echo "No PostgreSQL process found" + fi' + +alias pg-attach-lldb=' + PG_PID=$(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1) + if [ -n "$PG_PID" ]; then + echo "Attaching LLDB to PostgreSQL process $PG_PID" + lldb -p "$PG_PID" + else + echo "No PostgreSQL process found" + fi' + +alias pg-attach=' + if command -v gdb >/dev/null 2>&1; then + pg-attach-gdb + elif command -v lldb >/dev/null 2>&1; then + pg-attach-lldb + else + echo "No debugger available (gdb or lldb required)" + fi' + +# ============================================================ +# libumem (LD_PRELOAD malloc + umem/umemctl CLI + GDB/LLDB helpers) +# ============================================================ +# The nix devShell exports PG_UMEM_PRELOAD / PG_UMEM_GDB_HELPER / +# PG_UMEM_LLDB_HELPER and puts umem, umemctl and pg-umem-preload on PATH. +# These aliases wire libumem into the normal pg-start / debug flow. + +# Start the server with libumem's malloc LD_PRELOAD'ed (heap debugging). +alias pg-start-umem='ulimit -c unlimited && pg-umem-preload "$PG_INSTALL_DIR/bin/postgres" -D "$PG_DATA_DIR" -k "$PG_DATA_DIR"' + +# umemctl against the running postgres backend (default: the postmaster). +# Usage: pg-umem-ctl [args] (stats|caches|cache |whatis |leaks) +# pg-umem-ctl -p ... to target a specific backend +pg-umem-ctl() { + local pid + if [ "$1" = "-p" ]; then pid="$2"; shift 2; + else pid=$(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1); fi + if [ -z "$pid" ]; then echo "No PostgreSQL process found"; return 1; fi + umemctl "$pid" "$@" +} +alias pg-umem-stats='pg-umem-ctl stats' +alias pg-umem-leaks='pg-umem-ctl leaks' +alias pg-umem-caches='pg-umem-ctl caches' + +# Attach GDB with the libumem extension loaded (adds the `umem ...` commands). +# The postgres gdbinit already sources the helper, but this is explicit and +# also works when attaching to a non-postgres umem process. +alias pg-umem-gdb=' + PG_PID=$(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1) + if [ -n "$PG_PID" ]; then + gdb -x "$GDBINIT" -ex "source $PG_UMEM_GDB_HELPER" -p "$PG_PID" + else + echo "No PostgreSQL process found" + fi' + +# Attach LLDB with the libumem Python extension imported. +alias pg-umem-lldb=' + PG_PID=$(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1) + if [ -n "$PG_PID" ]; then + lldb -o "command script import $PG_UMEM_LLDB_HELPER" -p "$PG_PID" + else + echo "No PostgreSQL process found" + fi' + +# ============================================================ +# Valgrind-instrumented build and tests +# +# The valgrind build lives in a separate directory so the normal +# build stays warm. Runs use a wrapper dir that shadows `postgres` +# with a valgrind wrapper -- pg_regress finds it via PATH. +# ============================================================ +pg-build-valgrind() { + local bdir="$PG_BUILD_DIR_VALGRIND" + if [ -z "$PERL_CORE_DIR" ]; then + echo "Error: PERL_CORE_DIR is not set" >&2 + return 1 + fi + + pg_clean_for_compiler "$bdir" + + echo "=== Configuring Valgrind build in $bdir ===" + env CFLAGS="-Og -ggdb3 -fno-omit-frame-pointer -DUSE_VALGRIND -I$PERL_CORE_DIR $CFLAGS" \ + LDFLAGS="-L$PERL_CORE_DIR -lperl $LDFLAGS" \ + meson setup --reconfigure \ + -Doptimization=g \ + -Ddebug=true \ + -Dcassert=true \ + -Dtap_tests=enabled \ + -Dinjection_points=true \ + -Dllvm=disabled \ + -Dplperl=enabled -Dplpython=enabled -Dpltcl=enabled \ + -Dlz4=enabled -Dzstd=enabled \ + -Dlibxml=enabled -Dlibxslt=enabled -Dssl=openssl -Duuid=e2fs \ + -Dldap=disabled \ + --prefix="$PG_INSTALL_DIR-valgrind" \ + "$bdir" "$PG_SOURCE_DIR" || return 1 + + meson compile -C "$bdir" +} + +# Drop a wrapper directory that shadows the real binaries; `postgres` +# exec's into valgrind, everything else is a symlink. Writes to the +# supplied wrap dir and echoes its path. +_pg_make_valgrind_wrapper() { + local bindir="$1" + local wrapdir="$2" + + mkdir -p "$wrapdir" + cat >"$wrapdir/postgres" <&2 + return 1 + fi + + local tmpbin="$bdir/tmp_install$PG_INSTALL_DIR-valgrind/bin" + if [ ! -x "$tmpbin/postgres" ]; then + echo "Populating tmp_install..." + meson test -C "$bdir" tmp_install install_test_files initdb_cache >/dev/null || return 1 + fi + + local wrap + wrap=$(mktemp -d /tmp/pg-vg-wrap-XXXXXX) + _pg_make_valgrind_wrapper "$tmpbin" "$wrap" + + mkdir -p "$PG_BENCH_DIR" + echo "Valgrind logs: $PG_BENCH_DIR/valgrind-*.log" + echo "Wrapper dir: $wrap (will be removed on exit)" + echo "Expect the regress suite to take 15-45 minutes under valgrind." + + local rc=0 + (cd "$bdir" && PATH="$wrap:$PATH" meson test -t 60 --print-errorlogs regress/regress) || rc=$? + + trash "$wrap" 2>/dev/null || rm -rf "$wrap" + return "$rc" +} + +pg-valgrind-test() { + local bdir="$PG_BUILD_DIR_VALGRIND" + if [ ! -x "$bdir/src/backend/postgres" ]; then + echo "Valgrind build not found; run 'pg-build-valgrind' first." >&2 + return 1 + fi + + echo "This runs the FULL postgres test suite under valgrind." + echo "Expect many hours, and tens of GB of valgrind log output." + echo "Logs: $PG_BENCH_DIR/valgrind-*.log" + local yn + read -r -p "Continue? [y/N] " yn + case "$yn" in + y | Y | yes) ;; + *) echo "Aborted."; return 0 ;; + esac + + local tmpbin="$bdir/tmp_install$PG_INSTALL_DIR-valgrind/bin" + if [ ! -x "$tmpbin/postgres" ]; then + echo "Populating tmp_install..." + meson test -C "$bdir" tmp_install install_test_files initdb_cache >/dev/null || return 1 + fi + + local wrap + wrap=$(mktemp -d /tmp/pg-vg-wrap-XXXXXX) + _pg_make_valgrind_wrapper "$tmpbin" "$wrap" + mkdir -p "$PG_BENCH_DIR" + + local rc=0 + (cd "$bdir" && PATH="$wrap:$PATH" meson test -t 60 --print-errorlogs) || rc=$? + + trash "$wrap" 2>/dev/null || rm -rf "$wrap" + return "$rc" +} + +# ============================================================ +# AddressSanitizer / UndefinedBehaviorSanitizer build and tests +# ============================================================ +pg-build-asan() { + local bdir="$PG_BUILD_DIR_ASAN" + if [ -z "$PERL_CORE_DIR" ]; then + echo "Error: PERL_CORE_DIR is not set" >&2 + return 1 + fi + + pg_clean_for_compiler "$bdir" + + echo "=== Configuring ASan+UBSan build in $bdir ===" + env CFLAGS="-Og -ggdb3 -fno-omit-frame-pointer -fsanitize=address,undefined -fno-sanitize-recover=all -I$PERL_CORE_DIR $CFLAGS" \ + LDFLAGS="-fsanitize=address,undefined -L$PERL_CORE_DIR -lperl $LDFLAGS" \ + meson setup --reconfigure \ + -Doptimization=g \ + -Ddebug=true \ + -Dcassert=true \ + -Dtap_tests=enabled \ + -Dinjection_points=true \ + -Dllvm=disabled \ + -Dplperl=enabled -Dplpython=enabled -Dpltcl=enabled \ + -Dlz4=enabled -Dzstd=enabled \ + -Dlibxml=enabled -Dlibxslt=enabled -Dssl=openssl -Duuid=e2fs \ + -Dldap=disabled \ + --prefix="$PG_INSTALL_DIR-asan" \ + "$bdir" "$PG_SOURCE_DIR" || return 1 + + meson compile -C "$bdir" +} + +pg-asan-regress() { + local bdir="$PG_BUILD_DIR_ASAN" + if [ ! -x "$bdir/src/backend/postgres" ]; then + echo "ASan build not found; run 'pg-build-asan' first." >&2 + return 1 + fi + + # halt_on_error=0 lets regress continue past the first diagnostic so + # the whole suite runs; abort_on_error=1 makes each hit fail the test. + ASAN_OPTIONS="halt_on_error=0:abort_on_error=1:detect_leaks=0:print_summary=1:print_stacktrace=1" \ + UBSAN_OPTIONS="halt_on_error=1:abort_on_error=1:print_stacktrace=1:print_summary=1" \ + meson test -t 5 --print-errorlogs -C "$bdir" regress/regress +} + +# ============================================================ +# rr (deterministic record-and-replay) +# Requires kernel.perf_event_paranoid <= 1. rr is the single most +# effective tool for postgres bugs that reproduce intermittently. +# ============================================================ +pg-rr-check() { + if ! command -v rr >/dev/null; then + echo "rr is not installed (expected in the dev shell)." >&2 + return 1 + fi + local paranoid + paranoid=$(cat /proc/sys/kernel/perf_event_paranoid 2>/dev/null || echo 99) + if [ "$paranoid" -gt 1 ]; then + echo "rr requires kernel.perf_event_paranoid <= 1; currently $paranoid" + echo "To enable (root needed):" + echo " echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid" + return 1 + fi + echo "rr ready (perf_event_paranoid=$paranoid)" +} + +pg-rr-record() { + pg-rr-check >/dev/null || { + pg-rr-check + return 1 + } + ulimit -c unlimited + rr record -- "$PG_INSTALL_DIR/bin/postgres" -D "$PG_DATA_DIR" -k "$PG_DATA_DIR" +} + +pg-rr-replay() { + rr replay "$@" +} + +# ============================================================ +# perf wrappers (parallel to the flame-graph helper) +# ============================================================ +pg-perf-record() { + local pid + pid=$(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1) + if [ -z "$pid" ]; then + echo "No postgres running under $PG_DATA_DIR" >&2 + return 1 + fi + mkdir -p "$PG_BENCH_DIR" + local out="$PG_BENCH_DIR/perf-$(date +%Y%m%d_%H%M%S).data" + echo "Recording to $out (Ctrl-C to stop)" + perf record -F 997 --call-graph dwarf -p "$pid" -o "$out" "$@" + echo "Saved: $out" +} + +pg-perf-report() { + local data + data=$(ls -t "$PG_BENCH_DIR"/perf-*.data 2>/dev/null | head -1) + if [ -z "$data" ]; then + echo "No perf data in $PG_BENCH_DIR" >&2 + return 1 + fi + echo "Reading $data" + perf report -i "$data" "$@" +} + +pg-perf-annotate() { + local data + data=$(ls -t "$PG_BENCH_DIR"/perf-*.data 2>/dev/null | head -1) + if [ -z "$data" ]; then + echo "No perf data in $PG_BENCH_DIR" >&2 + return 1 + fi + perf annotate -i "$data" "$@" +} + +# ============================================================ +# Single regression test / group runner. +# Runs pg_regress directly against the existing build so you skip the +# full meson-driven suite wrapper. Usage: pg-test-one boolean [name ...] +# ============================================================ +pg-test-one() { + if [ $# -eq 0 ]; then + echo "usage: pg-test-one TESTNAME [TESTNAME ...]" + echo "example: pg-test-one boolean" + return 2 + fi + local bdir="${PG_BUILD_DIR_ONE:-$PG_BUILD_DIR}" + local tmpbin="$bdir/tmp_install$PG_INSTALL_DIR/bin" + if [ ! -x "$tmpbin/postgres" ]; then + echo "Populating tmp_install..." + meson test -C "$bdir" tmp_install install_test_files initdb_cache >/dev/null || return 1 + fi + local outdir + outdir=$(mktemp -d /tmp/pg-test-one-XXXXXX) + echo "Test output: $outdir" + "$bdir/src/test/regress/pg_regress" \ + --bindir="$tmpbin" \ + --inputdir="$PG_SOURCE_DIR/src/test/regress" \ + --expecteddir="$PG_SOURCE_DIR/src/test/regress" \ + --dlpath="$bdir/src/test/regress" \ + --outputdir="$outdir" \ + --temp-instance="$outdir/tmp" \ + --port=40099 \ + "$@" +} + +# Full flame graph / benchmark aliases +alias pg-flame='pg-flame-generate' +alias pg-flame-30='pg-flame-generate 30' +alias pg-flame-60='pg-flame-generate 60' +alias pg-flame-120='pg-flame-generate 120' + +pg-flame-custom() { + local duration=${1:-30} + local output_dir=${2:-$PG_FLAME_DIR} + echo "Generating flame graph for ${duration}s, output to: $output_dir" + pg-flame-generate "$duration" "$output_dir" +} + +alias pg-bench='pg-bench-run' +alias pg-bench-quick='pg-bench-run 5 1 100 1 30 select-only' +alias pg-bench-standard='pg-bench-run 10 2 1000 10 60 tpcb-like' +alias pg-bench-heavy='pg-bench-run 50 4 5000 100 300 tpcb-like' +alias pg-bench-readonly='pg-bench-run 20 4 2000 50 120 select-only' + +pg-bench-custom() { + local clients=${1:-10} + local threads=${2:-2} + local transactions=${3:-1000} + local scale=${4:-10} + local duration=${5:-60} + local test_type=${6:-tpcb-like} + + echo "Running custom benchmark:" + echo " Clients: $clients, Threads: $threads" + echo " Transactions: $transactions, Scale: $scale" + echo " Duration: ${duration}s, Type: $test_type" + + pg-bench-run "$clients" "$threads" "$transactions" "$scale" "$duration" "$test_type" +} + +pg-bench-flame() { + local duration=${1:-60} + local clients=${2:-10} + local scale=${3:-10} + + echo "Running benchmark with flame graph generation" + echo "Duration: ${duration}s, Clients: $clients, Scale: $scale" + + pg-bench-run "$clients" 2 1000 "$scale" "$duration" tpcb-like & + local bench_pid=$! + + sleep 5 + + local flame_duration=$((duration - 10)) + if [ $flame_duration -gt 10 ]; then + pg-flame-generate "$flame_duration" & + local flame_pid=$! + fi + + wait $bench_pid + if [ -n "${flame_pid:-}" ]; then + wait $flame_pid + fi + + echo "Benchmark and flame graph generation completed" +} + +# Live monitoring +alias pg-perf='perf top -p $(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | head -1)' +alias pg-htop='htop -p $(pgrep -f "postgres.*-D.*$PG_DATA_DIR" | tr "\n" "," | sed "s/,$//")' + +pg-stats() { + local duration=${1:-30} + echo "Collecting system stats for ${duration}s..." + + iostat -x 1 "$duration" >"$PG_BENCH_DIR/iostat_$(date +%Y%m%d_%H%M%S).log" & + vmstat 1 "$duration" >"$PG_BENCH_DIR/vmstat_$(date +%Y%m%d_%H%M%S).log" & + + wait + echo "System stats saved to $PG_BENCH_DIR" +} + +# ============================================================ +# Code quality helpers +# ============================================================ +pg-format() { + local since=${1:-HEAD} + + if [ ! -f "$PG_SOURCE_DIR/src/tools/pgindent/pgindent" ]; then + echo "Error: pgindent not found at $PG_SOURCE_DIR/src/tools/pgindent/pgindent" + else + + modified_files=$(git diff --name-only "${since}" | grep -E "\.c$|\.h$") + + if [ -z "$modified_files" ]; then + echo "No modified .c or .h files found" + else + + echo "Formatting modified files with pgindent:" + for file in $modified_files; do + if [ -f "$file" ]; then + echo " Formatting: $file" + "$PG_SOURCE_DIR/src/tools/pgindent/pgindent" "$file" + else + echo " Warning: File not found: $file" + fi + done + + echo "Checking files for whitespace:" + git diff --check "${since}" + fi + fi +} + +pg-tidy() { + local since=${1:-HEAD} + local files + files=$(git diff --name-only "$since" | grep -E "\.(c|h)$") + if [ -z "$files" ]; then + echo "No modified .c or .h files." + return 0 + fi + for f in $files; do + [ -f "$f" ] || continue + echo "clang-tidy: $f" + clang-tidy -p "$PG_BUILD_DIR" "$f" 2>&1 | head -50 + done +} + +pg-spell() { + local since=${1:-HEAD} + local files=$(git diff --name-only "$since" | grep -E '\.(c|h|sgml|md)$') + if [ -z "$files" ]; then + echo "No .c/.h/.sgml/.md files changed since $since" + return 0 + fi + for f in $files; do + [ -f "$f" ] || continue + case "$f" in + *.c | *.h) + grep -nE '^\s*(/\*|\*|//)' "$f" | codespell --stdin-single-line - 2>/dev/null \ + && echo " $f: ok" || true + ;; + *.sgml | *.md) + codespell "$f" || true + ;; + esac + done +} + +# ============================================================ +# Core dump one-shots (one-time, requires root). kernel.core_pattern +# is a system-wide sysctl -- we don't touch it on every shell entry. +# ============================================================ +pg-cores-status() { + echo "ulimit -c: $(ulimit -c)" + echo "kernel.core_pattern: $(cat /proc/sys/kernel/core_pattern 2>/dev/null || echo unreadable)" + echo "cwd: $(pwd)" +} + +pg-enable-cores() { + ulimit -c unlimited + if ! [ -w /proc/sys/kernel/core_pattern ]; then + echo "Setting kernel.core_pattern (requires sudo)..." + echo "core.%p" | sudo tee /proc/sys/kernel/core_pattern >/dev/null || { + echo "Failed to write /proc/sys/kernel/core_pattern" >&2 + return 1 + } + else + echo "core.%p" >/proc/sys/kernel/core_pattern + fi + pg-cores-status +} + +pg-disable-cores() { + ulimit -c 0 + if ! [ -w /proc/sys/kernel/core_pattern ]; then + echo "Restoring kernel.core_pattern to 'core' (requires sudo)..." + echo "core" | sudo tee /proc/sys/kernel/core_pattern >/dev/null || { + echo "Failed to restore /proc/sys/kernel/core_pattern" >&2 + return 1 + } + else + echo "core" >/proc/sys/kernel/core_pattern + fi + pg-cores-status +} + +# ============================================================ +# Logs and results +# ============================================================ +alias pg-log='tail -f "$PG_DATA_DIR/log/postgresql-$(date +%Y-%m-%d).log" 2>/dev/null || echo "No log file found"' +alias pg-log-errors='grep -i error "$PG_DATA_DIR/log/"*.log 2>/dev/null || echo "No error logs found"' + +alias pg-build-log='cat "$PG_BUILD_DIR/meson-logs/meson-log.txt"' +alias pg-build-errors='grep -i error "$PG_BUILD_DIR/meson-logs/meson-log.txt" 2>/dev/null || echo "No build errors found"' + +alias pg-bench-results='ls -la "$PG_BENCH_DIR" && echo "Latest results:" && tail -20 "$PG_BENCH_DIR"/results_*.txt 2>/dev/null | tail -20' +alias pg-flame-results='ls -la "$PG_FLAME_DIR" && echo "Open flame graphs with: firefox $PG_FLAME_DIR/*.svg"' + +pg-clean-results() { + local days=${1:-7} + echo "Cleaning benchmark and flame graph results older than $days days..." + find "$PG_BENCH_DIR" -type f -mtime +$days -delete 2>/dev/null || true + find "$PG_FLAME_DIR" -type f -mtime +$days -delete 2>/dev/null || true + echo "Cleanup completed" +} + +# ============================================================ +# Info +# ============================================================ +alias pg-info=' + echo "=== PostgreSQL Development Environment ===" + echo "Source: $PG_SOURCE_DIR" + echo "Build (default): $PG_BUILD_DIR" + echo "Build (valgrind):$PG_BUILD_DIR_VALGRIND" + echo "Build (asan): $PG_BUILD_DIR_ASAN" + echo "Install: $PG_INSTALL_DIR" + echo "Data: $PG_DATA_DIR" + echo "Benchmarks: $PG_BENCH_DIR" + echo "Flame graphs: $PG_FLAME_DIR" + echo "Compiler: $CC" + echo "" + echo "Available commands:" + echo " Setup/build: pg-setup, pg-build, pg-install" + echo " Database: pg-init, pg-start, pg-stop, pg-psql" + echo " Tests: pg-test, pg-test-one NAME" + echo " Valgrind: pg-build-valgrind, pg-valgrind-regress, pg-valgrind-test" + echo " ASan/UBSan: pg-build-asan, pg-asan-regress" + echo " Debug: pg-debug, pg-attach" + echo " libumem: pg-start-umem, pg-umem-ctl, pg-umem-stats/leaks/caches, pg-umem-gdb/lldb" + echo " Record/replay: pg-rr-check, pg-rr-record, pg-rr-replay" + echo " Perf: pg-perf-record, pg-perf-report, pg-perf-annotate, pg-perf" + echo " Flame graphs: pg-flame, pg-flame-30, pg-flame-60, pg-flame-custom" + echo " Benchmarks: pg-bench-quick, pg-bench-standard, pg-bench-heavy" + echo " Combined: pg-bench-flame" + echo " Results: pg-bench-results, pg-flame-results" + echo " Logs: pg-log, pg-build-log" + echo " Clean: pg-clean, pg-full-clean, pg-clean-results" + echo " Code quality: pg-format, pg-tidy, pg-spell" + echo " Cores: pg-enable-cores, pg-disable-cores, pg-cores-status" + echo "=========================================="' + +echo "PostgreSQL aliases loaded. Run 'pg-info' for available commands." diff --git a/shell.nix b/shell.nix new file mode 100644 index 0000000000000..d1cd518c00382 --- /dev/null +++ b/shell.nix @@ -0,0 +1,729 @@ +{ + pkgs, + pkgs-unstable, + system, + libumemPkg, +}: let + # Create a patched glibc only for the dev shell. + # + # Glibc's features.h emits a `-Wcpp` diagnostic when _FORTIFY_SOURCE is + # defined without an optimization level. Meson's dependency probes + # (notably the libcurl thread-safety check) compile small snippets with + # `-O0 -Werror` under the nixpkgs hardening defaults, which turns that cpp + # warning into a hard error and breaks reconfigure. The patch drops the + # warning. It is scoped to this dev shell only and never leaks into system + # glibc or release builds. + # Patch glibc's features.h to drop a -Wcpp warning. + # + # Glibc's features.h emits a `-Wcpp` diagnostic when _FORTIFY_SOURCE is + # defined without an optimization level. Meson's dependency probes + # (notably the libcurl thread-safety check) compile small snippets with + # `-O0 -Werror` under the nixpkgs hardening defaults, which turns that cpp + # warning into a hard error and breaks reconfigure. The patch drops the + # warning. It is scoped to this dev shell only and never leaks into system + # glibc or release builds. Kept on the stable nixpkgs glibc so it matches + # the shell's base tools and the (stable-following) libumem input. + patchedGlibc = pkgs.glibc.overrideAttrs (oldAttrs: { + patches = + (oldAttrs.patches or []) + ++ [ + ./glibc-no-fortify-warning.patch + ]; + }); + + # Modern toolchain, pinned to nixpkgs-unstable. + gccPkg = pkgs-unstable.gcc16; + gdbPkg = pkgs-unstable.gdb; + llvmPkgs = pkgs-unstable.llvmPackages_22; + + # libumem debugger helpers (installed by the flake package under + # $out/share/umem/debugger/{gdb,lldb}). + umemGdbHelper = "${libumemPkg}/share/umem/debugger/gdb/umem_gdb.py"; + umemLldbHelper = "${libumemPkg}/share/umem/debugger/lldb/umem_lldb.py"; + umemPreload = "${libumemPkg}/lib/libumem_malloc.so"; + + # Configuration constants + config = { + pgSourceDir = "$PWD"; + pgBuildDir = "$PWD/build"; + pgBuildDirValgrind = "$PWD/build-valgrind"; + pgBuildDirAsan = "$PWD/build-asan"; + pgInstallDir = "$PWD/install"; + pgDataDir = "/tmp/test-db-$(basename $PWD)"; + pgBenchDir = "/tmp/pgbench-results-$(basename $PWD)"; + pgFlameDir = "/tmp/flame-graphs-$(basename $PWD)"; + }; + + # PostgreSQL build + development dependencies (glibc target). + postgreSQLDeps = with pkgs; + [ + # Build system (always use host tools) + pkgs-unstable.meson + pkgs-unstable.ninja + pkg-config + autoconf + git + which + binutils + gnumake + mold # fast linker, big wins on large postgres links + + # Parser/lexer tools + bison + flex + + gettext # msgfmt/xgettext for NLS (translation) builds + + # Perl with required packages + (perl.withPackages (ps: with ps; [IPCRun])) + + # Documentation + docbook_xml_dtd_45 + docbook-xsl-nons + libxslt + libxml2 + fop + + # Development tools (always use host tools) + coreutils + shellcheck + ripgrep + valgrind + curl + uv + pylint + black + lcov + strace + ltrace + perf-tools + linuxPackages.perf + flamegraph + bpftrace # kernel-level tracing (probes, uprobes) + rr # record-and-replay deterministic debugger + htop + iotop + sysstat + ccache + cppcheck + compdb + + # Spell checking + aspell + aspellDicts.en + codespell + + # GCC 16 + GDB (nixpkgs-unstable) + gccPkg + gdbPkg + + # LLVM 22 toolchain + llvmPkgs.llvm + llvmPkgs.llvm.dev + llvmPkgs.clang-tools + llvmPkgs.lldb + + # libumem: LD_PRELOAD malloc + umem/umemctl CLI + debugger helpers + libumemPkg + + # Language support + (python3.withPackages (ps: with ps; [requests browser-cookie3])) + tcl + ] + ++ [ + # Glibc target libraries + readline + zlib + openssl + icu + lz4 + zstd + libuuid + libkrb5 + linux-pam + libxcrypt + numactl + openldap + liburing + libselinux + patchedGlibc + patchedGlibc.dev + ]; + + # GDB configuration for PostgreSQL debugging + gdbConfig = pkgs.writeText "gdbinit-postgres" '' + # PostgreSQL-specific GDB configuration + + # Pretty-print PostgreSQL data structures + define print_node + if $arg0 + printf "Node type: %s\n", nodeTagNames[$arg0->type] + print *$arg0 + else + printf "NULL node\n" + end + end + document print_node + Print a PostgreSQL Node with type information + Usage: print_node + end + + define print_list + set $list = (List*)$arg0 + if $list + printf "List length: %d\n", $list->length + set $cell = $list->head + set $i = 0 + while $cell && $i < $list->length + printf " [%d]: ", $i + print_node $cell->data.ptr_value + set $cell = $cell->next + set $i = $i + 1 + end + else + printf "NULL list\n" + end + end + document print_list + Print a PostgreSQL List structure + Usage: print_list + end + + define print_query + set $query = (Query*)$arg0 + if $query + printf "Query type: %d, command type: %d\n", $query->querySource, $query->commandType + print *$query + else + printf "NULL query\n" + end + end + document print_query + Print a PostgreSQL Query structure + Usage: print_query + end + + define print_relcache + set $rel = (Relation)$arg0 + if $rel + printf "Relation: %s.%s (OID: %u)\n", $rel->rd_rel->relnamespace, $rel->rd_rel->relname.data, $rel->rd_id + printf " natts: %d, relkind: %c\n", $rel->rd_rel->relnatts, $rel->rd_rel->relkind + else + printf "NULL relation\n" + end + end + document print_relcache + Print relation cache entry information + Usage: print_relcache + end + + define print_tupdesc + set $desc = (TupleDesc)$arg0 + if $desc + printf "TupleDesc: %d attributes\n", $desc->natts + set $i = 0 + while $i < $desc->natts + set $attr = $desc->attrs[$i] + printf " [%d]: %s (type: %u, len: %d)\n", $i, $attr->attname.data, $attr->atttypid, $attr->attlen + set $i = $i + 1 + end + else + printf "NULL tuple descriptor\n" + end + end + document print_tupdesc + Print tuple descriptor information + Usage: print_tupdesc + end + + define print_slot + set $slot = (TupleTableSlot*)$arg0 + if $slot + printf "TupleTableSlot: %s\n", $slot->tts_ops->name + printf " empty: %d, shouldFree: %d\n", $slot->tts_empty, $slot->tts_shouldFree + if $slot->tts_tupleDescriptor + print_tupdesc $slot->tts_tupleDescriptor + end + else + printf "NULL slot\n" + end + end + document print_slot + Print tuple table slot information + Usage: print_slot + end + + # Memory context debugging + define print_mcxt + set $context = (MemoryContext)$arg0 + if $context + printf "MemoryContext: %s\n", $context->name + printf " type: %s, parent: %p\n", $context->methods->name, $context->parent + printf " total: %zu, free: %zu\n", $context->mem_allocated, $context->freep - $context->freeptr + else + printf "NULL memory context\n" + end + end + document print_mcxt + Print memory context information + Usage: print_mcxt + end + + # Process debugging + define print_proc + set $proc = (PGPROC*)$arg0 + if $proc + printf "PGPROC: pid=%d, database=%u\n", $proc->pid, $proc->databaseId + printf " waiting: %d, waitStatus: %d\n", $proc->waiting, $proc->waitStatus + else + printf "NULL process\n" + end + end + document print_proc + Print process information + Usage: print_proc + end + + # Set useful defaults + set print pretty on + set print object on + set print static-members off + set print vtbl on + set print demangle on + set demangle-style gnu-v3 + set print sevenbit-strings off + set history save on + set history size 1000 + set history filename ~/.gdb_history_postgres + + # Common breakpoints for PostgreSQL debugging + define pg_break_common + break elog + break errfinish + break ExceptionalCondition + break ProcessInterrupts + end + document pg_break_common + Set common PostgreSQL debugging breakpoints + end + + # libumem GDB helpers (umem heap/event inspection). Loaded if present; + # provides the 'umem' command family (umem break, umem events, ...). + python + try: + gdb.execute("source ${umemGdbHelper}") + except Exception as e: + pass + end + + printf "PostgreSQL GDB configuration loaded.\n" + printf "Available commands: print_node, print_list, print_query, print_relcache,\n" + printf " print_tupdesc, print_slot, print_mcxt, print_proc, pg_break_common\n" + ''; + + # Flame graph generation script + flameGraphScript = pkgs.writeScriptBin "pg-flame-generate" '' + #!${pkgs.bash}/bin/bash + set -euo pipefail + + DURATION=''${1:-30} + OUTPUT_DIR=''${2:-${config.pgFlameDir}} + TIMESTAMP=$(date +%Y%m%d_%H%M%S) + + mkdir -p "$OUTPUT_DIR" + + echo "Generating flame graph for PostgreSQL (duration: ''${DURATION}s)" + + # Find PostgreSQL processes + PG_PIDS=$(pgrep -f "postgres.*-D.*${config.pgDataDir}" || true) + + if [ -z "$PG_PIDS" ]; then + echo "Error: No PostgreSQL processes found" + exit 1 + fi + + echo "Found PostgreSQL processes: $PG_PIDS" + + # Record perf data + PERF_DATA="$OUTPUT_DIR/perf_$TIMESTAMP.data" + echo "Recording perf data to $PERF_DATA" + + ${pkgs.linuxPackages.perf}/bin/perf record \ + -F 997 \ + -g \ + --call-graph dwarf \ + -p "$(echo $PG_PIDS | tr ' ' ',')" \ + -o "$PERF_DATA" \ + sleep "$DURATION" + + # Generate flame graph + FLAME_SVG="$OUTPUT_DIR/postgres_flame_$TIMESTAMP.svg" + echo "Generating flame graph: $FLAME_SVG" + + ${pkgs.linuxPackages.perf}/bin/perf script -i "$PERF_DATA" | \ + ${pkgs.flamegraph}/bin/stackcollapse-perf.pl | \ + ${pkgs.flamegraph}/bin/flamegraph.pl \ + --title "PostgreSQL Flame Graph ($TIMESTAMP)" \ + --width 1200 \ + --height 800 \ + > "$FLAME_SVG" + + echo "Flame graph generated: $FLAME_SVG" + echo "Perf data saved: $PERF_DATA" + + # Generate summary report + REPORT="$OUTPUT_DIR/report_$TIMESTAMP.txt" + echo "Generating performance report: $REPORT" + + { + echo "PostgreSQL Performance Analysis Report" + echo "Generated: $(date)" + echo "Duration: ''${DURATION}s" + echo "Processes: $PG_PIDS" + echo "" + echo "=== Top Functions ===" + ${pkgs.linuxPackages.perf}/bin/perf report -i "$PERF_DATA" --stdio --sort comm,dso,symbol | head -50 + echo "" + echo "=== Call Graph ===" + ${pkgs.linuxPackages.perf}/bin/perf report -i "$PERF_DATA" --stdio -g --sort comm,dso,symbol | head -100 + } > "$REPORT" + + echo "Report generated: $REPORT" + echo "" + echo "Files created:" + echo " Flame graph: $FLAME_SVG" + echo " Perf data: $PERF_DATA" + echo " Report: $REPORT" + ''; + + # pgbench wrapper script + pgbenchScript = pkgs.writeScriptBin "pg-bench-run" '' + #!${pkgs.bash}/bin/bash + set -euo pipefail + + # Default parameters + CLIENTS=''${1:-10} + THREADS=''${2:-2} + TRANSACTIONS=''${3:-1000} + SCALE=''${4:-10} + DURATION=''${5:-60} + TEST_TYPE=''${6:-tpcb-like} + + OUTPUT_DIR="${config.pgBenchDir}" + TIMESTAMP=$(date +%Y%m%d_%H%M%S) + + mkdir -p "$OUTPUT_DIR" + + echo "=== PostgreSQL Benchmark Configuration ===" + echo "Clients: $CLIENTS" + echo "Threads: $THREADS" + echo "Transactions: $TRANSACTIONS" + echo "Scale factor: $SCALE" + echo "Duration: ''${DURATION}s" + echo "Test type: $TEST_TYPE" + echo "Output directory: $OUTPUT_DIR" + echo "============================================" + + # Check if PostgreSQL is running + if ! pgrep -f "postgres.*-D.*${config.pgDataDir}" >/dev/null; then + echo "Error: PostgreSQL is not running. Start it with 'pg-start'" + exit 1 + fi + + PGBENCH="${config.pgInstallDir}/bin/pgbench" + PSQL="${config.pgInstallDir}/bin/psql" + CREATEDB="${config.pgInstallDir}/bin/createdb" + DROPDB="${config.pgInstallDir}/bin/dropdb" + + DB_NAME="pgbench_test_$TIMESTAMP" + RESULTS_FILE="$OUTPUT_DIR/results_$TIMESTAMP.txt" + LOG_FILE="$OUTPUT_DIR/pgbench_$TIMESTAMP.log" + + echo "Creating test database: $DB_NAME" + "$CREATEDB" -h "${config.pgDataDir}" "$DB_NAME" || { + echo "Failed to create database" + exit 1 + } + + # Initialize pgbench tables + echo "Initializing pgbench tables (scale factor: $SCALE)" + "$PGBENCH" -h "${config.pgDataDir}" -i -s "$SCALE" "$DB_NAME" || { + echo "Failed to initialize pgbench tables" + "$DROPDB" -h "${config.pgDataDir}" "$DB_NAME" 2>/dev/null || true + exit 1 + } + + # Run benchmark based on test type + echo "Running benchmark..." + + case "$TEST_TYPE" in + "tpcb-like"|"default") + BENCH_ARGS="" + ;; + "select-only") + BENCH_ARGS="-S" + ;; + "simple-update") + BENCH_ARGS="-N" + ;; + "read-write") + BENCH_ARGS="-b select-only@70 -b tpcb-like@30" + ;; + *) + echo "Unknown test type: $TEST_TYPE" + echo "Available types: tpcb-like, select-only, simple-update, read-write" + "$DROPDB" -h "${config.pgDataDir}" "$DB_NAME" 2>/dev/null || true + exit 1 + ;; + esac + + { + echo "PostgreSQL Benchmark Results" + echo "Generated: $(date)" + echo "Test type: $TEST_TYPE" + echo "Clients: $CLIENTS, Threads: $THREADS" + echo "Transactions: $TRANSACTIONS, Duration: ''${DURATION}s" + echo "Scale factor: $SCALE" + echo "Database: $DB_NAME" + echo "" + echo "=== System Information ===" + echo "CPU: $(nproc) cores" + echo "Memory: $(free -h | grep '^Mem:' | awk '{print $2}')" + echo "Compiler: $CC" + echo "PostgreSQL version: $("$PSQL" --no-psqlrc -h "${config.pgDataDir}" -d "$DB_NAME" -t -c "SELECT version();" | head -1)" + echo "" + echo "=== Benchmark Results ===" + } > "$RESULTS_FILE" + + # Run the actual benchmark + "$PGBENCH" \ + -h "${config.pgDataDir}" \ + -c "$CLIENTS" \ + -j "$THREADS" \ + -T "$DURATION" \ + -P 5 \ + --log \ + --log-prefix="$OUTPUT_DIR/pgbench_$TIMESTAMP" \ + $BENCH_ARGS \ + "$DB_NAME" 2>&1 | tee -a "$RESULTS_FILE" + + # Collect additional statistics + { + echo "" + echo "=== Database Statistics ===" + "$PSQL" --no-psqlrc -h "${config.pgDataDir}" -d "$DB_NAME" -c " + SELECT + schemaname, + relname, + n_tup_ins as inserts, + n_tup_upd as updates, + n_tup_del as deletes, + n_live_tup as live_tuples, + n_dead_tup as dead_tuples + FROM pg_stat_user_tables; + " + + echo "" + echo "=== Index Statistics ===" + "$PSQL" --no-psqlrc -h "${config.pgDataDir}" -d "$DB_NAME" -c " + SELECT + schemaname, + relname, + indexrelname, + idx_scan, + idx_tup_read, + idx_tup_fetch + FROM pg_stat_user_indexes; + " + } >> "$RESULTS_FILE" + + # Clean up + echo "Cleaning up test database: $DB_NAME" + "$DROPDB" -h "${config.pgDataDir}" "$DB_NAME" 2>/dev/null || true + + echo "" + echo "Benchmark completed!" + echo "Results saved to: $RESULTS_FILE" + echo "Transaction logs: $OUTPUT_DIR/pgbench_$TIMESTAMP*" + + # Show summary + echo "" + echo "=== Quick Summary ===" + grep -E "(tps|latency)" "$RESULTS_FILE" | tail -5 + ''; + + # LD_PRELOAD libumem's malloc into a command: pg-umem-preload + # A script (not a shell function) so it survives subshells and non-login + # invocations. Preloads $out/lib/libumem_malloc.so ahead of any existing + # LD_PRELOAD. + umemPreloadScript = pkgs.writeScriptBin "pg-umem-preload" '' + #!${pkgs.bash}/bin/bash + if [ $# -eq 0 ]; then + echo "usage: pg-umem-preload [args...]" >&2 + echo " runs with libumem_malloc.so LD_PRELOAD'ed" >&2 + exit 2 + fi + export LD_PRELOAD="${umemPreload}''${LD_PRELOAD:+:$LD_PRELOAD}" + exec "$@" + ''; + + # Shared shellHook fragments. Each devShell prepends its own compiler/CFLAGS + # block, then appends the common tail via ${commonHookTail variant}. + commonHookHead = icon: '' + # History configuration + export HISTFILE=.history + export HISTSIZE=1000000 + export HISTFILESIZE=1000000 + + # Clean environment + unset LD_LIBRARY_PATH LD_PRELOAD LIBRARY_PATH C_INCLUDE_PATH CPLUS_INCLUDE_PATH + + # Essential tools in PATH + export PATH="${pkgs.which}/bin:${pkgs.coreutils}/bin:$PATH" + export PS1="$(echo -e '\u${icon}') {\[$(tput sgr0)\]\[\033[38;5;228m\]\w\[$(tput sgr0)\]\[\033[38;5;15m\]} ($(git rev-parse --abbrev-ref HEAD)) \\$ \[$(tput sgr0)\]" + + # Ccache configuration + export PATH=${pkgs.ccache}/bin:$PATH + export CCACHE_COMPILERCHECK=content + # Loosen a few rules so ccache hits across rebuilds with touched headers. + export CCACHE_SLOPPINESS=pch_defines,time_macros,include_file_mtime,include_file_ctime + export CCACHE_DIR=$HOME/.ccache/pg/$(basename $PWD) + mkdir -p "$CCACHE_DIR" + + # Development tools in PATH + export PATH=${pkgs.clang-tools}/bin:$PATH + export PATH=${pkgs.cppcheck}/bin:$PATH + ''; + + # Tail shared by every devShell: PG env vars, GDB, tool PATH, per-process + # setup and alias load. Kernel core_pattern is NOT touched here -- + # run 'pg-enable-cores' explicitly if you need per-PID cores in CWD. + commonHookTail = label: '' + # PostgreSQL environment + export PG_SOURCE_DIR="${config.pgSourceDir}" + export PG_BUILD_DIR="${config.pgBuildDir}" + export PG_BUILD_DIR_VALGRIND="${config.pgBuildDirValgrind}" + export PG_BUILD_DIR_ASAN="${config.pgBuildDirAsan}" + export PG_INSTALL_DIR="${config.pgInstallDir}" + export PG_DATA_DIR="${config.pgDataDir}" + export PG_BENCH_DIR="${config.pgBenchDir}" + export PG_FLAME_DIR="${config.pgFlameDir}" + export PERL_CORE_DIR=$(find ${pkgs.perl} -maxdepth 5 -path "*/CORE" -type d) + + # GDB configuration + export GDBINIT="${gdbConfig}" + + # libumem: LD_PRELOAD malloc, the umem/umemctl CLI, and debugger helpers. + # The CLI and 'pg-umem-preload' are already on PATH (buildInputs). Export + # the preload object and helper paths; do NOT preload by default -- opt in + # per-command with 'pg-umem-preload ' or export + # LD_PRELOAD="$PG_UMEM_PRELOAD". + export PG_UMEM_PRELOAD="${umemPreload}" + export PG_UMEM_GDB_HELPER="${umemGdbHelper}" + export PG_UMEM_LLDB_HELPER="${umemLldbHelper}" + + # Performance tools in PATH + export PATH="${flameGraphScript}/bin:${pgbenchScript}/bin:${umemPreloadScript}/bin:$PATH" + + # Create output directories + mkdir -p "$PG_BENCH_DIR" "$PG_FLAME_DIR" + + # Per-process core dump size limit. Kernel core_pattern is NOT + # touched here -- run 'pg-enable-cores' explicitly when you need + # per-PID cores in CWD. + ulimit -c unlimited + + # Local git excludes + git config core.excludesFile .local-gitignore 2>/dev/null || true + + # Load PostgreSQL development aliases + if [ -f ./pg-aliases.sh ]; then + source ./pg-aliases.sh + else + echo "Warning: pg-aliases.sh not found in current directory" + fi + + echo "" + echo "PostgreSQL Development Environment Ready (${label})" + echo "Run 'pg-info' for available commands" + echo "libumem: umem/umemctl on PATH; 'pg-umem-preload ' to LD_PRELOAD libumem_malloc" + ''; + + # Development shell (GCC 16 + glibc) + devShell = pkgs.mkShell { + name = "postgresql-dev"; + buildInputs = + postgreSQLDeps + ++ [ + flameGraphScript + pgbenchScript + umemPreloadScript + ]; + + shellHook = + (commonHookHead "f121") + + '' + # LLVM configuration + export LLVM_CONFIG="${llvmPkgs.llvm}/bin/llvm-config" + export PATH="${llvmPkgs.llvm}/bin:$PATH" + export PKG_CONFIG_PATH="${llvmPkgs.llvm.dev}/lib/pkgconfig:$PKG_CONFIG_PATH" + export LLVM_DIR="${llvmPkgs.llvm.dev}/lib/cmake/llvm" + export LLVM_ROOT="${llvmPkgs.llvm}" + + # PostgreSQL Development CFLAGS + export CFLAGS="" + export CXXFLAGS="" + + # Python UV + UV_PYTHON_DOWNLOADS=never + + # GCC 16 (default compiler) + export CC="${gccPkg}/bin/gcc" + export CXX="${gccPkg}/bin/g++" + + echo "Environment configured:" + echo " Compiler: $CC ($($CC -dumpversion 2>/dev/null))" + echo " libc: glibc" + echo " LLVM: $(llvm-config --version 2>/dev/null || echo 'not available')" + '' + + (commonHookTail "GCC 16 + glibc"); + }; + + # Clang 22 + glibc variant + clangDevShell = pkgs.mkShell { + name = "postgresql-clang-glibc"; + buildInputs = + postgreSQLDeps + ++ [ + llvmPkgs.clang + llvmPkgs.lld + llvmPkgs.compiler-rt + flameGraphScript + pgbenchScript + umemPreloadScript + ]; + + shellHook = + (commonHookHead "f121") + + '' + # LLVM configuration + export LLVM_CONFIG="${llvmPkgs.llvm}/bin/llvm-config" + export PATH="${llvmPkgs.llvm}/bin:$PATH" + export PKG_CONFIG_PATH="${llvmPkgs.llvm.dev}/lib/pkgconfig:$PKG_CONFIG_PATH" + export LLVM_DIR="${llvmPkgs.llvm.dev}/lib/cmake/llvm" + export LLVM_ROOT="${llvmPkgs.llvm}" + + # Clang 22 + glibc + export CC="${llvmPkgs.clang}/bin/clang" + export CXX="${llvmPkgs.clang}/bin/clang++" + + echo "Environment configured:" + echo " Compiler: $CC ($($CC -dumpversion 2>/dev/null))" + echo " libc: glibc" + echo " LLVM: $(llvm-config --version 2>/dev/null || echo 'not available')" + '' + + (commonHookTail "Clang 22 + glibc"); + }; +in { + inherit devShell clangDevShell gdbConfig flameGraphScript pgbenchScript; +} diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c index 13944701bc7b0..278cbfb735ce0 100644 --- a/src/test/regress/pg_regress.c +++ b/src/test/regress/pg_regress.c @@ -1243,7 +1243,7 @@ spawn_process(const char *cmdline) char *cmdline2; cmdline2 = psprintf("exec %s", cmdline); - execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); + execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); /* Not using the normal bail() here as we want _exit */ bail_noatexit("could not exec \"%s\": %m", shellprog); } diff --git a/src/tools/pgindent/pgindent b/src/tools/pgindent/pgindent index f75c006bbff43..5e79bea47cf96 100755 --- a/src/tools/pgindent/pgindent +++ b/src/tools/pgindent/pgindent @@ -1,4 +1,4 @@ -#!/usr/bin/perl +#!/usr/bin/env perl # Copyright (c) 2021-2026, PostgreSQL Global Development Group From 17721b3e6c9bda52890571f3ccb8628bfca9c52d Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Thu, 1 Oct 2026 10:19:05 -0400 Subject: [PATCH 04/12] dev: drop patched glibc, fix pg-test-one/pg-format gcc16 (nixpkgs-unstable) links glibc 2.42 but the stable patchedGlibc in buildInputs put a 2.40 libc.so.6 first in RUNPATH, so every meson sanity-check binary failed with a GLIBC_PRIVATE symbol lookup error. The patched glibc only existed to silence the fortify-without-O warning; hardeningDisable does that without a second glibc. pg-test-one: set LD_LIBRARY_PATH to the tmp_install lib dir, as the meson test runner does, so initdb finds libpq. pg-format: put the in-tree pg_bsd_indent build on PATH. --- pg-aliases.sh | 3 +++ shell.nix | 34 +++++----------------------------- 2 files changed, 8 insertions(+), 29 deletions(-) diff --git a/pg-aliases.sh b/pg-aliases.sh index 42bddff4c2730..7979ec292620a 100644 --- a/pg-aliases.sh +++ b/pg-aliases.sh @@ -451,6 +451,7 @@ pg-test-one() { local outdir outdir=$(mktemp -d /tmp/pg-test-one-XXXXXX) echo "Test output: $outdir" + LD_LIBRARY_PATH="$bdir/tmp_install$PG_INSTALL_DIR/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \ "$bdir/src/test/regress/pg_regress" \ --bindir="$tmpbin" \ --inputdir="$PG_SOURCE_DIR/src/test/regress" \ @@ -548,6 +549,8 @@ pg-format() { if [ ! -f "$PG_SOURCE_DIR/src/tools/pgindent/pgindent" ]; then echo "Error: pgindent not found at $PG_SOURCE_DIR/src/tools/pgindent/pgindent" else + # pgindent needs pg_bsd_indent; meson builds it in-tree. + export PATH="$PG_BUILD_DIR/src/tools/pg_bsd_indent:$PATH" modified_files=$(git diff --name-only "${since}" | grep -E "\.c$|\.h$") diff --git a/shell.nix b/shell.nix index d1cd518c00382..d62e76802e5a6 100644 --- a/shell.nix +++ b/shell.nix @@ -4,33 +4,6 @@ system, libumemPkg, }: let - # Create a patched glibc only for the dev shell. - # - # Glibc's features.h emits a `-Wcpp` diagnostic when _FORTIFY_SOURCE is - # defined without an optimization level. Meson's dependency probes - # (notably the libcurl thread-safety check) compile small snippets with - # `-O0 -Werror` under the nixpkgs hardening defaults, which turns that cpp - # warning into a hard error and breaks reconfigure. The patch drops the - # warning. It is scoped to this dev shell only and never leaks into system - # glibc or release builds. - # Patch glibc's features.h to drop a -Wcpp warning. - # - # Glibc's features.h emits a `-Wcpp` diagnostic when _FORTIFY_SOURCE is - # defined without an optimization level. Meson's dependency probes - # (notably the libcurl thread-safety check) compile small snippets with - # `-O0 -Werror` under the nixpkgs hardening defaults, which turns that cpp - # warning into a hard error and breaks reconfigure. The patch drops the - # warning. It is scoped to this dev shell only and never leaks into system - # glibc or release builds. Kept on the stable nixpkgs glibc so it matches - # the shell's base tools and the (stable-following) libumem input. - patchedGlibc = pkgs.glibc.overrideAttrs (oldAttrs: { - patches = - (oldAttrs.patches or []) - ++ [ - ./glibc-no-fortify-warning.patch - ]; - }); - # Modern toolchain, pinned to nixpkgs-unstable. gccPkg = pkgs-unstable.gcc16; gdbPkg = pkgs-unstable.gdb; @@ -146,8 +119,6 @@ openldap liburing libselinux - patchedGlibc - patchedGlibc.dev ]; # GDB configuration for PostgreSQL debugging @@ -652,6 +623,10 @@ # Development shell (GCC 16 + glibc) devShell = pkgs.mkShell { name = "postgresql-dev"; + # Meson probes compile with -O0 -Werror; glibc's fortify-without-O + # warning would fail them. Drop fortify rather than carry a patched + # glibc, which collided with the glibc gcc16 (unstable) links against. + hardeningDisable = ["fortify" "fortify3"]; buildInputs = postgreSQLDeps ++ [ @@ -692,6 +667,7 @@ # Clang 22 + glibc variant clangDevShell = pkgs.mkShell { name = "postgresql-clang-glibc"; + hardeningDisable = ["fortify" "fortify3"]; buildInputs = postgreSQLDeps ++ [ From 6e9d59a3f6b8762e7d3c0cb5c869c180adf908ef Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Sat, 24 Oct 2026 14:32:11 -0400 Subject: [PATCH 05/12] Compute the distance between a point and a box exactly box_closept_point(), behind <->(point, box), <->(box, point) and ##(point, box), found the closest point by projecting the point onto each edge's line with the generic line-segment code. That code compares slopes and intersections with the EPSILON fuzz, so for a point near an edge's line it can pick the wrong edge or corner. The point (0.9999996, 2) is 1 from the box ((0,0),(1,1)), but the operator said 1.00000000000008, and (1e-300, -1e-300) is 1e-300 from ((2,2),(0,0)), but the operator said 1.41421356237e-300, the distance to the corner. The GiST and SP-GiST box opclasses compute the distance to a box their own way, exactly, and order a nearest-neighbor scan by that. So an index scan could return rows in an order the operator disagrees with: for the point above, a box 1.00000000000004 away came after the box the operator put at 1.00000000000008. A box's edges are parallel to the axes, so the closest point is the point clamped to the box, and the distance follows from that with no line arithmetic. Do that, and have GiST's and SP-GiST's distance to a box (computeDistance() for non-leaf entries and point_box_distance()) call the same function, so the index orders by the operator's own distance. SP-GiST's version now errors on overflow, as GiST's and the operator always did, and returns NaN when any coordinate is NaN. The geometry regression test changes in four rows, all corrections: the 1e-300 distance above, its closest point, and the closest points to (1e+300,Infinity) and (Infinity,1e+300), which are now the box's top right corner rather than its top left. This is independent of the rest of the series, which uses it to let the box opclasses return their distances. Author: Greg Burd Discussion: https://postgr.es/m/8s5lT8erXzBMugXJQ6Wginbp_gc2B4hmCYhF0Q0GpnuK87eCAOcKs5K31AWCwraCNFIQt42wwkow_MPPubHg485MWv-zDBY4NL_JX9yJEEg=@burd.me --- src/backend/access/gist/gistproc.c | 63 ++----------- src/backend/access/spgist/spgproc.c | 28 ------ src/backend/utils/adt/geo_ops.c | 78 ++++++++-------- src/include/utils/geo_decls.h | 2 + src/test/regress/expected/box.out | 35 ++++++++ src/test/regress/expected/geometry.out | 118 ++++++++++++------------- src/test/regress/sql/box.sql | 24 +++++ 7 files changed, 162 insertions(+), 186 deletions(-) diff --git a/src/backend/access/gist/gistproc.c b/src/backend/access/gist/gistproc.c index f1044f49d6c2d..fae9c7757e91d 100644 --- a/src/backend/access/gist/gistproc.c +++ b/src/backend/access/gist/gistproc.c @@ -1223,67 +1223,18 @@ gist_point_fetch(PG_FUNCTION_ARGS) static float8 computeDistance(bool isLeaf, BOX *box, Point *point) { - float8 result = 0.0; - if (isLeaf) { /* simple point to point distance */ - result = point_point_distance(point, &box->low); - } - else if (point->x <= box->high.x && point->x >= box->low.x && - point->y <= box->high.y && point->y >= box->low.y) - { - /* point inside the box */ - result = 0.0; - } - else if (point->x <= box->high.x && point->x >= box->low.x) - { - /* point is over or below box */ - Assert(box->low.y <= box->high.y); - if (point->y > box->high.y) - result = float8_mi(point->y, box->high.y); - else if (point->y < box->low.y) - result = float8_mi(box->low.y, point->y); - else - elog(ERROR, "inconsistent point values"); - } - else if (point->y <= box->high.y && point->y >= box->low.y) - { - /* point is to left or right of box */ - Assert(box->low.x <= box->high.x); - if (point->x > box->high.x) - result = float8_mi(point->x, box->high.x); - else if (point->x < box->low.x) - result = float8_mi(box->low.x, point->x); - else - elog(ERROR, "inconsistent point values"); - } - else - { - /* closest point will be a vertex */ - Point p; - float8 subresult; - - result = point_point_distance(point, &box->low); - - subresult = point_point_distance(point, &box->high); - if (result > subresult) - result = subresult; - - p.x = box->low.x; - p.y = box->high.y; - subresult = point_point_distance(point, &p); - if (result > subresult) - result = subresult; - - p.x = box->high.x; - p.y = box->low.y; - subresult = point_point_distance(point, &p); - if (result > subresult) - result = subresult; + return point_point_distance(point, &box->low); } - return result; + /* + * The distance to the box, as <->(box, point) computes it: exact for + * box_ops' keys, a lower bound for the bounding box of point_ops' + * internal ones. + */ + return point_box_distance(point, box); } static bool diff --git a/src/backend/access/spgist/spgproc.c b/src/backend/access/spgist/spgproc.c index 78873f23855bf..01b2091d3103a 100644 --- a/src/backend/access/spgist/spgproc.c +++ b/src/backend/access/spgist/spgproc.c @@ -26,34 +26,6 @@ DatumGetFloat8(DirectFunctionCall2(point_distance, \ PointPGetDatum(p1), PointPGetDatum(p2))) -/* Point-box distance in the assumption that box is aligned by axis */ -static double -point_box_distance(Point *point, BOX *box) -{ - double dx, - dy; - - if (isnan(point->x) || isnan(box->low.x) || - isnan(point->y) || isnan(box->low.y)) - return get_float8_nan(); - - if (point->x < box->low.x) - dx = box->low.x - point->x; - else if (point->x > box->high.x) - dx = point->x - box->high.x; - else - dx = 0.0; - - if (point->y < box->low.y) - dy = box->low.y - point->y; - else if (point->y > box->high.y) - dy = point->y - box->high.y; - else - dy = 0.0; - - return hypot(dx, dy); -} - /* * Returns distances from given key to array of ordering scan keys. Leaf key * is expected to be point, non-leaf key is expected to be box. Scan key diff --git a/src/backend/utils/adt/geo_ops.c b/src/backend/utils/adt/geo_ops.c index 73324b91fe521..5d99579953ffa 100644 --- a/src/backend/utils/adt/geo_ops.c +++ b/src/backend/utils/adt/geo_ops.c @@ -2951,64 +2951,56 @@ close_lseg(PG_FUNCTION_ARGS) /* - * Closest point on or in box to specified point. + * The closest point of 'box' to 'pt': 'pt' itself clamped to the box. * - * If *result is not NULL, set it to the closest point on the box to the - * given point, and return the distance of the two points. + * Return the distance between them. If *result is not NULL, set it to the + * closest point. A box's edges are parallel to the axes, so this needs no + * line arithmetic, and the result is exact up to hypot()'s rounding. The + * GiST and SP-GiST support functions use it through point_box_distance(), + * so the distances an index scan orders by are the operator's. */ static float8 box_closept_point(Point *result, BOX *box, Point *pt) { - float8 dist, - d; - Point point, - closept; - LSEG lseg; + Point closept; - if (box_contain_point(box, pt)) + if (isnan(pt->x) || isnan(pt->y) || + isnan(box->low.x) || isnan(box->low.y) || + isnan(box->high.x) || isnan(box->high.y)) { if (result != NULL) *result = *pt; - - return 0.0; + return get_float8_nan(); } - /* pairwise check lseg distances */ - point.x = box->low.x; - point.y = box->high.y; - statlseg_construct(&lseg, &box->low, &point); - dist = lseg_closept_point(result, &lseg, pt); + if (pt->x < box->low.x) + closept.x = box->low.x; + else if (pt->x > box->high.x) + closept.x = box->high.x; + else + closept.x = pt->x; - statlseg_construct(&lseg, &box->high, &point); - d = lseg_closept_point(&closept, &lseg, pt); - if (float8_lt(d, dist)) - { - dist = d; - if (result != NULL) - *result = closept; - } + if (pt->y < box->low.y) + closept.y = box->low.y; + else if (pt->y > box->high.y) + closept.y = box->high.y; + else + closept.y = pt->y; - point.x = box->high.x; - point.y = box->low.y; - statlseg_construct(&lseg, &box->low, &point); - d = lseg_closept_point(&closept, &lseg, pt); - if (float8_lt(d, dist)) - { - dist = d; - if (result != NULL) - *result = closept; - } + if (result != NULL) + *result = closept; - statlseg_construct(&lseg, &box->high, &point); - d = lseg_closept_point(&closept, &lseg, pt); - if (float8_lt(d, dist)) - { - dist = d; - if (result != NULL) - *result = closept; - } + return point_dt(&closept, pt, NULL); +} - return dist; +/* + * Distance from 'pt' to the nearest point of 'box', for the index support + * functions; see box_closept_point(). + */ +float8 +point_box_distance(Point *pt, BOX *box) +{ + return box_closept_point(NULL, box, pt); } Datum diff --git a/src/include/utils/geo_decls.h b/src/include/utils/geo_decls.h index a7014cded5190..5f99e1e200722 100644 --- a/src/include/utils/geo_decls.h +++ b/src/include/utils/geo_decls.h @@ -274,4 +274,6 @@ CirclePGetDatum(const CIRCLE *X) #define PG_GETARG_CIRCLE_P(n) DatumGetCircleP(PG_GETARG_DATUM(n)) #define PG_RETURN_CIRCLE_P(x) return CirclePGetDatum(x) +extern float8 point_box_distance(Point *pt, BOX *box); + #endif /* GEO_DECLS_H */ diff --git a/src/test/regress/expected/box.out b/src/test/regress/expected/box.out index 10760870ce7ce..9b01c19745c74 100644 --- a/src/test/regress/expected/box.out +++ b/src/test/regress/expected/box.out @@ -641,6 +641,41 @@ WHERE seq.id IS NULL OR idx.id IS NULL; RESET enable_seqscan; RESET enable_indexscan; RESET enable_bitmapscan; +-- The distance from a point to a box is exact, so a kNN index scan, which +-- orders by the index's own distance, returns rows in the operator's order. +-- For the point (0.9999996,2), box 1 is 1 away and box 2 is 1.00000000000004 +-- away. (The operator used to give box 1 1.00000000000008, out of order.) +-- Both distances are exact on any platform: the point is above both boxes, +-- so each is hypot(0, dy) = |dy|. +CREATE TEMP TABLE box_knn (id int, b box); +INSERT INTO box_knn VALUES (1, '((0,0),(1,1))'), + (2, box(point(0.4999996, 0), point(1.4999996, 0.99999999999996))); +SET extra_float_digits = 0; +SET enable_seqscan = OFF; +SET enable_bitmapscan = OFF; +CREATE INDEX box_knn_gist ON box_knn USING gist (b); +SELECT id, b <-> point(0.9999996, 2) AS dist FROM box_knn + ORDER BY b <-> point(0.9999996, 2); + id | dist +----+------------------ + 1 | 1 + 2 | 1.00000000000004 +(2 rows) + +DROP INDEX box_knn_gist; +CREATE INDEX box_knn_spgist ON box_knn USING spgist (b); +SELECT id, b <-> point(0.9999996, 2) AS dist FROM box_knn + ORDER BY b <-> point(0.9999996, 2); + id | dist +----+------------------ + 1 | 1 + 2 | 1.00000000000004 +(2 rows) + +RESET extra_float_digits; +RESET enable_seqscan; +RESET enable_bitmapscan; +DROP TABLE box_knn; -- test non-error-throwing API for some core types SELECT pg_input_is_valid('200', 'box'); pg_input_is_valid diff --git a/src/test/regress/expected/geometry.out b/src/test/regress/expected/geometry.out index 1d168b21cbca0..a0a3a204f8554 100644 --- a/src/test/regress/expected/geometry.out +++ b/src/test/regress/expected/geometry.out @@ -682,58 +682,58 @@ SELECT p.f1, l.s, p.f1 <-> l.s AS dist_ps, l.s <-> p.f1 AS dist_sp FROM POINT_TB -- Distance to box SELECT p.f1, b.f1, p.f1 <-> b.f1 AS dist_pb, b.f1 <-> p.f1 AS dist_bp FROM POINT_TBL p, BOX_TBL b; - f1 | f1 | dist_pb | dist_bp --------------------+---------------------+--------------------+-------------------- - (0,0) | (2,2),(0,0) | 0 | 0 - (0,0) | (3,3),(1,1) | 1.41421356237 | 1.41421356237 - (0,0) | (-2,2),(-8,-10) | 2 | 2 - (0,0) | (2.5,3.5),(2.5,2.5) | 3.53553390593 | 3.53553390593 - (0,0) | (3,3),(3,3) | 4.24264068712 | 4.24264068712 - (-10,0) | (2,2),(0,0) | 10 | 10 - (-10,0) | (3,3),(1,1) | 11.0453610172 | 11.0453610172 - (-10,0) | (-2,2),(-8,-10) | 2 | 2 - (-10,0) | (2.5,3.5),(2.5,2.5) | 12.747548784 | 12.747548784 - (-10,0) | (3,3),(3,3) | 13.3416640641 | 13.3416640641 - (-3,4) | (2,2),(0,0) | 3.60555127546 | 3.60555127546 - (-3,4) | (3,3),(1,1) | 4.12310562562 | 4.12310562562 - (-3,4) | (-2,2),(-8,-10) | 2 | 2 - (-3,4) | (2.5,3.5),(2.5,2.5) | 5.52268050859 | 5.52268050859 - (-3,4) | (3,3),(3,3) | 6.0827625303 | 6.0827625303 - (5.1,34.5) | (2,2),(0,0) | 32.6475113906 | 32.6475113906 - (5.1,34.5) | (3,3),(1,1) | 31.5699223946 | 31.5699223946 - (5.1,34.5) | (-2,2),(-8,-10) | 33.2664996656 | 33.2664996656 - (5.1,34.5) | (2.5,3.5),(2.5,2.5) | 31.108841187 | 31.108841187 - (5.1,34.5) | (3,3),(3,3) | 31.5699223946 | 31.5699223946 - (-5,-12) | (2,2),(0,0) | 13 | 13 - (-5,-12) | (3,3),(1,1) | 14.3178210633 | 14.3178210633 - (-5,-12) | (-2,2),(-8,-10) | 2 | 2 - (-5,-12) | (2.5,3.5),(2.5,2.5) | 16.3248277173 | 16.3248277173 - (-5,-12) | (3,3),(3,3) | 17 | 17 - (1e-300,-1e-300) | (2,2),(0,0) | 1.41421356237e-300 | 1.41421356237e-300 - (1e-300,-1e-300) | (3,3),(1,1) | 1.41421356237 | 1.41421356237 - (1e-300,-1e-300) | (-2,2),(-8,-10) | 2 | 2 - (1e-300,-1e-300) | (2.5,3.5),(2.5,2.5) | 3.53553390593 | 3.53553390593 - (1e-300,-1e-300) | (3,3),(3,3) | 4.24264068712 | 4.24264068712 - (1e+300,Infinity) | (2,2),(0,0) | Infinity | Infinity - (1e+300,Infinity) | (3,3),(1,1) | Infinity | Infinity - (1e+300,Infinity) | (-2,2),(-8,-10) | Infinity | Infinity - (1e+300,Infinity) | (2.5,3.5),(2.5,2.5) | Infinity | Infinity - (1e+300,Infinity) | (3,3),(3,3) | Infinity | Infinity - (Infinity,1e+300) | (2,2),(0,0) | Infinity | Infinity - (Infinity,1e+300) | (3,3),(1,1) | Infinity | Infinity - (Infinity,1e+300) | (-2,2),(-8,-10) | Infinity | Infinity - (Infinity,1e+300) | (2.5,3.5),(2.5,2.5) | Infinity | Infinity - (Infinity,1e+300) | (3,3),(3,3) | Infinity | Infinity - (NaN,NaN) | (2,2),(0,0) | NaN | NaN - (NaN,NaN) | (3,3),(1,1) | NaN | NaN - (NaN,NaN) | (-2,2),(-8,-10) | NaN | NaN - (NaN,NaN) | (2.5,3.5),(2.5,2.5) | NaN | NaN - (NaN,NaN) | (3,3),(3,3) | NaN | NaN - (10,10) | (2,2),(0,0) | 11.313708499 | 11.313708499 - (10,10) | (3,3),(1,1) | 9.89949493661 | 9.89949493661 - (10,10) | (-2,2),(-8,-10) | 14.4222051019 | 14.4222051019 - (10,10) | (2.5,3.5),(2.5,2.5) | 9.92471662064 | 9.92471662064 - (10,10) | (3,3),(3,3) | 9.89949493661 | 9.89949493661 + f1 | f1 | dist_pb | dist_bp +-------------------+---------------------+---------------+--------------- + (0,0) | (2,2),(0,0) | 0 | 0 + (0,0) | (3,3),(1,1) | 1.41421356237 | 1.41421356237 + (0,0) | (-2,2),(-8,-10) | 2 | 2 + (0,0) | (2.5,3.5),(2.5,2.5) | 3.53553390593 | 3.53553390593 + (0,0) | (3,3),(3,3) | 4.24264068712 | 4.24264068712 + (-10,0) | (2,2),(0,0) | 10 | 10 + (-10,0) | (3,3),(1,1) | 11.0453610172 | 11.0453610172 + (-10,0) | (-2,2),(-8,-10) | 2 | 2 + (-10,0) | (2.5,3.5),(2.5,2.5) | 12.747548784 | 12.747548784 + (-10,0) | (3,3),(3,3) | 13.3416640641 | 13.3416640641 + (-3,4) | (2,2),(0,0) | 3.60555127546 | 3.60555127546 + (-3,4) | (3,3),(1,1) | 4.12310562562 | 4.12310562562 + (-3,4) | (-2,2),(-8,-10) | 2 | 2 + (-3,4) | (2.5,3.5),(2.5,2.5) | 5.52268050859 | 5.52268050859 + (-3,4) | (3,3),(3,3) | 6.0827625303 | 6.0827625303 + (5.1,34.5) | (2,2),(0,0) | 32.6475113906 | 32.6475113906 + (5.1,34.5) | (3,3),(1,1) | 31.5699223946 | 31.5699223946 + (5.1,34.5) | (-2,2),(-8,-10) | 33.2664996656 | 33.2664996656 + (5.1,34.5) | (2.5,3.5),(2.5,2.5) | 31.108841187 | 31.108841187 + (5.1,34.5) | (3,3),(3,3) | 31.5699223946 | 31.5699223946 + (-5,-12) | (2,2),(0,0) | 13 | 13 + (-5,-12) | (3,3),(1,1) | 14.3178210633 | 14.3178210633 + (-5,-12) | (-2,2),(-8,-10) | 2 | 2 + (-5,-12) | (2.5,3.5),(2.5,2.5) | 16.3248277173 | 16.3248277173 + (-5,-12) | (3,3),(3,3) | 17 | 17 + (1e-300,-1e-300) | (2,2),(0,0) | 1e-300 | 1e-300 + (1e-300,-1e-300) | (3,3),(1,1) | 1.41421356237 | 1.41421356237 + (1e-300,-1e-300) | (-2,2),(-8,-10) | 2 | 2 + (1e-300,-1e-300) | (2.5,3.5),(2.5,2.5) | 3.53553390593 | 3.53553390593 + (1e-300,-1e-300) | (3,3),(3,3) | 4.24264068712 | 4.24264068712 + (1e+300,Infinity) | (2,2),(0,0) | Infinity | Infinity + (1e+300,Infinity) | (3,3),(1,1) | Infinity | Infinity + (1e+300,Infinity) | (-2,2),(-8,-10) | Infinity | Infinity + (1e+300,Infinity) | (2.5,3.5),(2.5,2.5) | Infinity | Infinity + (1e+300,Infinity) | (3,3),(3,3) | Infinity | Infinity + (Infinity,1e+300) | (2,2),(0,0) | Infinity | Infinity + (Infinity,1e+300) | (3,3),(1,1) | Infinity | Infinity + (Infinity,1e+300) | (-2,2),(-8,-10) | Infinity | Infinity + (Infinity,1e+300) | (2.5,3.5),(2.5,2.5) | Infinity | Infinity + (Infinity,1e+300) | (3,3),(3,3) | Infinity | Infinity + (NaN,NaN) | (2,2),(0,0) | NaN | NaN + (NaN,NaN) | (3,3),(1,1) | NaN | NaN + (NaN,NaN) | (-2,2),(-8,-10) | NaN | NaN + (NaN,NaN) | (2.5,3.5),(2.5,2.5) | NaN | NaN + (NaN,NaN) | (3,3),(3,3) | NaN | NaN + (10,10) | (2,2),(0,0) | 11.313708499 | 11.313708499 + (10,10) | (3,3),(1,1) | 9.89949493661 | 9.89949493661 + (10,10) | (-2,2),(-8,-10) | 14.4222051019 | 14.4222051019 + (10,10) | (2.5,3.5),(2.5,2.5) | 9.92471662064 | 9.92471662064 + (10,10) | (3,3),(3,3) | 9.89949493661 | 9.89949493661 (50 rows) -- Distance to path @@ -1224,19 +1224,19 @@ SELECT p.f1, b.f1, p.f1 ## b.f1 FROM POINT_TBL p, BOX_TBL b; (-5,-12) | (-2,2),(-8,-10) | (-5,-10) (-5,-12) | (2.5,3.5),(2.5,2.5) | (2.5,2.5) (-5,-12) | (3,3),(3,3) | (3,3) - (1e-300,-1e-300) | (2,2),(0,0) | (0,0) + (1e-300,-1e-300) | (2,2),(0,0) | (1e-300,0) (1e-300,-1e-300) | (3,3),(1,1) | (1,1) (1e-300,-1e-300) | (-2,2),(-8,-10) | (-2,-1e-300) (1e-300,-1e-300) | (2.5,3.5),(2.5,2.5) | (2.5,2.5) (1e-300,-1e-300) | (3,3),(3,3) | (3,3) - (1e+300,Infinity) | (2,2),(0,0) | (0,2) - (1e+300,Infinity) | (3,3),(1,1) | (1,3) - (1e+300,Infinity) | (-2,2),(-8,-10) | (-8,2) + (1e+300,Infinity) | (2,2),(0,0) | (2,2) + (1e+300,Infinity) | (3,3),(1,1) | (3,3) + (1e+300,Infinity) | (-2,2),(-8,-10) | (-2,2) (1e+300,Infinity) | (2.5,3.5),(2.5,2.5) | (2.5,3.5) (1e+300,Infinity) | (3,3),(3,3) | (3,3) - (Infinity,1e+300) | (2,2),(0,0) | (0,2) - (Infinity,1e+300) | (3,3),(1,1) | (1,3) - (Infinity,1e+300) | (-2,2),(-8,-10) | (-8,2) + (Infinity,1e+300) | (2,2),(0,0) | (2,2) + (Infinity,1e+300) | (3,3),(1,1) | (3,3) + (Infinity,1e+300) | (-2,2),(-8,-10) | (-2,2) (Infinity,1e+300) | (2.5,3.5),(2.5,2.5) | (2.5,3.5) (Infinity,1e+300) | (3,3),(3,3) | (3,3) (NaN,NaN) | (2,2),(0,0) | diff --git a/src/test/regress/sql/box.sql b/src/test/regress/sql/box.sql index 2d0868e88985b..13f89c18b6c22 100644 --- a/src/test/regress/sql/box.sql +++ b/src/test/regress/sql/box.sql @@ -282,6 +282,30 @@ RESET enable_seqscan; RESET enable_indexscan; RESET enable_bitmapscan; +-- The distance from a point to a box is exact, so a kNN index scan, which +-- orders by the index's own distance, returns rows in the operator's order. +-- For the point (0.9999996,2), box 1 is 1 away and box 2 is 1.00000000000004 +-- away. (The operator used to give box 1 1.00000000000008, out of order.) +-- Both distances are exact on any platform: the point is above both boxes, +-- so each is hypot(0, dy) = |dy|. +CREATE TEMP TABLE box_knn (id int, b box); +INSERT INTO box_knn VALUES (1, '((0,0),(1,1))'), + (2, box(point(0.4999996, 0), point(1.4999996, 0.99999999999996))); +SET extra_float_digits = 0; +SET enable_seqscan = OFF; +SET enable_bitmapscan = OFF; +CREATE INDEX box_knn_gist ON box_knn USING gist (b); +SELECT id, b <-> point(0.9999996, 2) AS dist FROM box_knn + ORDER BY b <-> point(0.9999996, 2); +DROP INDEX box_knn_gist; +CREATE INDEX box_knn_spgist ON box_knn USING spgist (b); +SELECT id, b <-> point(0.9999996, 2) AS dist FROM box_knn + ORDER BY b <-> point(0.9999996, 2); +RESET extra_float_digits; +RESET enable_seqscan; +RESET enable_bitmapscan; +DROP TABLE box_knn; + -- test non-error-throwing API for some core types SELECT pg_input_is_valid('200', 'box'); SELECT * FROM pg_input_error_info('200', 'box'); From 0add4fa527a00050bda400e11c53737f8390d425 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Thu, 1 Oct 2026 10:19:05 -0400 Subject: [PATCH 06/12] Let an ordering index scan hand its ORDER BY values to the target list An Index Scan that orders by an operator (ORDER BY col <-> const) computes each returned row's ORDER BY value inside the index AM and reports it in xs_orderbyvals. If the same expression also appears in the target list, the executor then evaluates it a second time in the scan's projection (a third time if the AM asked for a recheck). For an expensive distance operator that is wasted work. The AM contract only requires a non-rechecked distance to sort the same way as the operator's result, not to equal it, so the executor cannot substitute it blindly. Add an optional IndexAmRoutine callback, amcanreturnorderby(opfamily, opcintype, orderbyop): does the opclass report, for a non-rechecked leaf entry and that ordering operator, exactly the value the operator would return? It takes the operator because one opfamily can hold several ordering operators. The planner keeps the callback in IndexOptInfo, like amcostestimate, and create_indexscan_plan() asks it for each ORDER BY key, carrying one bool per key into the IndexScan. For each qualifying key, setrefs.c replaces a top-level target list expression that is equal() to that indexorderbyorig expression with Var(INNER_VAR, i), and nodeIndexscan.c stores the returned row's ORDER BY values in a virtual slot it installs as the projection's inner tuple (a scan node has no inner plan, so INNER_VAR is otherwise unused there). The values stored are the AM's for a row it reported exact, the recomputed ones for a row it asked to recheck, a reorder queue entry's copy for a queued row, and freshly recomputed ones for an EvalPlanQual test tuple. Only float4/float8 ORDER BY expressions qualify, as those are the only types index_store_float8_orderby_distances() can deliver. GiST answers yes for point_ops and SP-GiST for quad_point_ops and kd_point_ops, whose leaf distances call point_distance(), the function behind <->(point, point). Both answer yes for box_ops too, whose distances the previous commit made the operator's own. The polygon and circle opclasses report lower bounds and are not affected, nor is any other AM or any out-of-core opclass, which all default to no. EXPLAIN VERBOSE deparses the INNER_VAR back to the ORDER BY expression and reports "Order By Values Used: N", the number of ORDER BY keys the targetlist reads this way. Above such a scan, an upper node's reference to the value now prints with one more pair of parentheses, the same way any reference to a computed scan column does. The documentation describes the line, in the EXPLAIN chapter and with nearest-neighbor searches. Author: Greg Burd Discussion: https://postgr.es/m/8s5lT8erXzBMugXJQ6Wginbp_gc2B4hmCYhF0Q0GpnuK87eCAOcKs5K31AWCwraCNFIQt42wwkow_MPPubHg485MWv-zDBY4NL_JX9yJEEg=@burd.me --- contrib/btree_gist/expected/timestamp.out | 34 +++ contrib/btree_gist/sql/timestamp.sql | 18 ++ doc/src/sgml/indexam.sgml | 37 +++ doc/src/sgml/indices.sgml | 19 ++ doc/src/sgml/perform.sgml | 31 ++ src/backend/access/gist/gist.c | 1 + src/backend/access/gist/gistget.c | 25 ++ src/backend/access/spgist/spgscan.c | 26 ++ src/backend/access/spgist/spgutils.c | 1 + src/backend/commands/explain.c | 4 + src/backend/executor/nodeIndexscan.c | 87 +++++- src/backend/optimizer/plan/createplan.c | 20 ++ src/backend/optimizer/plan/setrefs.c | 60 ++++ src/backend/optimizer/util/plancat.c | 2 + src/backend/utils/adt/ruleutils.c | 18 ++ src/include/access/amapi.h | 9 + src/include/access/gist_private.h | 1 + src/include/access/spgist.h | 1 + src/include/nodes/execnodes.h | 8 + src/include/nodes/pathnodes.h | 2 + src/include/nodes/plannodes.h | 16 + .../isolation/expected/eval-plan-qual-knn.out | 40 +++ src/test/isolation/isolation_schedule | 1 + .../isolation/specs/eval-plan-qual-knn.spec | 44 +++ src/test/regress/expected/box.out | 12 +- src/test/regress/expected/gist.out | 284 ++++++++++++++++++ src/test/regress/sql/gist.sql | 154 ++++++++++ src/tools/pgindent/typedefs.list | 1 + 28 files changed, 948 insertions(+), 8 deletions(-) create mode 100644 src/test/isolation/expected/eval-plan-qual-knn.out create mode 100644 src/test/isolation/specs/eval-plan-qual-knn.spec diff --git a/contrib/btree_gist/expected/timestamp.out b/contrib/btree_gist/expected/timestamp.out index 0d94f2f245cea..6589fc8125fc2 100644 --- a/contrib/btree_gist/expected/timestamp.out +++ b/contrib/btree_gist/expected/timestamp.out @@ -89,3 +89,37 @@ SELECT a, a <-> '2004-10-26 08:55:08' FROM timestamptmp ORDER BY a <-> '2004-10- Mon Nov 29 20:12:43 2004 | @ 34 days 11 hours 17 mins 35 secs (3 rows) +-- An ordering scan that hands an exact point distance to the target list +-- while another ORDER BY key, here an interval, is of a by-reference type. +CREATE TABLE timestamppt (p point, a timestamp); +INSERT INTO timestamppt + SELECT point(i % 7, i % 11), '2004-10-26 08:55:08'::timestamp + i * interval '37 min' + FROM generate_series(1, 600) i; +CREATE INDEX timestamppt_idx ON timestamppt USING gist (p, a); +SET enable_indexonlyscan=off; +EXPLAIN (VERBOSE, COSTS OFF) +SELECT p <-> point(3, 4) AS d, a <-> '2004-10-26 08:55:08' AS da + FROM timestamppt + ORDER BY p <-> point(3, 4), a <-> '2004-10-26 08:55:08' LIMIT 3; + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------------------------- + Limit + Output: ((p <-> '(3,4)'::point)), ((a <-> 'Tue Oct 26 08:55:08 2004'::timestamp without time zone)) + -> Index Scan using timestamppt_idx on public.timestamppt + Output: ((p <-> '(3,4)'::point)), (a <-> 'Tue Oct 26 08:55:08 2004'::timestamp without time zone) + Order By: ((timestamppt.p <-> '(3,4)'::point) AND (timestamppt.a <-> 'Tue Oct 26 08:55:08 2004'::timestamp without time zone)) + Order By Values Used: 1 +(6 rows) + +SELECT p <-> point(3, 4) AS d, a <-> '2004-10-26 08:55:08' AS da + FROM timestamppt + ORDER BY p <-> point(3, 4), a <-> '2004-10-26 08:55:08' LIMIT 3; + d | da +---+--------------------------- + 0 | @ 1 day 12 hours 23 mins + 0 | @ 3 days 11 hours 52 mins + 0 | @ 5 days 11 hours 21 mins +(3 rows) + +RESET enable_indexonlyscan; +DROP TABLE timestamppt; diff --git a/contrib/btree_gist/sql/timestamp.sql b/contrib/btree_gist/sql/timestamp.sql index 95effebfc47a7..216b0211a3611 100644 --- a/contrib/btree_gist/sql/timestamp.sql +++ b/contrib/btree_gist/sql/timestamp.sql @@ -35,3 +35,21 @@ SELECT count(*) FROM timestamptmp WHERE a > '2004-10-26 08:55:08'::timestamp; EXPLAIN (COSTS OFF) SELECT a, a <-> '2004-10-26 08:55:08' FROM timestamptmp ORDER BY a <-> '2004-10-26 08:55:08' LIMIT 3; SELECT a, a <-> '2004-10-26 08:55:08' FROM timestamptmp ORDER BY a <-> '2004-10-26 08:55:08' LIMIT 3; + +-- An ordering scan that hands an exact point distance to the target list +-- while another ORDER BY key, here an interval, is of a by-reference type. +CREATE TABLE timestamppt (p point, a timestamp); +INSERT INTO timestamppt + SELECT point(i % 7, i % 11), '2004-10-26 08:55:08'::timestamp + i * interval '37 min' + FROM generate_series(1, 600) i; +CREATE INDEX timestamppt_idx ON timestamppt USING gist (p, a); +SET enable_indexonlyscan=off; +EXPLAIN (VERBOSE, COSTS OFF) +SELECT p <-> point(3, 4) AS d, a <-> '2004-10-26 08:55:08' AS da + FROM timestamppt + ORDER BY p <-> point(3, 4), a <-> '2004-10-26 08:55:08' LIMIT 3; +SELECT p <-> point(3, 4) AS d, a <-> '2004-10-26 08:55:08' AS da + FROM timestamppt + ORDER BY p <-> point(3, 4), a <-> '2004-10-26 08:55:08' LIMIT 3; +RESET enable_indexonlyscan; +DROP TABLE timestamppt; diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml index f48da3185307c..ee20c47750adb 100644 --- a/doc/src/sgml/indexam.sgml +++ b/doc/src/sgml/indexam.sgml @@ -157,6 +157,7 @@ typedef struct IndexAmRoutine ambulkdelete_function ambulkdelete; amvacuumcleanup_function amvacuumcleanup; amcanreturn_function amcanreturn; /* can be NULL */ + amcanreturnorderby_function amcanreturnorderby; /* can be NULL */ amcostestimate_function amcostestimate; amgettreeheight_function amgettreeheight; /* can be NULL */ amoptions_function amoptions; @@ -481,6 +482,32 @@ amcanreturn (Relation indexRelation, int attno); +bool +amcanreturnorderby (Oid opfamily, Oid opcintype, Oid orderbyop); + + Check whether, for an ordering scan using ordering operator + orderbyop on a column of the operator class given + by opfamily and opcintype, + the values the access method reports in + scan->xs_orderbyvals with + xs_recheckorderby false are exactly the values the + operator would return for the heap tuple. If so, the executor may return + them for a target list entry that is the same expression as the + ORDER BY key, instead of evaluating the operator again. + The answer is for that operator class and operator, not for the access + method as a whole: an operator class whose distances merely sort like the + operator's result, or differ from it in any digit, or depend on anything + other than the indexed value, must answer false. The planner calls this + while building the plan; it is only consulted for an + float8 or float4 ORDER BY + expression, and for float4 it is the float4 + rounding of the distance the access method reports that must equal the + operator's result. If the amcanreturnorderby + field is set to NULL, no substitution is made. + + + + void amcostestimate (PlannerInfo *root, IndexPath *path, @@ -989,6 +1016,16 @@ amtranslatecmptype (CompareType cmptype, Oid opfamily, Oid opcintype); of that form can be passed to amrescan as described previously. + + Such an access method returns each entry's ordering values in + scan->xs_orderbyvals, and sets + scan->xs_recheckorderby to true when those values are + only lower bounds that the executor must recompute from the heap + tuple. When xs_recheckorderby is false the values + need only sort in the same order as the ordering operator's results; + they need not equal them. See amcanreturnorderby + below for how an access method can promise that they do. + diff --git a/doc/src/sgml/indices.sgml b/doc/src/sgml/indices.sgml index 55f39b0df2f2e..f425bea463580 100644 --- a/doc/src/sgml/indices.sgml +++ b/doc/src/sgml/indices.sgml @@ -290,6 +290,25 @@ SELECT * FROM places ORDER BY location <-> point '(101,456)' LIMIT 10; In , operators that can be used in this way are listed in the column Ordering Operators. + + + To order the rows, the index computes each one's distance from the target. + For some operator classes that distance is exactly what the ordering + operator returns, and when the query also selects the distance, as in + point '(101,456)' AS dist + FROM places ORDER BY location <-> point '(101,456)' LIMIT 10; +]]> + + the index scan returns the distance it already computed instead of + evaluating the operator again for each row. This applies to the + built-in point_ops and box_ops + operator classes of GiST and the quad_point_ops, + kd_point_ops and box_ops operator + classes of SP-GiST. EXPLAIN VERBOSE shows it as an + Order By Values Used line on the index scan; see + . + diff --git a/doc/src/sgml/perform.sgml b/doc/src/sgml/perform.sgml index ea8da01b7797f..bb25d5cd6d2f4 100644 --- a/doc/src/sgml/perform.sgml +++ b/doc/src/sgml/perform.sgml @@ -702,6 +702,37 @@ FROM tenk1 t1 WHERE t1.ten = (SELECT (random() * 10)::integer); happen without the sub-SELECT construct. + + A nearest-neighbor index scan (see + ) computes each row's distance in order to + return the rows in distance order. When the query also selects that + distance, the scan can return the value it computed instead of + evaluating the ordering operator again: + + +EXPLAIN (VERBOSE, COSTS OFF) SELECT id, location <-> point '(101,456)' AS dist +FROM places ORDER BY location <-> point '(101,456)' LIMIT 10; + + QUERY PLAN +------------------------------------------------------------- + Limit + Output: id, ((location <-> '(101,456)'::point)) + -> Index Scan using places_location_idx on public.places + Output: id, ((location <-> '(101,456)'::point)) + Order By: (places.location <-> '(101,456)'::point) + Order By Values Used: 1 + + + The Order By Values Used line counts the + ORDER BY keys whose values the scan returns this way. + It appears only in EXPLAIN VERBOSE output, and only + when the count is not zero. In the index scan's + Output line, the extra parentheses around the + distance show that the scan supplies the value rather than computing it. + This happens only for operator classes whose distances are exactly the + operator's result. + + diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 8565e225be7fd..f6c4592a1f5ff 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -93,6 +93,7 @@ gisthandler(PG_FUNCTION_ARGS) .ambulkdelete = gistbulkdelete, .amvacuumcleanup = gistvacuumcleanup, .amcanreturn = gistcanreturn, + .amcanreturnorderby = gistcanreturnorderby, .amcostestimate = gistcostestimate, .amgettreeheight = NULL, .amoptions = gistoptions, diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c index 111c310f9df5b..ce966c053bda4 100644 --- a/src/backend/access/gist/gistget.c +++ b/src/backend/access/gist/gistget.c @@ -23,6 +23,8 @@ #include "pgstat.h" #include "storage/predicate.h" #include "utils/float.h" +#include "utils/fmgroids.h" +#include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -813,3 +815,26 @@ gistcanreturn(Relation index, int attno) else return false; } + +/* + * Are the distances the opclass reports for 'orderbyop' on a leaf entry, + * when it does not set *recheck, the operator's own result? + * + * The GiST contract only requires a non-rechecked distance to *sort* like + * the operator's result, so we cannot assume this of an arbitrary opclass. + * point_ops computes a leaf's distance by calling point_distance(), the + * function behind <->(point, point), so the executor may return it in place + * of evaluating the operator. Likewise box_ops, whose leaf distance is + * point_box_distance(), the function behind <->(box, point). Other + * opclasses do not qualify. + */ +bool +gistcanreturnorderby(Oid opfamily, Oid opcintype, Oid orderbyop) +{ + RegProcedure distproc = get_opfamily_proc(opfamily, opcintype, opcintype, + GIST_DISTANCE_PROC); + RegProcedure opcode = get_opcode(orderbyop); + + return ((distproc == F_GIST_POINT_DISTANCE && opcode == F_POINT_DISTANCE) || + (distproc == F_GIST_BOX_DISTANCE && opcode == F_DIST_BP)); +} diff --git a/src/backend/access/spgist/spgscan.c b/src/backend/access/spgist/spgscan.c index 2cc5f06f5d779..2b01fa346cf17 100644 --- a/src/backend/access/spgist/spgscan.c +++ b/src/backend/access/spgist/spgscan.c @@ -24,6 +24,7 @@ #include "storage/bufmgr.h" #include "utils/datum.h" #include "utils/float.h" +#include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -1088,3 +1089,28 @@ spgcanreturn(Relation index, int attno) return cache->config.canReturnData; } + +/* + * Are the distances the opclass reports for 'orderbyop' on a leaf tuple, + * when it does not set recheckDistances, the operator's own result? + * + * As for GiST, the contract only requires them to sort like the operator's + * result. quad_point_ops and kd_point_ops (whose leaf function is + * spg_quad_leaf_consistent()) compute a leaf's distance with + * point_distance(), the function behind <->(point, point). box_ops (whose + * leaf function is spg_box_quad_leaf_consistent()) computes it with + * point_box_distance(), the function behind <->(box, point); poly_ops shares + * that leaf function but sets recheckDistances, and its operator is another. + */ +bool +spgcanreturnorderby(Oid opfamily, Oid opcintype, Oid orderbyop) +{ + RegProcedure leafproc = get_opfamily_proc(opfamily, opcintype, opcintype, + SPGIST_LEAF_CONSISTENT_PROC); + RegProcedure opcode = get_opcode(orderbyop); + + return ((leafproc == F_SPG_QUAD_LEAF_CONSISTENT && + opcode == F_POINT_DISTANCE) || + (leafproc == F_SPG_BOX_QUAD_LEAF_CONSISTENT && + opcode == F_DIST_BP)); +} diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c index f2ee333f60d84..d07e197a594cf 100644 --- a/src/backend/access/spgist/spgutils.c +++ b/src/backend/access/spgist/spgutils.c @@ -78,6 +78,7 @@ spghandler(PG_FUNCTION_ARGS) .ambulkdelete = spgbulkdelete, .amvacuumcleanup = spgvacuumcleanup, .amcanreturn = spgcanreturn, + .amcanreturnorderby = spgcanreturnorderby, .amcostestimate = spgcostestimate, .amgettreeheight = NULL, .amoptions = spgoptions, diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 4b1ad4227acb4..afc45c0969280 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -1974,6 +1974,10 @@ ExplainNode(PlanState *planstate, List *ancestors, planstate, es); show_scan_qual(((IndexScan *) plan)->indexorderbyorig, "Order By", planstate, ancestors, es); + if (es->verbose && ((IndexScan *) plan)->indexorderbynused > 0) + ExplainPropertyInteger("Order By Values Used", NULL, + ((IndexScan *) plan)->indexorderbynused, + es); show_scan_qual(plan->qual, "Filter", planstate, ancestors, es); if (plan->qual) show_instrumentation_count("Rows Removed by Filter", 1, diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 129d005f187dd..6f11c1938fd3b 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -163,6 +163,46 @@ IndexNext(IndexScanState *node) return ExecClearTuple(slot); } +/* + * Hand the ORDER BY values of the row about to be returned to the projection, + * when the targetlist asks for them (indexorderbynused > 0, set by setrefs.c + * only for keys whose opclass passes amcanreturnorderby). vals and nulls + * are what the caller established as this row's ordering values: the AM's + * when it reported them exact, the recomputed ones when it asked for a + * recheck, or a reorder queue entry's copy. Never a lossy AM's lower bound. + * The targetlist reads only float8 and float4 keys (see setrefs.c's + * replace_orderby_tlist_refs()), which are passed by value, so the slot can + * keep them after the caller frees its copies. Another key may be of a + * by-reference type, such as btree_gist's interval-valued <->; nothing reads + * it here, so store a null rather than a pointer that is about to dangle. + */ +static void +IndexStoreOrderByValues(IndexScanState *node, const Datum *vals, + const bool *nulls) +{ + TupleTableSlot *oslot = node->iss_OrderBySlot; + int i; + + if (oslot == NULL) + return; + ExecClearTuple(oslot); + for (i = 0; i < node->iss_NumOrderByKeys; i++) + { + if (node->iss_OrderByTypByVals[i]) + { + oslot->tts_values[i] = vals[i]; + oslot->tts_isnull[i] = nulls[i]; + } + else + { + oslot->tts_values[i] = (Datum) 0; + oslot->tts_isnull[i] = true; + } + } + ExecStoreVirtualTuple(oslot); + node->ss.ps.ps_ExprContext->ecxt_innertuple = oslot; +} + /* ---------------------------------------------------------------- * IndexNextWithReorder * @@ -251,6 +291,9 @@ IndexNextWithReorder(IndexScanState *node) { HeapTuple tuple; + /* the values are by-value; see IndexStoreOrderByValues() */ + IndexStoreOrderByValues(node, topmost->orderbyvals, + topmost->orderbynulls); tuple = reorderqueue_pop(node); /* Pass 'true', as the tuple in the queue is a palloc'd copy */ @@ -352,6 +395,7 @@ IndexNextWithReorder(IndexScanState *node) else { /* Can return this tuple immediately. */ + IndexStoreOrderByValues(node, lastfetched_vals, lastfetched_nulls); return slot; } } @@ -404,7 +448,22 @@ IndexRecheck(IndexScanState *node, TupleTableSlot *slot) /* Does the tuple meet the indexqual condition? */ econtext->ecxt_scantuple = slot; - return ExecQualAndReset(node->indexqualorig, econtext); + if (!ExecQualAndReset(node->indexqualorig, econtext)) + return false; + + /* + * The EPQ test tuple never went through IndexNext, so the ORDER BY values + * the targetlist reads (if any) are not those of this row. Recompute + * them from the tuple, as we would for a rechecked row. + */ + if (node->iss_OrderBySlot != NULL) + { + EvalOrderByExpressions(node, econtext); + IndexStoreOrderByValues(node, node->iss_OrderByValues, + node->iss_OrderByNulls); + } + + return true; } @@ -954,7 +1013,21 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags) * Initialize result type and projection. */ ExecInitResultTypeTL(&indexstate->ss.ps); - ExecAssignScanProjectionInfo(&indexstate->ss); + + /* + * A targetlist that reads the ORDER BY values (INNER_VAR Vars, see + * setrefs.c) is never the scan tuple's physical layout, so it always + * projects; ExecConditionalAssignProjectionInfo would assert on the + * INNER_VAR while testing whether it can skip projection. + */ + if (node->indexorderbynused > 0) + { + ExecInitResultSlot(&indexstate->ss.ps, &TTSOpsVirtual); + ExecAssignProjectionInfo(&indexstate->ss.ps, + indexstate->ss.ss_ScanTupleSlot->tts_tupleDescriptor); + } + else + ExecAssignScanProjectionInfo(&indexstate->ss); /* * initialize child expressions @@ -1024,6 +1097,16 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags) NULL, /* no ArrayKeys */ NULL); + /* + * A targetlist that reads the ORDER BY values needs a slot to read them + * from, typed as the ORDER BY expressions (setrefs.c built it so). + */ + if (node->indexorderbynused > 0) + indexstate->iss_OrderBySlot = + ExecAllocTableSlot(&estate->es_tupleTable, + ExecTypeFromExprList(node->indexorderbyorig), + &TTSOpsVirtual, 0); + /* Initialize sort support, if we need to re-check ORDER BY exprs */ if (indexstate->iss_NumOrderByKeys > 0) { diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 9c1a93f19fb82..87d8b95e9be4c 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -3062,6 +3062,26 @@ create_indexscan_plan(PlannerInfo *root, indexorderbyops, best_path->indexscandir); + if (!indexonly && indexorderbys != NIL) + { + List *exact = NIL; + ListCell *lo, + *lc; + + forboth(lo, best_path->indexorderbys, lc, best_path->indexorderbycols) + { + int indexcol = lfirst_int(lc); + OpExpr *orderby = lfirst_node(OpExpr, lo); + + exact = lappend_int(exact, + indexinfo->amcanreturnorderby != NULL && + indexinfo->amcanreturnorderby(indexinfo->opfamily[indexcol], + indexinfo->opcintype[indexcol], + orderby->opno)); + } + ((IndexScan *) scan_plan)->indexorderbyexact = exact; + } + copy_generic_path_info(&scan_plan->plan, &best_path->path); return scan_plan; diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index af78892fafc4e..0036bbb4d1739 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -134,6 +134,7 @@ static bool flatten_rtes_walker(Node *node, flatten_rtes_walker_context *cxt); static void add_rte_to_flat_rtable(PlannerGlobal *glob, List *rteperminfos, RangeTblEntry *rte); static Plan *set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset); +static void replace_orderby_tlist_refs(IndexScan *splan); static Plan *set_indexonlyscan_references(PlannerInfo *root, IndexOnlyScan *plan, int rtoffset); @@ -704,6 +705,8 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) splan->indexorderbyorig = fix_scan_list(root, splan->indexorderbyorig, rtoffset, NUM_EXEC_QUAL(plan)); + /* after both are fixed, so equal() compares like with like */ + replace_orderby_tlist_refs(splan); } break; case T_IndexOnlyScan: @@ -1367,6 +1370,63 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) return plan; } +/* + * replace_orderby_tlist_refs + * Let an ordering IndexScan hand its ORDER BY values to its targetlist. + * + * For each ORDER BY key whose index column's opclass promises that the + * distances it reports for an exact (non-rechecked) row are the ordering + * operator's own result (indexorderbyexact[i], from amcanreturnorderby), + * replace each top-level targetlist expression that is equal() to the i'th + * indexorderbyorig expression with Var(INNER_VAR, i + 1). nodeIndexscan.c + * then fills a virtual slot with the ORDER BY values of every returned row -- + * the AM's for an exact row, the recomputed ones for a rechecked row -- and + * installs it as the projection's inner tuple. A scan node has no inner + * plan, so INNER_VAR is otherwise unused here. + * + * Only float8/float4 expressions qualify: those are the only types + * index_store_float8_orderby_distances() can deliver. + */ +static void +replace_orderby_tlist_refs(IndexScan *splan) +{ + ListCell *lc; + Bitmapset *used = NULL; + + if (splan->indexorderbyexact == NIL) + return; + Assert(list_length(splan->indexorderbyexact) == + list_length(splan->indexorderbyorig)); + + foreach(lc, splan->scan.plan.targetlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + ListCell *lo, + *le; + int i = 0; + + forboth(lo, splan->indexorderbyorig, le, splan->indexorderbyexact) + { + Expr *orig = (Expr *) lfirst(lo); + Oid typ = exprType((Node *) orig); + + i++; + if (lfirst_int(le) && + (typ == FLOAT8OID || typ == FLOAT4OID) && + equal(tle->expr, orig)) + { + tle->expr = (Expr *) makeVar(INNER_VAR, i, typ, + exprTypmod((Node *) orig), + exprCollation((Node *) orig), + 0); + used = bms_add_member(used, i); + break; + } + } + } + splan->indexorderbynused = bms_num_members(used); +} + /* * set_indexonlyscan_references * Do set_plan_references processing on an IndexOnlyScan diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 22e0e8b11f8d5..bd496d71a2ca2 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -316,6 +316,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, amroutine->amrestrpos != NULL); info->amcostestimate = amroutine->amcostestimate; Assert(info->amcostestimate != NULL); + info->amcanreturnorderby = amroutine->amcanreturnorderby; /* Fetch index opclass options */ info->opclassoptions = RelationGetIndexAttOptions(indexRelation, true); @@ -414,6 +415,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, info->amhasgetbitmap = false; info->amcanmarkpos = false; info->amcostestimate = NULL; + info->amcanreturnorderby = NULL; info->sortopfamily = NULL; info->reverse_sort = NULL; diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index 8f6a46fa614e6..f2b5102543455 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -7999,6 +7999,24 @@ resolve_special_varno(Node *node, deparse_context *context, context->appendparents = save_appendparents; return; } + else if (var->varno == INNER_VAR && dpns->plan != NULL && + IsA(dpns->plan, IndexScan) && + ((IndexScan *) dpns->plan)->indexorderbynused > 0) + { + /* + * An IndexScan whose targetlist reads its own ORDER BY values (see + * replace_orderby_tlist_refs in setrefs.c) uses INNER_VAR Vars + * numbered by ORDER BY key; they deparse as the ORDER BY expressions. + */ + List *orderbys = ((IndexScan *) dpns->plan)->indexorderbyorig; + + if (var->varattno < 1 || var->varattno > list_length(orderbys)) + elog(ERROR, "bogus varattno for ORDER BY value var: %d", + var->varattno); + resolve_special_varno((Node *) list_nth(orderbys, var->varattno - 1), + context, callback, callback_arg); + return; + } else if (var->varno == INNER_VAR && dpns->inner_tlist) { TargetEntry *tle; diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index 7924033353031..f520ba06e6a77 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -144,6 +144,14 @@ typedef IndexBulkDeleteResult *(*amvacuumcleanup_function) (IndexVacuumInfo *inf /* can indexscan return IndexTuples? */ typedef bool (*amcanreturn_function) (Relation indexRelation, int attno); +/* + * Are the ORDER BY distances an opclass reports for ordering operator + * 'orderbyop' on an exact (xs_recheckorderby = false) leaf entry the + * operator's own result? + */ +typedef bool (*amcanreturnorderby_function) (Oid opfamily, Oid opcintype, + Oid orderbyop); + /* estimate cost of an indexscan */ typedef void (*amcostestimate_function) (PlannerInfo *root, IndexPath *path, @@ -300,6 +308,7 @@ typedef struct IndexAmRoutine ambulkdelete_function ambulkdelete; amvacuumcleanup_function amvacuumcleanup; amcanreturn_function amcanreturn; /* can be NULL */ + amcanreturnorderby_function amcanreturnorderby; /* can be NULL */ amcostestimate_function amcostestimate; amgettreeheight_function amgettreeheight; /* can be NULL */ amoptions_function amoptions; diff --git a/src/include/access/gist_private.h b/src/include/access/gist_private.h index 39fdd27f5b97c..d2b3b5d7ea36d 100644 --- a/src/include/access/gist_private.h +++ b/src/include/access/gist_private.h @@ -462,6 +462,7 @@ extern void gistkillitems(IndexScanDesc scan); extern bool gistgettuple(IndexScanDesc scan, ScanDirection dir); extern int64 gistgetbitmap(IndexScanDesc scan, TIDBitmap *tbm); extern bool gistcanreturn(Relation index, int attno); +extern bool gistcanreturnorderby(Oid opfamily, Oid opcintype, Oid orderbyop); /* gistvalidate.c */ extern bool gistvalidate(Oid opclassoid); diff --git a/src/include/access/spgist.h b/src/include/access/spgist.h index 083d93f8ffdc0..f293c14455b68 100644 --- a/src/include/access/spgist.h +++ b/src/include/access/spgist.h @@ -210,6 +210,7 @@ extern void spgrescan(IndexScanDesc scan, ScanKey scankey, int nscankeys, extern int64 spggetbitmap(IndexScanDesc scan, TIDBitmap *tbm); extern bool spggettuple(IndexScanDesc scan, ScanDirection dir); extern bool spgcanreturn(Relation index, int attno); +extern bool spgcanreturnorderby(Oid opfamily, Oid opcintype, Oid orderbyop); /* spgvacuum.c */ extern IndexBulkDeleteResult *spgbulkdelete(IndexVacuumInfo *info, diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 8087a986204cd..bb18b2f5c23be 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1723,6 +1723,7 @@ typedef struct * OrderByTypByVals is the datatype of order by expression pass-by-value? * OrderByTypLens typlens of the datatypes of order by expressions * PscanLen size of parallel index scan descriptor + * OrderBySlot returned row's ORDER BY values, for the targetlist * ---------------- */ typedef struct IndexScanState @@ -1752,6 +1753,13 @@ typedef struct IndexScanState bool *iss_OrderByTypByVals; int16 *iss_OrderByTypLens; Size iss_PscanLen; + + /* + * When the plan's targetlist reads the ORDER BY values (indexorderbynused + * > 0), the values of the row being returned, as a virtual slot installed + * as the projection's inner tuple. + */ + TupleTableSlot *iss_OrderBySlot; } IndexScanState; /* ---------------- diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index ce22529bec03f..ca2a024b915c0 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -1450,6 +1450,8 @@ typedef struct IndexOptInfo /* AM's cost estimator */ /* Rather than include amapi.h here, we declare amcostestimate like this */ void (*amcostestimate) (struct PlannerInfo *, struct IndexPath *, double, Cost *, Cost *, Selectivity *, double *, double *) pg_node_attr(read_write_ignore); + /* AM's amcanreturnorderby, or NULL */ + bool (*amcanreturnorderby) (Oid, Oid, Oid) pg_node_attr(read_write_ignore); } IndexOptInfo; /* diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index 10f6b105f91ff..798818c94b37c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -616,6 +616,22 @@ typedef struct IndexScan List *indexorderbyops; /* forward or backward or don't care */ ScanDirection indexorderdir; + + /* + * Per indexorderbyorig entry: may setrefs.c replace a target-list + * occurrence of that expression with an INNER_VAR reference to the scan's + * ORDER BY value? True when the index column's opclass reports exact + * distances that are the operator's result (amcanreturnorderby). + * setrefs.c replaces only float8 and float4 expressions, the only types + * the AMs deliver. Only setrefs.c reads it, so it isn't written out with + * the plan. + * + * indexorderbynused counts the ORDER BY values the targetlist reads (0 if + * none); the executor sets up the ORDER BY slot and projection when it's + * nonzero, and EXPLAIN shows it. + */ + List *indexorderbyexact pg_node_attr(read_write_ignore, read_as(NIL)); + int indexorderbynused; } IndexScan; /* ---------------- diff --git a/src/test/isolation/expected/eval-plan-qual-knn.out b/src/test/isolation/expected/eval-plan-qual-knn.out new file mode 100644 index 0000000000000..3c1c4794b64f4 --- /dev/null +++ b/src/test/isolation/expected/eval-plan-qual-knn.out @@ -0,0 +1,40 @@ +Parsed test spec with 2 sessions + +starting permutation: s2e s1b s1u s2s s1c +step s2e: + EXPLAIN (VERBOSE, COSTS OFF) + SELECT id, p <-> point(0, 0) AS d FROM epq_knn + WHERE p <@ box(point(0, 0), point(10, 10)) + ORDER BY p <-> point(0, 0) LIMIT 4 FOR UPDATE; + +QUERY PLAN +------------------------------------------------------------- +Limit + Output: id, ((p <-> '(0,0)'::point)), ctid + -> LockRows + Output: id, ((p <-> '(0,0)'::point)), ctid + -> Index Scan using epq_knn_p on public.epq_knn + Output: id, ((p <-> '(0,0)'::point)), ctid + Index Cond: (epq_knn.p <@ '(10,10),(0,0)'::box) + Order By: (epq_knn.p <-> '(0,0)'::point) + Order By Values Used: 1 +(9 rows) + +step s1b: BEGIN; +step s1u: UPDATE epq_knn SET p = point(3.5, 3.5) WHERE id = 3; +step s2s: + SELECT id, p <-> point(0, 0) AS d, (p <-> point(0, 0)) = d AS same + FROM (SELECT id, p, p <-> point(0, 0) AS d FROM epq_knn + WHERE p <@ box(point(0, 0), point(10, 10)) + ORDER BY p <-> point(0, 0) LIMIT 4 FOR UPDATE) s; + +step s1c: COMMIT; +step s2s: <... completed> +id| d|same +--+------------------+---- + 1|1.4142135623730951|t + 2|2.8284271247461903|t + 3| 4.949747468305833|t + 4| 5.656854249492381|t +(4 rows) + diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index bf9a2037c705a..1703a3ec50bd7 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -44,6 +44,7 @@ test: subxid-overflow test: eval-plan-qual test: eval-plan-qual-trigger test: eval-plan-qual-partition-prune +test: eval-plan-qual-knn test: inplace-inval test: intra-grant-inplace test: intra-grant-inplace-db diff --git a/src/test/isolation/specs/eval-plan-qual-knn.spec b/src/test/isolation/specs/eval-plan-qual-knn.spec new file mode 100644 index 0000000000000..b485e88dbc3c1 --- /dev/null +++ b/src/test/isolation/specs/eval-plan-qual-knn.spec @@ -0,0 +1,44 @@ +# An ordering index scan that returns its ORDER BY values (amcanreturnorderby) +# must not hand back the value it computed for the old row version when +# EvalPlanQual re-checks a concurrently updated row: the value is recomputed +# from the new version. + +setup +{ + CREATE TABLE epq_knn (id int, p point); + INSERT INTO epq_knn SELECT i, point(i, i) FROM generate_series(1, 20) i; + CREATE INDEX epq_knn_p ON epq_knn USING gist (p); +} + +teardown +{ + DROP TABLE epq_knn; +} + +session s1 +setup { SET enable_seqscan = off; SET enable_bitmapscan = off; } +step s1b { BEGIN; } +step s1u { UPDATE epq_knn SET p = point(3.5, 3.5) WHERE id = 3; } +step s1c { COMMIT; } + +session s2 +setup { SET enable_seqscan = off; SET enable_bitmapscan = off; } +step s2e +{ + EXPLAIN (VERBOSE, COSTS OFF) + SELECT id, p <-> point(0, 0) AS d FROM epq_knn + WHERE p <@ box(point(0, 0), point(10, 10)) + ORDER BY p <-> point(0, 0) LIMIT 4 FOR UPDATE; +} +step s2s +{ + SELECT id, p <-> point(0, 0) AS d, (p <-> point(0, 0)) = d AS same + FROM (SELECT id, p, p <-> point(0, 0) AS d FROM epq_knn + WHERE p <@ box(point(0, 0), point(10, 10)) + ORDER BY p <-> point(0, 0) LIMIT 4 FOR UPDATE) s; +} + +# s2's scan computes id 3's distance from (3,3), then waits on s1's lock; +# after s1 commits, EvalPlanQual re-checks id 3 with p = (3.5,3.5), and the +# value returned must be that row's distance. +permutation s2e s1b s1u s2s s1c diff --git a/src/test/regress/expected/box.out b/src/test/regress/expected/box.out index 9b01c19745c74..595ddff7140dd 100644 --- a/src/test/regress/expected/box.out +++ b/src/test/regress/expected/box.out @@ -594,10 +594,10 @@ SET enable_bitmapscan = OFF; EXPLAIN (COSTS OFF) SELECT rank() OVER (ORDER BY b <-> point '123,456') n, b <-> point '123,456' dist, id FROM quad_box_tbl; - QUERY PLAN --------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------- WindowAgg - Window: w1 AS (ORDER BY (b <-> '(123,456)'::point) ROWS UNBOUNDED PRECEDING) + Window: w1 AS (ORDER BY ((b <-> '(123,456)'::point)) ROWS UNBOUNDED PRECEDING) -> Index Scan using quad_box_tbl_idx on quad_box_tbl Order By: (b <-> '(123,456)'::point) (4 rows) @@ -617,10 +617,10 @@ WHERE seq.id IS NULL OR idx.id IS NULL; EXPLAIN (COSTS OFF) SELECT rank() OVER (ORDER BY b <-> point '123,456') n, b <-> point '123,456' dist, id FROM quad_box_tbl WHERE b <@ box '((200,300),(500,600))'; - QUERY PLAN --------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------- WindowAgg - Window: w1 AS (ORDER BY (b <-> '(123,456)'::point) ROWS UNBOUNDED PRECEDING) + Window: w1 AS (ORDER BY ((b <-> '(123,456)'::point)) ROWS UNBOUNDED PRECEDING) -> Index Scan using quad_box_tbl_idx on quad_box_tbl Index Cond: (b <@ '(500,600),(200,300)'::box) Order By: (b <-> '(123,456)'::point) diff --git a/src/test/regress/expected/gist.out b/src/test/regress/expected/gist.out index ac79f94aa8028..648566a0bbd33 100644 --- a/src/test/regress/expected/gist.out +++ b/src/test/regress/expected/gist.out @@ -387,6 +387,290 @@ select p from gist_tbl order by circle(p,1) <-> point(0,0) limit 1; select p from gist_tbl order by circle(p,1) <-> point(0,0) limit 1; ERROR: lossy distance functions are not supported in index-only scans +drop index gist_tbl_multi_index; +-- Test that an ordering Index Scan returns the AM's ORDER BY values to a +-- targetlist entry equal to the ORDER BY expression, instead of evaluating +-- the expression again, for an opclass that passes amcanreturnorderby. +set enable_indexonlyscan = off; +-- An expression index over a counting function proves the operator is not +-- re-evaluated: the count stays at zero for the rows returned. +create sequence gist_cnt_seq; +create function gist_cnt_pt(point) returns point language plpgsql immutable + as $$ begin perform nextval('public.gist_cnt_seq'); return $1; end $$; +create index gist_tbl_cnt_index on gist_tbl using gist (gist_cnt_pt(p)); +explain (verbose, costs off) +select p, gist_cnt_pt(p) <-> point(0.201, 0.201) as dist + from gist_tbl order by gist_cnt_pt(p) <-> point(0.201, 0.201) limit 3; + QUERY PLAN +------------------------------------------------------------------------ + Limit + Output: p, ((gist_cnt_pt(p) <-> '(0.201,0.201)'::point)) + -> Index Scan using gist_tbl_cnt_index on public.gist_tbl + Output: p, ((gist_cnt_pt(p) <-> '(0.201,0.201)'::point)) + Order By: (gist_cnt_pt(gist_tbl.p) <-> '(0.201,0.201)'::point) + Order By Values Used: 1 +(6 rows) + +select setval('gist_cnt_seq', 1, false); + setval +-------- + 1 +(1 row) + +-- the distance numbers are not exactly the same across platforms +set extra_float_digits = 0; +select p, gist_cnt_pt(p) <-> point(0.201, 0.201) as dist + from gist_tbl order by gist_cnt_pt(p) <-> point(0.201, 0.201) limit 3; + p | dist +-------------+-------------------- + (0.2,0.2) | 0.0014142135623731 + (0.25,0.25) | 0.0692964645562816 + (0.15,0.15) | 0.0721248916810279 +(3 rows) + +reset extra_float_digits; +select nextval('gist_cnt_seq') - 1 as calls_during_scan; + calls_during_scan +------------------- + 0 +(1 row) + +-- Every copy of the ORDER BY expression is served from the index; an +-- expression that merely contains it is not. +explain (verbose, costs off) +select gist_cnt_pt(p) <-> point(0,0) as d1, + (gist_cnt_pt(p) <-> point(0,0)) * 2 as d2, + gist_cnt_pt(p) <-> point(0,0) as d3 + from gist_tbl order by gist_cnt_pt(p) <-> point(0,0) limit 2; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------- + Limit + Output: ((gist_cnt_pt(p) <-> '(0,0)'::point)), (((gist_cnt_pt(p) <-> '(0,0)'::point) * '2'::double precision)), ((gist_cnt_pt(p) <-> '(0,0)'::point)) + -> Index Scan using gist_tbl_cnt_index on public.gist_tbl + Output: ((gist_cnt_pt(p) <-> '(0,0)'::point)), ((gist_cnt_pt(p) <-> '(0,0)'::point) * '2'::double precision), ((gist_cnt_pt(p) <-> '(0,0)'::point)) + Order By: (gist_cnt_pt(gist_tbl.p) <-> '(0,0)'::point) + Order By Values Used: 1 +(6 rows) + +drop index gist_tbl_cnt_index; +drop function gist_cnt_pt(point); +drop sequence gist_cnt_seq; +-- The values must equal the operator's, also for a row re-fetched under +-- row locking, and for a rescan with a new ORDER BY argument (LATERAL). +create index gist_tbl_point_index on gist_tbl using gist (p); +select count(*) filter (where dist = p <-> point(5.2, 5.91)) as same, + count(*) as total + from (select p, p <-> point(5.2, 5.91) as dist from gist_tbl + order by p <-> point(5.2, 5.91) limit 200) ss; + same | total +------+------- + 200 | 200 +(1 row) + +set extra_float_digits = 0; +select p, p <-> point(5.2, 5.91) as dist from gist_tbl + where p <@ box(point(5,5), point(6,6)) order by p <-> point(5.2, 5.91) + limit 3 for update; + p | dist +-------------+------------------- + (5.55,5.55) | 0.502095608425328 + (5.6,5.6) | 0.5060632371552 + (5.5,5.5) | 0.508035431835221 +(3 rows) + +reset extra_float_digits; +explain (verbose, costs off) +select v.q, n.dist + from (values (point(0.3, 0.3)), (point(7.1, 7.2)), (point(42, 42))) v(q), + lateral (select p <-> v.q as dist from gist_tbl + order by p <-> v.q limit 2) n; + QUERY PLAN +---------------------------------------------------------------------- + Nested Loop + Output: "*VALUES*".column1, ((gist_tbl.p <-> "*VALUES*".column1)) + -> Values Scan on "*VALUES*" + Output: "*VALUES*".column1 + -> Limit + Output: ((gist_tbl.p <-> "*VALUES*".column1)) + -> Index Scan using gist_tbl_point_index on public.gist_tbl + Output: ((gist_tbl.p <-> "*VALUES*".column1)) + Order By: (gist_tbl.p <-> "*VALUES*".column1) + Order By Values Used: 1 +(10 rows) + +select count(*) filter (where n.dist = n.p <-> v.q) as same, count(*) as total + from (values (point(0.3, 0.3)), (point(7.1, 7.2)), (point(42, 42))) v(q), + lateral (select p, p <-> v.q as dist from gist_tbl + order by p <-> v.q limit 50) n; + same | total +------+------- + 150 | 150 +(1 row) + +-- A multi-column index ordered by an exact key and a lossy one: the lossy +-- one's distances are rechecked, rows pass through the reorder queue, and +-- the exact key's value is still the operator's. +create index gist_tbl_pc_index on gist_tbl using gist (p, c); +explain (verbose, costs off) +select p <-> point(1, 1) as dp, c <-> point(2, 2) as dc from gist_tbl + order by p <-> point(1, 1), c <-> point(2, 2) limit 5; + QUERY PLAN +----------------------------------------------------------------------------------------- + Limit + Output: ((p <-> '(1,1)'::point)), ((c <-> '(2,2)'::point)) + -> Index Scan using gist_tbl_pc_index on public.gist_tbl + Output: ((p <-> '(1,1)'::point)), (c <-> '(2,2)'::point) + Order By: ((gist_tbl.p <-> '(1,1)'::point) AND (gist_tbl.c <-> '(2,2)'::point)) + Order By Values Used: 1 +(6 rows) + +select count(*) filter (where dp = p <-> point(1, 1)) as same_p, + count(*) filter (where dc = c <-> point(2, 2)) as same_c, + count(*) as total + from (select p, c, p <-> point(1, 1) as dp, c <-> point(2, 2) as dc + from gist_tbl + order by p <-> point(1, 1), c <-> point(2, 2) limit 200) ss; + same_p | same_c | total +--------+--------+------- + 200 | 200 | 200 +(1 row) + +drop index gist_tbl_pc_index; +drop index gist_tbl_point_index; +-- The box opclass qualifies too: its leaf distance is the operator's. +create index gist_tbl_box_index on gist_tbl using gist (b); +explain (verbose, costs off) +select b, b <-> point(5.2, 5.91) as dist from gist_tbl + where b <@ box(point(5,5), point(6,6)) order by b <-> point(5.2, 5.91); + QUERY PLAN +-------------------------------------------------------- + Index Scan using gist_tbl_box_index on public.gist_tbl + Output: b, ((b <-> '(5.2,5.91)'::point)) + Index Cond: (gist_tbl.b <@ '(6,6),(5,5)'::box) + Order By: (gist_tbl.b <-> '(5.2,5.91)'::point) + Order By Values Used: 1 +(5 rows) + +select count(*) filter (where dist = b <-> point(5.2, 5.91)) as same, + count(*) as total + from (select b, b <-> point(5.2, 5.91) as dist from gist_tbl + order by b <-> point(5.2, 5.91) limit 300) ss; + same | total +------+------- + 300 | 300 +(1 row) + +-- Including for a point just off an edge's line, where the operator used to +-- be off in the last digits. +create temp table gist_box_eps (b box); +insert into gist_box_eps values ('((0,0),(1,1))'); +create index on gist_box_eps using gist (b); +select count(*) filter (where dist = b <-> point(0.9999996, 2)) as same, + count(*) as total + from (select b, b <-> point(0.9999996, 2) as dist from gist_box_eps + order by b <-> point(0.9999996, 2) limit 1) ss; + same | total +------+------- + 1 | 1 +(1 row) + +drop table gist_box_eps; +drop index gist_tbl_box_index; +-- The circle opclass's distance is a lower bound that is always rechecked, +-- and its opclass does not pass amcanreturnorderby: the targetlist is left +-- alone and the operator is evaluated as before. +create index gist_tbl_circle_index on gist_tbl using gist (c); +explain (verbose, costs off) +select c <-> point(5.2, 5.91) as dist from gist_tbl + order by c <-> point(5.2, 5.91) limit 3; + QUERY PLAN +----------------------------------------------------------------- + Limit + Output: ((c <-> '(5.2,5.91)'::point)) + -> Index Scan using gist_tbl_circle_index on public.gist_tbl + Output: (c <-> '(5.2,5.91)'::point) + Order By: (gist_tbl.c <-> '(5.2,5.91)'::point) +(5 rows) + +select count(*) filter (where dist = c <-> point(5.2, 5.91)) as same, + count(*) as total + from (select c, c <-> point(5.2, 5.91) as dist from gist_tbl + order by c <-> point(5.2, 5.91) limit 200) ss; + same | total +------+------- + 200 | 200 +(1 row) + +drop index gist_tbl_circle_index; +reset enable_indexonlyscan; +-- SP-GiST's quad and k-d point opclasses and its box opclass qualify as well. +create temp table spg_orderby (p point, b box); +insert into spg_orderby + select point(i % 97 * 0.31, i / 97 * 0.29), + box(point(i % 97 * 0.31, i / 97 * 0.29), + point(i % 97 * 0.31 + 0.2, i / 97 * 0.29 + 0.1)) + from generate_series(1, 5000) i; +create index spg_orderby_quad on spg_orderby using spgist (p); +set enable_indexonlyscan = off; +explain (verbose, costs off) +select p <-> point(3, 4) as dist from spg_orderby + order by p <-> point(3, 4) limit 3; + QUERY PLAN +---------------------------------------------------------------- + Limit + Output: ((p <-> '(3,4)'::point)) + -> Index Scan using spg_orderby_quad on pg_temp.spg_orderby + Output: ((p <-> '(3,4)'::point)) + Order By: (spg_orderby.p <-> '(3,4)'::point) + Order By Values Used: 1 +(6 rows) + +select count(*) filter (where dist = p <-> point(3, 4)) as same, + count(*) as total + from (select p, p <-> point(3, 4) as dist from spg_orderby + order by p <-> point(3, 4) limit 300) ss; + same | total +------+------- + 300 | 300 +(1 row) + +drop index spg_orderby_quad; +create index spg_orderby_kd on spg_orderby using spgist (p kd_point_ops); +select count(*) filter (where dist = p <-> point(3, 4)) as same, + count(*) as total + from (select p, p <-> point(3, 4) as dist from spg_orderby + order by p <-> point(3, 4) limit 300) ss; + same | total +------+------- + 300 | 300 +(1 row) + +drop index spg_orderby_kd; +create index spg_orderby_box on spg_orderby using spgist (b); +explain (verbose, costs off) +select b <-> point(3, 4) as dist from spg_orderby + order by b <-> point(3, 4) limit 3; + QUERY PLAN +--------------------------------------------------------------- + Limit + Output: ((b <-> '(3,4)'::point)) + -> Index Scan using spg_orderby_box on pg_temp.spg_orderby + Output: ((b <-> '(3,4)'::point)) + Order By: (spg_orderby.b <-> '(3,4)'::point) + Order By Values Used: 1 +(6 rows) + +select count(*) filter (where dist = b <-> point(3, 4)) as same, + count(*) as total + from (select b, b <-> point(3, 4) as dist from spg_orderby + order by b <-> point(3, 4) limit 300) ss; + same | total +------+------- + 300 | 300 +(1 row) + +reset enable_indexonlyscan; +drop table spg_orderby; -- Test that an index-only scan deforms the tuple it reconstructs with the -- descriptor the AM formed it with, not the scan slot's descriptor. create temp table gist_ios_tupdesc (a inet, r numrange); diff --git a/src/test/regress/sql/gist.sql b/src/test/regress/sql/gist.sql index 57dcc08245037..74772e357a612 100644 --- a/src/test/regress/sql/gist.sql +++ b/src/test/regress/sql/gist.sql @@ -169,6 +169,160 @@ explain (verbose, costs off) select p from gist_tbl order by circle(p,1) <-> point(0,0) limit 1; select p from gist_tbl order by circle(p,1) <-> point(0,0) limit 1; +drop index gist_tbl_multi_index; + +-- Test that an ordering Index Scan returns the AM's ORDER BY values to a +-- targetlist entry equal to the ORDER BY expression, instead of evaluating +-- the expression again, for an opclass that passes amcanreturnorderby. +set enable_indexonlyscan = off; + +-- An expression index over a counting function proves the operator is not +-- re-evaluated: the count stays at zero for the rows returned. +create sequence gist_cnt_seq; +create function gist_cnt_pt(point) returns point language plpgsql immutable + as $$ begin perform nextval('public.gist_cnt_seq'); return $1; end $$; +create index gist_tbl_cnt_index on gist_tbl using gist (gist_cnt_pt(p)); + +explain (verbose, costs off) +select p, gist_cnt_pt(p) <-> point(0.201, 0.201) as dist + from gist_tbl order by gist_cnt_pt(p) <-> point(0.201, 0.201) limit 3; + +select setval('gist_cnt_seq', 1, false); +-- the distance numbers are not exactly the same across platforms +set extra_float_digits = 0; +select p, gist_cnt_pt(p) <-> point(0.201, 0.201) as dist + from gist_tbl order by gist_cnt_pt(p) <-> point(0.201, 0.201) limit 3; +reset extra_float_digits; +select nextval('gist_cnt_seq') - 1 as calls_during_scan; + +-- Every copy of the ORDER BY expression is served from the index; an +-- expression that merely contains it is not. +explain (verbose, costs off) +select gist_cnt_pt(p) <-> point(0,0) as d1, + (gist_cnt_pt(p) <-> point(0,0)) * 2 as d2, + gist_cnt_pt(p) <-> point(0,0) as d3 + from gist_tbl order by gist_cnt_pt(p) <-> point(0,0) limit 2; + +drop index gist_tbl_cnt_index; +drop function gist_cnt_pt(point); +drop sequence gist_cnt_seq; + +-- The values must equal the operator's, also for a row re-fetched under +-- row locking, and for a rescan with a new ORDER BY argument (LATERAL). +create index gist_tbl_point_index on gist_tbl using gist (p); + +select count(*) filter (where dist = p <-> point(5.2, 5.91)) as same, + count(*) as total + from (select p, p <-> point(5.2, 5.91) as dist from gist_tbl + order by p <-> point(5.2, 5.91) limit 200) ss; + +set extra_float_digits = 0; +select p, p <-> point(5.2, 5.91) as dist from gist_tbl + where p <@ box(point(5,5), point(6,6)) order by p <-> point(5.2, 5.91) + limit 3 for update; +reset extra_float_digits; + +explain (verbose, costs off) +select v.q, n.dist + from (values (point(0.3, 0.3)), (point(7.1, 7.2)), (point(42, 42))) v(q), + lateral (select p <-> v.q as dist from gist_tbl + order by p <-> v.q limit 2) n; +select count(*) filter (where n.dist = n.p <-> v.q) as same, count(*) as total + from (values (point(0.3, 0.3)), (point(7.1, 7.2)), (point(42, 42))) v(q), + lateral (select p, p <-> v.q as dist from gist_tbl + order by p <-> v.q limit 50) n; + +-- A multi-column index ordered by an exact key and a lossy one: the lossy +-- one's distances are rechecked, rows pass through the reorder queue, and +-- the exact key's value is still the operator's. +create index gist_tbl_pc_index on gist_tbl using gist (p, c); +explain (verbose, costs off) +select p <-> point(1, 1) as dp, c <-> point(2, 2) as dc from gist_tbl + order by p <-> point(1, 1), c <-> point(2, 2) limit 5; +select count(*) filter (where dp = p <-> point(1, 1)) as same_p, + count(*) filter (where dc = c <-> point(2, 2)) as same_c, + count(*) as total + from (select p, c, p <-> point(1, 1) as dp, c <-> point(2, 2) as dc + from gist_tbl + order by p <-> point(1, 1), c <-> point(2, 2) limit 200) ss; +drop index gist_tbl_pc_index; +drop index gist_tbl_point_index; + +-- The box opclass qualifies too: its leaf distance is the operator's. +create index gist_tbl_box_index on gist_tbl using gist (b); + +explain (verbose, costs off) +select b, b <-> point(5.2, 5.91) as dist from gist_tbl + where b <@ box(point(5,5), point(6,6)) order by b <-> point(5.2, 5.91); + +select count(*) filter (where dist = b <-> point(5.2, 5.91)) as same, + count(*) as total + from (select b, b <-> point(5.2, 5.91) as dist from gist_tbl + order by b <-> point(5.2, 5.91) limit 300) ss; +-- Including for a point just off an edge's line, where the operator used to +-- be off in the last digits. +create temp table gist_box_eps (b box); +insert into gist_box_eps values ('((0,0),(1,1))'); +create index on gist_box_eps using gist (b); +select count(*) filter (where dist = b <-> point(0.9999996, 2)) as same, + count(*) as total + from (select b, b <-> point(0.9999996, 2) as dist from gist_box_eps + order by b <-> point(0.9999996, 2) limit 1) ss; +drop table gist_box_eps; + +drop index gist_tbl_box_index; + +-- The circle opclass's distance is a lower bound that is always rechecked, +-- and its opclass does not pass amcanreturnorderby: the targetlist is left +-- alone and the operator is evaluated as before. +create index gist_tbl_circle_index on gist_tbl using gist (c); + +explain (verbose, costs off) +select c <-> point(5.2, 5.91) as dist from gist_tbl + order by c <-> point(5.2, 5.91) limit 3; + +select count(*) filter (where dist = c <-> point(5.2, 5.91)) as same, + count(*) as total + from (select c, c <-> point(5.2, 5.91) as dist from gist_tbl + order by c <-> point(5.2, 5.91) limit 200) ss; + +drop index gist_tbl_circle_index; +reset enable_indexonlyscan; + +-- SP-GiST's quad and k-d point opclasses and its box opclass qualify as well. +create temp table spg_orderby (p point, b box); +insert into spg_orderby + select point(i % 97 * 0.31, i / 97 * 0.29), + box(point(i % 97 * 0.31, i / 97 * 0.29), + point(i % 97 * 0.31 + 0.2, i / 97 * 0.29 + 0.1)) + from generate_series(1, 5000) i; +create index spg_orderby_quad on spg_orderby using spgist (p); +set enable_indexonlyscan = off; +explain (verbose, costs off) +select p <-> point(3, 4) as dist from spg_orderby + order by p <-> point(3, 4) limit 3; +select count(*) filter (where dist = p <-> point(3, 4)) as same, + count(*) as total + from (select p, p <-> point(3, 4) as dist from spg_orderby + order by p <-> point(3, 4) limit 300) ss; +drop index spg_orderby_quad; +create index spg_orderby_kd on spg_orderby using spgist (p kd_point_ops); +select count(*) filter (where dist = p <-> point(3, 4)) as same, + count(*) as total + from (select p, p <-> point(3, 4) as dist from spg_orderby + order by p <-> point(3, 4) limit 300) ss; +drop index spg_orderby_kd; +create index spg_orderby_box on spg_orderby using spgist (b); +explain (verbose, costs off) +select b <-> point(3, 4) as dist from spg_orderby + order by b <-> point(3, 4) limit 3; +select count(*) filter (where dist = b <-> point(3, 4)) as same, + count(*) as total + from (select b, b <-> point(3, 4) as dist from spg_orderby + order by b <-> point(3, 4) limit 300) ss; +reset enable_indexonlyscan; +drop table spg_orderby; + -- Test that an index-only scan deforms the tuple it reconstructs with the -- descriptor the AM formed it with, not the scan slot's descriptor. create temp table gist_ios_tupdesc (a inet, r numrange); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 553162c08106d..a9275933ed5a2 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3567,6 +3567,7 @@ ambuildempty_function ambuildphasename_function ambulkdelete_function amcanreturn_function +amcanreturnorderby_function amcostestimate_function amendscan_function amestimateparallelscan_function From 819716cf5f8bc712b718ebb163aa2e0ec31134d9 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Wed, 7 Oct 2026 01:50:04 +0000 Subject: [PATCH 07/12] Don't charge an ordering index scan for ORDER BY values it returns The previous commit lets a plain Index Scan return its ORDER BY values to a target list entry that matches the ORDER BY expression, instead of evaluating that expression again. The planner didn't know that. An ORDER BY expression is always part of the scan/join target (as a sort column, if nothing else), and apply_scanjoin_target_to_paths() charges every path rows * cost(expression) when it applies that target. For the Index Scan that cost is now never paid, while Seq Scan + Sort and Bitmap Heap Scan + Sort really do pay it, once per row. With a costly ORDER BY expression the planner therefore overestimated the kNN scan by up to ~32x and picked a Sort plan that was up to 13x slower. Teach create_projection_path() and apply_projection_to_path() to cost the target a path pays for with a new helper, path_target_cost(), which for a plain IndexScan leaves out top-level target expressions the scan will take from its ORDER BY values. The test for "will take" is index_orderby_returnable() plus orderby_tlist_match(); setrefs.c's rewrite and create_indexscan_plan() use the same two, so the cost model and the plan cannot drift apart. orderby_tlist_match() accepts the expression equal() to the ORDER BY key, and also its commuted form: match_clause_to_ordering_op() turns "const <-> col" into "col <-> const" for the index, so a target entry written the original way is the same value but not equal() to the key. An expression that only contains the ORDER BY expression, e.g. (expr) * 2, isn't rewritten and is still charged. Because this makes an IndexPath cheaper relative to its siblings, apply_scanjoin_target_to_paths() can no longer assume that applying the target preserves the cost ordering of rel->pathlist and rel->partial_pathlist, which add_path_precheck() and the linitial(partial_pathlist) callers rely on. Re-sort them with a new sort_pathlist_by_cost(), a stable insertion sort that is a single pass in the usual already-sorted case. Nothing changes for an index whose opclass does not pass amcanreturnorderby, for IndexOnlyScan, for bitmap scans, or for any query without a returnable ORDER BY expression in the target. Author: Greg Burd Discussion: https://postgr.es/m/8s5lT8erXzBMugXJQ6Wginbp_gc2B4hmCYhF0Q0GpnuK87eCAOcKs5K31AWCwraCNFIQt42wwkow_MPPubHg485MWv-zDBY4NL_JX9yJEEg=@burd.me --- src/backend/optimizer/plan/createplan.c | 13 +- src/backend/optimizer/plan/planner.c | 10 +- src/backend/optimizer/plan/setrefs.c | 16 ++- src/backend/optimizer/util/pathnode.c | 173 +++++++++++++++++++++++- src/include/optimizer/pathnode.h | 4 + src/test/regress/expected/gist.out | 93 +++++++++++++ src/test/regress/sql/gist.sql | 70 ++++++++++ 7 files changed, 354 insertions(+), 25 deletions(-) diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 87d8b95e9be4c..a42ac73e7ca29 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -3068,17 +3068,12 @@ create_indexscan_plan(PlannerInfo *root, ListCell *lo, *lc; + /* must agree with path_target_cost() in pathnode.c */ forboth(lo, best_path->indexorderbys, lc, best_path->indexorderbycols) - { - int indexcol = lfirst_int(lc); - OpExpr *orderby = lfirst_node(OpExpr, lo); - exact = lappend_int(exact, - indexinfo->amcanreturnorderby != NULL && - indexinfo->amcanreturnorderby(indexinfo->opfamily[indexcol], - indexinfo->opcintype[indexcol], - orderby->opno)); - } + index_orderby_returnable(indexinfo, + lfirst_int(lc), + (Expr *) lfirst(lo))); ((IndexScan *) scan_plan)->indexorderbyexact = exact; } diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 233b13f4cdfc5..567edfe03e2e0 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -8260,8 +8260,10 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, * If the tlist exprs are the same, we can just inject the sortgroupref * information into the existing pathtargets. Otherwise, replace each * path with a projection path that generates the SRF-free scan/join - * target. This can't change the ordering of paths within rel->pathlist, - * so we just modify the list in place. + * target. We modify the lists in place. That usually adds the same cost + * to every path, but a plain IndexScan doesn't pay for target entries it + * takes from its ORDER BY values, so it can become cheaper relative to + * the others; restore the cost ordering afterwards. */ foreach(lc, rel->pathlist) { @@ -8282,6 +8284,8 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, lfirst(lc) = newpath; } } + if (!tlist_same_exprs) + sort_pathlist_by_cost(rel->pathlist); /* Likewise adjust the targets for any partial paths. */ foreach(lc, rel->partial_pathlist) @@ -8303,6 +8307,8 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, lfirst(lc) = newpath; } } + if (!tlist_same_exprs) + sort_pathlist_by_cost(rel->partial_pathlist); /* * Now, if final scan/join target contains SRFs, insert ProjectSetPath(s) diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index 0036bbb4d1739..e6b6439ff2d01 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -1377,15 +1377,19 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) * For each ORDER BY key whose index column's opclass promises that the * distances it reports for an exact (non-rechecked) row are the ordering * operator's own result (indexorderbyexact[i], from amcanreturnorderby), - * replace each top-level targetlist expression that is equal() to the i'th - * indexorderbyorig expression with Var(INNER_VAR, i + 1). nodeIndexscan.c + * replace each top-level targetlist expression that is the i'th + * indexorderbyorig expression (equal() to it, or its commuted form; see + * orderby_tlist_match()) with Var(INNER_VAR, i + 1). nodeIndexscan.c * then fills a virtual slot with the ORDER BY values of every returned row -- * the AM's for an exact row, the recomputed ones for a rechecked row -- and * installs it as the projection's inner tuple. A scan node has no inner * plan, so INNER_VAR is otherwise unused here. * - * Only float8/float4 expressions qualify: those are the only types - * index_store_float8_orderby_distances() can deliver. + * indexorderbyexact[i] comes from index_orderby_returnable(), which also + * limits this to float8/float4, the only types + * index_store_float8_orderby_distances() can deliver. The planner costs + * the scan with the same test (path_target_cost() in pathnode.c), so what + * is costed is what is built. */ static void replace_orderby_tlist_refs(IndexScan *splan) @@ -1411,9 +1415,7 @@ replace_orderby_tlist_refs(IndexScan *splan) Oid typ = exprType((Node *) orig); i++; - if (lfirst_int(le) && - (typ == FLOAT8OID || typ == FLOAT4OID) && - equal(tle->expr, orig)) + if (lfirst_int(le) && orderby_tlist_match(tle->expr, orig)) { tle->expr = (Expr *) makeVar(INNER_VAR, i, typ, exprTypmod((Node *) orig), diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 3af494f269dfe..272a88d6dd023 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -15,10 +15,12 @@ #include "postgres.h" #include "access/htup_details.h" +#include "catalog/pg_type.h" #include "executor/nodeSetOp.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" +#include "nodes/nodeFuncs.h" #include "optimizer/appendinfo.h" #include "optimizer/clauses.h" #include "optimizer/cost.h" @@ -28,6 +30,7 @@ #include "optimizer/planmain.h" #include "optimizer/tlist.h" #include "parser/parsetree.h" +#include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/selfuncs.h" @@ -2579,6 +2582,157 @@ create_hashjoin_path(PlannerInfo *root, return pathnode; } +/* + * sort_pathlist_by_cost + * Restore the ordering add_path() and add_partial_path() maintain + * (fewest disabled nodes, then lowest total cost) after the costs of + * paths already in the list were changed in place. + * + * This is a stable insertion sort: it costs one pass when the list is still + * in order, which is the usual case, and keeps equal-cost paths in the order + * they were added. + */ +void +sort_pathlist_by_cost(List *pathlist) +{ + for (int i = 1; i < list_length(pathlist); i++) + { + Path *path = list_nth(pathlist, i); + int j; + + for (j = i; j > 0; j--) + { + Path *prev = list_nth(pathlist, j - 1); + + if (prev->disabled_nodes < path->disabled_nodes || + (prev->disabled_nodes == path->disabled_nodes && + prev->total_cost <= path->total_cost)) + break; + lfirst(list_nth_cell(pathlist, j)) = prev; + } + lfirst(list_nth_cell(pathlist, j)) = path; + } +} + +/* + * index_orderby_returnable + * Will an IndexScan on 'index' hand back the value of ORDER BY expression + * 'orderby', bound to index column 'indexcol', to its targetlist? + * + * create_indexscan_plan() records this per key for setrefs.c, and + * path_target_cost() uses it to cost the path, so that the cost we assign to + * a path is the cost of the plan we build from it. The opclass must promise + * exact values (amcanreturnorderby), and the expression must be float8 or + * float4: index_store_float8_orderby_distances() hands the executor float8 + * distances, rounding them for a float4 key, and can't deliver other types. + */ +bool +index_orderby_returnable(IndexOptInfo *index, int indexcol, Expr *orderby) +{ + Oid typ = exprType((Node *) orderby); + + if (typ != FLOAT8OID && typ != FLOAT4OID) + return false; + return (index->amcanreturnorderby != NULL && + index->amcanreturnorderby(index->opfamily[indexcol], + index->opcintype[indexcol], + castNode(OpExpr, orderby)->opno)); +} + +/* + * orderby_tlist_match + * Is targetlist expression 'expr' the value of ORDER BY expression + * 'orderby'? + * + * 'orderby' is an IndexPath's indexorderbys entry, which has the index key + * on the left; match_clause_to_ordering_op() commutes "const OP key" into + * that form. So a targetlist entry that the user wrote in the original + * "const OP key" form is the same value but not equal() to it. Accept it + * if it is the commuted operator applied to the swapped arguments; an + * operator's commutator yields the same result by definition. + * + * Shared by path_target_cost() and setrefs.c's replace_orderby_tlist_refs() + * so that what is costed is what is built. + */ +bool +orderby_tlist_match(Expr *expr, Expr *orderby) +{ + OpExpr *a, + *b; + + if (equal(expr, orderby)) + return true; + + if (!IsA(expr, OpExpr) || !IsA(orderby, OpExpr)) + return false; + a = (OpExpr *) expr; + b = (OpExpr *) orderby; + if (list_length(a->args) != 2 || list_length(b->args) != 2) + return false; + if (a->opresulttype != b->opresulttype || + a->opcollid != b->opcollid || + a->inputcollid != b->inputcollid || + get_commutator(a->opno) != b->opno) + return false; + + return equal(linitial(a->args), lsecond(b->args)) && + equal(lsecond(a->args), linitial(b->args)); +} + +/* + * path_target_cost + * Return the cost 'path' pays to evaluate 'target'. + * + * Normally that's just target->cost. But a plain IndexScan takes any + * top-level target expression that is a returnable ORDER BY expression + * (see orderby_tlist_match()) from its ORDER BY values instead of + * evaluating it (setrefs.c rewrites it into a reference to them), so it + * doesn't pay for those. Each target entry + * is counted at most once, as setrefs.c rewrites it once. + */ +static QualCost +path_target_cost(PlannerInfo *root, Path *path, PathTarget *target) +{ + QualCost cost = target->cost; + IndexPath *ipath; + ListCell *lc; + + if (!IsA(path, IndexPath) || path->pathtype != T_IndexScan) + return cost; + ipath = (IndexPath *) path; + if (ipath->indexorderbys == NIL) + return cost; + + foreach(lc, target->exprs) + { + Node *expr = (Node *) lfirst(lc); + ListCell *lo, + *lcol; + + if (IsA(expr, Var)) + continue; /* costs nothing anyway */ + + forboth(lo, ipath->indexorderbys, lcol, ipath->indexorderbycols) + { + Expr *orderby = (Expr *) lfirst(lo); + + if (index_orderby_returnable(ipath->indexinfo, lfirst_int(lcol), + orderby) && + orderby_tlist_match((Expr *) expr, orderby)) + { + QualCost ecost; + + cost_qual_eval_node(&ecost, expr, root); + cost.startup -= ecost.startup; + cost.per_tuple -= ecost.per_tuple; + break; + } + } + } + + return cost; +} + /* * create_projection_path * Creates a pathnode that represents performing a projection. @@ -2639,6 +2793,9 @@ create_projection_path(PlannerInfo *root, if (is_projection_capable_path(subpath) || equal(oldtarget->exprs, target->exprs)) { + QualCost oldcost = path_target_cost(root, subpath, oldtarget); + QualCost newcost = path_target_cost(root, subpath, target); + /* No separate Result node needed */ pathnode->dummypp = true; @@ -2648,10 +2805,10 @@ create_projection_path(PlannerInfo *root, pathnode->path.rows = subpath->rows; pathnode->path.disabled_nodes = subpath->disabled_nodes; pathnode->path.startup_cost = subpath->startup_cost + - (target->cost.startup - oldtarget->cost.startup); + (newcost.startup - oldcost.startup); pathnode->path.total_cost = subpath->total_cost + - (target->cost.startup - oldtarget->cost.startup) + - (target->cost.per_tuple - oldtarget->cost.per_tuple) * subpath->rows; + (newcost.startup - oldcost.startup) + + (newcost.per_tuple - oldcost.per_tuple) * subpath->rows; } else { @@ -2703,6 +2860,7 @@ apply_projection_to_path(PlannerInfo *root, PathTarget *target) { QualCost oldcost; + QualCost newcost; /* * If given path can't project, we might need a Result node, so make a @@ -2715,12 +2873,13 @@ apply_projection_to_path(PlannerInfo *root, * We can just jam the desired tlist into the existing path, being sure to * update its cost estimates appropriately. */ - oldcost = path->pathtarget->cost; + oldcost = path_target_cost(root, path, path->pathtarget); + newcost = path_target_cost(root, path, target); path->pathtarget = target; - path->startup_cost += target->cost.startup - oldcost.startup; - path->total_cost += target->cost.startup - oldcost.startup + - (target->cost.per_tuple - oldcost.per_tuple) * path->rows; + path->startup_cost += newcost.startup - oldcost.startup; + path->total_cost += newcost.startup - oldcost.startup + + (newcost.per_tuple - oldcost.per_tuple) * path->rows; /* * If the path happens to be a Gather or GatherMerge path, we'd like to diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index e74a95d0d22f1..6a4df53fee304 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -225,6 +225,10 @@ extern HashPath *create_hashjoin_path(PlannerInfo *root, Relids required_outer, List *hashclauses); +extern void sort_pathlist_by_cost(List *pathlist); +extern bool index_orderby_returnable(IndexOptInfo *index, int indexcol, + Expr *orderby); +extern bool orderby_tlist_match(Expr *expr, Expr *orderby); extern ProjectionPath *create_projection_path(PlannerInfo *root, RelOptInfo *rel, Path *subpath, diff --git a/src/test/regress/expected/gist.out b/src/test/regress/expected/gist.out index 648566a0bbd33..336a3965b4bfd 100644 --- a/src/test/regress/expected/gist.out +++ b/src/test/regress/expected/gist.out @@ -671,6 +671,99 @@ select count(*) filter (where dist = b <-> point(3, 4)) as same, reset enable_indexonlyscan; drop table spg_orderby; +-- The planner must not charge an ordering Index Scan for evaluating a +-- targetlist entry it takes from its ORDER BY values. With a costly ORDER BY +-- expression, the kNN scan should win over Seq Scan + Sort even when the +-- filter is selective, since the Sort plan really does evaluate it per row. +create function gist_costly_pt(point) returns point + language plpgsql immutable strict cost 10000 + as $$ begin return $1; end $$; +create temp table gist_costly (id int, p point, grp int); +insert into gist_costly + select g, point(g % 101, g % 103), g % 100 from generate_series(1, 10000) g; +create index on gist_costly using gist (gist_costly_pt(p)); +create index on gist_costly (grp); +vacuum analyze gist_costly; +set enable_bitmapscan = off; +-- (Seq Scan + Sort is a real choice here; the earlier tests turned it off.) +set enable_seqscan = on; +explain (costs off) +select id, gist_costly_pt(p) <-> point(0,0) as dist from gist_costly + where grp < 5 order by gist_costly_pt(p) <-> point(0,0); + QUERY PLAN +------------------------------------------------------------------ + Index Scan using gist_costly_gist_costly_pt_p_idx on gist_costly + Order By: (gist_costly_pt(p) <-> '(0,0)'::point) + Filter: (grp < 5) +(3 rows) + +set enable_seqscan = off; +-- The ORDER BY value is in every plan's target (as a sort column), so the +-- kNN scan used to be charged rows * cost(gist_costly_pt) for it, ~2.5e5 +-- here. It is now charged nothing for it; an expression that merely +-- contains it is still evaluated, and charged. +create function gist_costly_total(q text) returns float8 language plpgsql as +$$ declare j json; begin + execute 'explain (format json) ' || q into j; + return (j->0->'Plan'->>'Total Cost')::float8; + end $$; +set enable_sort = off; +select gist_costly_total('select id, gist_costly_pt(p) <-> point(0,0) + from gist_costly order by gist_costly_pt(p) <-> point(0,0)') < 10000 + as returned_is_free, + gist_costly_total('select id, (gist_costly_pt(p) <-> point(0,0)) * 2 + from gist_costly order by gist_costly_pt(p) <-> point(0,0)') > 100000 + as nested_is_charged; + returned_is_free | nested_is_charged +------------------+------------------- + t | t +(1 row) + +reset enable_sort; +drop function gist_costly_total(text); +reset enable_bitmapscan; +drop table gist_costly; +drop function gist_costly_pt(point); +-- The ORDER BY expression written as "const OP key" is commuted into +-- "key OP const" for the index; a targetlist entry in the original form is +-- still the same value, and is still taken from the scan. +create temp table gist_commute as + select g as id, point(g % 101, g % 103) as p from generate_series(1, 1000) g; +create index on gist_commute using gist (p); +vacuum analyze gist_commute; +set enable_seqscan = off; +set enable_bitmapscan = off; +set enable_indexonlyscan = off; +explain (verbose, costs off) +select point(5,5) <-> p as d from gist_commute order by point(5,5) <-> p limit 3; + QUERY PLAN +------------------------------------------------------------------- + Limit + Output: ((p <-> '(5,5)'::point)) + -> Index Scan using gist_commute_p_idx on pg_temp.gist_commute + Output: ((p <-> '(5,5)'::point)) + Order By: (gist_commute.p <-> '(5,5)'::point) + Order By Values Used: 1 +(6 rows) + +set extra_float_digits = 0; +select point(5,5) <-> p as d from gist_commute order by point(5,5) <-> p limit 3; + d +----------------- + 0 + 1.4142135623731 + 1.4142135623731 +(3 rows) + +reset extra_float_digits; +reset enable_seqscan; +reset enable_bitmapscan; +reset enable_indexonlyscan; +drop table gist_commute; +-- (The tests above reset these to their defaults; the rest of this file +-- wants them as set at the top.) +set enable_seqscan = off; +set enable_bitmapscan = off; -- Test that an index-only scan deforms the tuple it reconstructs with the -- descriptor the AM formed it with, not the scan slot's descriptor. create temp table gist_ios_tupdesc (a inet, r numrange); diff --git a/src/test/regress/sql/gist.sql b/src/test/regress/sql/gist.sql index 74772e357a612..209c72d1b538d 100644 --- a/src/test/regress/sql/gist.sql +++ b/src/test/regress/sql/gist.sql @@ -323,6 +323,76 @@ select count(*) filter (where dist = b <-> point(3, 4)) as same, reset enable_indexonlyscan; drop table spg_orderby; +-- The planner must not charge an ordering Index Scan for evaluating a +-- targetlist entry it takes from its ORDER BY values. With a costly ORDER BY +-- expression, the kNN scan should win over Seq Scan + Sort even when the +-- filter is selective, since the Sort plan really does evaluate it per row. +create function gist_costly_pt(point) returns point + language plpgsql immutable strict cost 10000 + as $$ begin return $1; end $$; +create temp table gist_costly (id int, p point, grp int); +insert into gist_costly + select g, point(g % 101, g % 103), g % 100 from generate_series(1, 10000) g; +create index on gist_costly using gist (gist_costly_pt(p)); +create index on gist_costly (grp); +vacuum analyze gist_costly; +set enable_bitmapscan = off; +-- (Seq Scan + Sort is a real choice here; the earlier tests turned it off.) +set enable_seqscan = on; + +explain (costs off) +select id, gist_costly_pt(p) <-> point(0,0) as dist from gist_costly + where grp < 5 order by gist_costly_pt(p) <-> point(0,0); +set enable_seqscan = off; + +-- The ORDER BY value is in every plan's target (as a sort column), so the +-- kNN scan used to be charged rows * cost(gist_costly_pt) for it, ~2.5e5 +-- here. It is now charged nothing for it; an expression that merely +-- contains it is still evaluated, and charged. +create function gist_costly_total(q text) returns float8 language plpgsql as +$$ declare j json; begin + execute 'explain (format json) ' || q into j; + return (j->0->'Plan'->>'Total Cost')::float8; + end $$; +set enable_sort = off; +select gist_costly_total('select id, gist_costly_pt(p) <-> point(0,0) + from gist_costly order by gist_costly_pt(p) <-> point(0,0)') < 10000 + as returned_is_free, + gist_costly_total('select id, (gist_costly_pt(p) <-> point(0,0)) * 2 + from gist_costly order by gist_costly_pt(p) <-> point(0,0)') > 100000 + as nested_is_charged; +reset enable_sort; +drop function gist_costly_total(text); + +reset enable_bitmapscan; +drop table gist_costly; +drop function gist_costly_pt(point); + +-- The ORDER BY expression written as "const OP key" is commuted into +-- "key OP const" for the index; a targetlist entry in the original form is +-- still the same value, and is still taken from the scan. +create temp table gist_commute as + select g as id, point(g % 101, g % 103) as p from generate_series(1, 1000) g; +create index on gist_commute using gist (p); +vacuum analyze gist_commute; +set enable_seqscan = off; +set enable_bitmapscan = off; +set enable_indexonlyscan = off; +explain (verbose, costs off) +select point(5,5) <-> p as d from gist_commute order by point(5,5) <-> p limit 3; +set extra_float_digits = 0; +select point(5,5) <-> p as d from gist_commute order by point(5,5) <-> p limit 3; +reset extra_float_digits; +reset enable_seqscan; +reset enable_bitmapscan; +reset enable_indexonlyscan; +drop table gist_commute; + +-- (The tests above reset these to their defaults; the rest of this file +-- wants them as set at the top.) +set enable_seqscan = off; +set enable_bitmapscan = off; + -- Test that an index-only scan deforms the tuple it reconstructs with the -- descriptor the AM formed it with, not the scan slot's descriptor. create temp table gist_ios_tupdesc (a inet, r numrange); From ef9ef7f7c90ee6f483822d3215ac601be0f33c06 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Wed, 7 Oct 2026 01:50:09 +0000 Subject: [PATCH 08/12] Let an ordering index scan below a join emit its ORDER BY values With the previous two commits, an ordering Index Scan returns its ORDER BY values to a target list entry that matches the ORDER BY expression, and isn't charged for it. But that only happens when the scan is the top of the scan/join tree. When the scan is below a join, its target is rel->reltarget, which holds only the Vars the rest of the query needs; the join computes the ORDER BY expression from those Vars, once per joined row, though the scan already had the value. Tom Lane pointed at this in the 2023 junk-column thread: a fix that only munges the final targetlist "leaves everything on the table as soon as there's more than one level of plan involved". Give such an IndexPath its own PathTarget: a copy of rel->reltarget with each returnable ORDER BY expression appended, but only if the query needs that expression above the scan (a top-level entry of the final targetlist, which includes sort columns). Most index paths keep sharing rel->reltarget as before. The appended expressions cost the scan nothing (cost_index() now uses path_target_cost()). Above the scan, setrefs.c's fix_join_expr() already matches whole non-Var expressions against the input targetlists before descending into them, so the join's copy becomes a reference to the scan's output with no setrefs change; and the join isn't charged for it either, since path_target_cost() leaves out of a join path's target the entries an input already emits (looking through Material and Memoize). use_physical_tlist() must not replace such a scan's targetlist with a physical one, which has only Vars; index_path_emits_orderby() tells it. Scope: plain IndexScan paths on plain base relations only. Not IndexOnlyScan (no ORDER BY value plumbing), not appendrel children (Append/MergeAppend build their own child targetlists, and the MergeAppend-over-kNN case already works through the scan/join target), and only ORDER BY expressions that pass index_orderby_returnable(). Because the join reads the value from the scan's output, an outer join nulls it correctly with the rest of that side's columns. Author: Greg Burd Discussion: https://postgr.es/m/8s5lT8erXzBMugXJQ6Wginbp_gc2B4hmCYhF0Q0GpnuK87eCAOcKs5K31AWCwraCNFIQt42wwkow_MPPubHg485MWv-zDBY4NL_JX9yJEEg=@burd.me Discussion: https://postgr.es/m/2935085.1703806977@sss.pgh.pa.us --- src/backend/optimizer/path/costsize.c | 12 +- src/backend/optimizer/plan/createplan.c | 9 ++ src/backend/optimizer/util/pathnode.c | 182 +++++++++++++++++++++++- src/include/optimizer/pathnode.h | 3 + src/test/regress/expected/gist.out | 93 ++++++++++++ src/test/regress/sql/gist.sql | 47 ++++++ 6 files changed, 340 insertions(+), 6 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 7bbddb8bee4fa..a844701b1d11c 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -564,6 +564,7 @@ cost_index(IndexPath *path, PlannerInfo *root, double loop_count, Cost min_IO_cost, max_IO_cost; QualCost qpqual_cost; + QualCost tlist_cost; Cost cpu_per_tuple; double tuples_fetched; double pages_fetched; @@ -799,9 +800,14 @@ cost_index(IndexPath *path, PlannerInfo *root, double loop_count, cpu_run_cost += cpu_per_tuple * tuples_fetched; - /* tlist eval costs are paid per output row, not per tuple scanned */ - startup_cost += path->path.pathtarget->cost.startup; - cpu_run_cost += path->path.pathtarget->cost.per_tuple * path->path.rows; + /* + * tlist eval costs are paid per output row, not per tuple scanned. An + * ordering IndexScan doesn't pay for target entries it takes from its + * ORDER BY values. + */ + tlist_cost = path_target_cost(root, (Path *) path, path->path.pathtarget); + startup_cost += tlist_cost.startup; + cpu_run_cost += tlist_cost.per_tuple * path->path.rows; /* Adjust costing for parallelism, if used. */ if (path->path.parallel_workers > 0) diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index a42ac73e7ca29..283043acfa0ce 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -938,6 +938,15 @@ use_physical_tlist(PlannerInfo *root, Path *path, int flags) return false; } + /* + * Nor if a plain index scan path was asked to emit its ORDER BY values + * (see index_path_orderby_target()); a physical tlist has only Vars, so + * whatever needs them would compute them again. + */ + if (path->pathtype == T_IndexScan && + index_path_emits_orderby((IndexPath *) path)) + return false; + /* * For an index-only scan, the "physical tlist" is the index's indextlist. * We can only return that without a projection if all the index's columns diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 272a88d6dd023..616eedf2897f1 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -1070,6 +1070,120 @@ create_samplescan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer return pathnode; } +/* + * index_path_orderby_target + * Choose the PathTarget for an ordering plain IndexScan path. + * + * Normally an IndexPath emits rel->reltarget, which holds only the Vars (and + * PlaceHolderVars) the rest of the query needs. An ORDER BY expression is + * not in it: it is computed by whatever plan node first needs it. When the + * scan is the top of the scan/join tree that is the scan itself, and + * setrefs.c lets it take the value from its ORDER BY values. But when the + * scan is below a join, the join computes the expression from the Vars, + * once per joined row, though the scan already had the value. + * + * So if the scan can return any of its ORDER BY values (see + * index_orderby_returnable()), and the query needs that expression above + * the scan (it is in the final targetlist, which includes sort columns), + * give the path a copy of rel->reltarget with those expressions appended. + * They cost the scan nothing (path_target_cost() exempts them), and + * setrefs.c's fix_join_expr() matches them in the parent join's expressions, + * replacing its copy with a reference to the scan's output. + * + * Restricted to plain base relations: an appendrel child's paths are wrapped + * by Append/MergeAppend, which build their own targetlists, and the top-level + * scan/join target already handles the unjoined case. Returns + * rel->reltarget unchanged when there's nothing to add, which is the common + * case, so most paths share it as before. + */ +static PathTarget * +index_path_orderby_target(PlannerInfo *root, IndexOptInfo *index, + List *indexorderbys, List *indexorderbycols) +{ + RelOptInfo *rel = index->rel; + PathTarget *target = NULL; + ListCell *lo, + *lcol; + + if (indexorderbys == NIL || rel->reloptkind != RELOPT_BASEREL || + root->processed_tlist == NIL) + return rel->reltarget; + + forboth(lo, indexorderbys, lcol, indexorderbycols) + { + Expr *orderby = (Expr *) lfirst(lo); + bool needed = false; + ListCell *lt; + + if (!index_orderby_returnable(index, lfirst_int(lcol), orderby)) + continue; + + /* + * Only if some top-level targetlist entry is this expression; an + * expression that merely contains it would not be rewritten, and + * emitting the value would just widen the tuple. + */ + foreach(lt, root->processed_tlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lt); + + if (orderby_tlist_match(tle->expr, orderby)) + { + needed = true; + break; + } + } + if (!needed) + continue; + + if (target == NULL) + target = copy_pathtarget(rel->reltarget); + add_new_column_to_pathtarget(target, orderby); + } + + if (target == NULL) + return rel->reltarget; + + /* add_new_column_to_pathtarget doesn't maintain cost and width */ + return set_pathtarget_cost_width(root, target); +} + +/* + * index_path_emits_orderby + * Does this plain IndexScan path's target hold a value it takes from its + * ORDER BY values? + * + * use_physical_tlist() must not replace such a target with the scan's + * physical tlist, which has only Vars, or whatever needs the value would + * compute it again. + */ +bool +index_path_emits_orderby(IndexPath *ipath) +{ + ListCell *lc; + + if (ipath->path.pathtype != T_IndexScan || ipath->indexorderbys == NIL) + return false; + + foreach(lc, ipath->path.pathtarget->exprs) + { + Expr *expr = (Expr *) lfirst(lc); + ListCell *lo, + *lcol; + + if (IsA(expr, Var)) + continue; + forboth(lo, ipath->indexorderbys, lcol, ipath->indexorderbycols) + { + if (index_orderby_returnable(ipath->indexinfo, lfirst_int(lcol), + (Expr *) lfirst(lo)) && + orderby_tlist_match(expr, (Expr *) lfirst(lo))) + return true; + } + } + return false; +} + /* * create_index_path * Creates a path node for an index scan. @@ -1109,7 +1223,9 @@ create_index_path(PlannerInfo *root, pathnode->path.pathtype = indexonly ? T_IndexOnlyScan : T_IndexScan; pathnode->path.parent = rel; - pathnode->path.pathtarget = rel->reltarget; + pathnode->path.pathtarget = indexonly ? rel->reltarget : + index_path_orderby_target(root, index, indexorderbys, + indexorderbycols); pathnode->path.param_info = get_baserel_parampathinfo(root, rel, required_outer); pathnode->path.parallel_aware = false; @@ -2679,24 +2795,84 @@ orderby_tlist_match(Expr *expr, Expr *orderby) equal(lsecond(a->args), linitial(b->args)); } +/* + * join_target_cost + * path_target_cost() for a join path. + * + * A join takes any top-level target expression that one of its inputs + * already emits from that input's output instead of evaluating it again: + * setrefs.c's fix_join_expr() matches whole non-Var expressions against the + * input targetlists before descending into them. Join inputs ordinarily + * emit only Vars and PlaceHolderVars, so this changes nothing; but an + * ordering IndexScan input may emit the ORDER BY values it returns (see + * index_path_orderby_target()). Look through Material and Memoize, which + * pass their input's targetlist up unchanged. We look only at the join's + * immediate inputs; the value can't reach a higher join without passing + * through this one's targetlist, which is the joinrel's and has only Vars. + */ +static QualCost +join_target_cost(PlannerInfo *root, JoinPath *jpath, PathTarget *target) +{ + QualCost cost = target->cost; + Path *inputs[2] = {jpath->outerjoinpath, jpath->innerjoinpath}; + ListCell *lc; + + foreach(lc, target->exprs) + { + Node *expr = (Node *) lfirst(lc); + + if (IsA(expr, Var) || IsA(expr, PlaceHolderVar)) + continue; + + for (int i = 0; i < 2; i++) + { + Path *input = inputs[i]; + + while (IsA(input, MaterialPath) || IsA(input, MemoizePath)) + input = IsA(input, MaterialPath) ? + ((MaterialPath *) input)->subpath : + ((MemoizePath *) input)->subpath; + + if (IsA(input, IndexPath) && + index_path_emits_orderby((IndexPath *) input) && + list_member(input->pathtarget->exprs, expr)) + { + QualCost ecost; + + cost_qual_eval_node(&ecost, expr, root); + cost.startup -= ecost.startup; + cost.per_tuple -= ecost.per_tuple; + break; + } + } + } + + return cost; +} + /* * path_target_cost * Return the cost 'path' pays to evaluate 'target'. * - * Normally that's just target->cost. But a plain IndexScan takes any + * Normally that's just target->cost. A join path doesn't pay for entries + * an input already emits (see join_target_cost()). And a plain IndexScan + * takes any * top-level target expression that is a returnable ORDER BY expression * (see orderby_tlist_match()) from its ORDER BY values instead of * evaluating it (setrefs.c rewrites it into a reference to them), so it * doesn't pay for those. Each target entry * is counted at most once, as setrefs.c rewrites it once. */ -static QualCost +QualCost path_target_cost(PlannerInfo *root, Path *path, PathTarget *target) { QualCost cost = target->cost; IndexPath *ipath; ListCell *lc; + if (IsA(path, NestPath) || IsA(path, MergePath) || IsA(path, HashPath)) + return join_target_cost(root, (JoinPath *) path, target); + if (!IsA(path, IndexPath) || path->pathtype != T_IndexScan) return cost; ipath = (IndexPath *) path; diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index 6a4df53fee304..9dade3b9416bf 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -229,6 +229,9 @@ extern void sort_pathlist_by_cost(List *pathlist); extern bool index_orderby_returnable(IndexOptInfo *index, int indexcol, Expr *orderby); extern bool orderby_tlist_match(Expr *expr, Expr *orderby); +extern QualCost path_target_cost(PlannerInfo *root, Path *path, + PathTarget *target); +extern bool index_path_emits_orderby(IndexPath *ipath); extern ProjectionPath *create_projection_path(PlannerInfo *root, RelOptInfo *rel, Path *subpath, diff --git a/src/test/regress/expected/gist.out b/src/test/regress/expected/gist.out index 336a3965b4bfd..06213604df003 100644 --- a/src/test/regress/expected/gist.out +++ b/src/test/regress/expected/gist.out @@ -760,6 +760,99 @@ reset enable_seqscan; reset enable_bitmapscan; reset enable_indexonlyscan; drop table gist_commute; +-- An ordering Index Scan below a join emits the ORDER BY value it returns, +-- so the join above reads it instead of evaluating the expression again. +-- The counting function shows the expression is evaluated zero times. +create sequence gist_join_cnt; +create function gist_join_pt(point) returns point + language plpgsql immutable strict cost 1000 + as $$ begin perform nextval('public.gist_join_cnt'); return $1; end $$; +create temp table gist_join_fact as + select g as id, point(g % 101, g % 103) as p from generate_series(1, 1000) g; +create index on gist_join_fact using gist (gist_join_pt(p)); +create temp table gist_join_dim as select g as id from generate_series(1, 1000, 2) g; +create index on gist_join_dim (id); +vacuum analyze gist_join_fact; +vacuum analyze gist_join_dim; +set enable_seqscan = off; +set enable_bitmapscan = off; +set enable_indexonlyscan = off; +set enable_hashjoin = off; +set enable_mergejoin = off; +set enable_sort = off; +explain (verbose, costs off) +select f.id, gist_join_pt(f.p) <-> point(5,5) as d + from gist_join_fact f join gist_join_dim j on j.id = f.id + order by gist_join_pt(f.p) <-> point(5,5) limit 3; + QUERY PLAN +-------------------------------------------------------------------------------------------- + Limit + Output: f.id, ((gist_join_pt(f.p) <-> '(5,5)'::point)) + -> Nested Loop + Output: f.id, ((gist_join_pt(f.p) <-> '(5,5)'::point)) + -> Index Scan using gist_join_fact_gist_join_pt_p_idx on pg_temp.gist_join_fact f + Output: f.id, f.p, ((gist_join_pt(f.p) <-> '(5,5)'::point)) + Order By: (gist_join_pt(f.p) <-> '(5,5)'::point) + Order By Values Used: 1 + -> Index Scan using gist_join_dim_id_idx on pg_temp.gist_join_dim j + Output: j.id + Index Cond: (j.id = f.id) +(11 rows) + +select setval('gist_join_cnt', 1, false); + setval +-------- + 1 +(1 row) + +set extra_float_digits = 0; +select f.id, gist_join_pt(f.p) <-> point(5,5) as d + from gist_join_fact f join gist_join_dim j on j.id = f.id + order by gist_join_pt(f.p) <-> point(5,5) limit 3; + id | d +-----+------------------ + 5 | 0 + 107 | 1.4142135623731 + 209 | 2.82842712474619 +(3 rows) + +reset extra_float_digits; +select nextval('gist_join_cnt') - 1 as calls_during_join; + calls_during_join +------------------- + 0 +(1 row) + +-- Nothing extra is emitted when the query doesn't need the value above the +-- scan. +explain (verbose, costs off) +select f.id from gist_join_fact f join gist_join_dim j on j.id = f.id + order by gist_join_pt(f.p) <-> point(5,5) limit 3; + QUERY PLAN +-------------------------------------------------------------------------------------------- + Limit + Output: f.id, ((gist_join_pt(f.p) <-> '(5,5)'::point)) + -> Nested Loop + Output: f.id, ((gist_join_pt(f.p) <-> '(5,5)'::point)) + -> Index Scan using gist_join_fact_gist_join_pt_p_idx on pg_temp.gist_join_fact f + Output: f.id, f.p, ((gist_join_pt(f.p) <-> '(5,5)'::point)) + Order By: (gist_join_pt(f.p) <-> '(5,5)'::point) + Order By Values Used: 1 + -> Index Scan using gist_join_dim_id_idx on pg_temp.gist_join_dim j + Output: j.id + Index Cond: (j.id = f.id) +(11 rows) + +reset enable_seqscan; +reset enable_bitmapscan; +reset enable_indexonlyscan; +reset enable_hashjoin; +reset enable_mergejoin; +reset enable_sort; +drop table gist_join_fact; +drop table gist_join_dim; +drop function gist_join_pt(point); +drop sequence gist_join_cnt; -- (The tests above reset these to their defaults; the rest of this file -- wants them as set at the top.) set enable_seqscan = off; diff --git a/src/test/regress/sql/gist.sql b/src/test/regress/sql/gist.sql index 209c72d1b538d..3cea6f47557c4 100644 --- a/src/test/regress/sql/gist.sql +++ b/src/test/regress/sql/gist.sql @@ -388,6 +388,53 @@ reset enable_bitmapscan; reset enable_indexonlyscan; drop table gist_commute; +-- An ordering Index Scan below a join emits the ORDER BY value it returns, +-- so the join above reads it instead of evaluating the expression again. +-- The counting function shows the expression is evaluated zero times. +create sequence gist_join_cnt; +create function gist_join_pt(point) returns point + language plpgsql immutable strict cost 1000 + as $$ begin perform nextval('public.gist_join_cnt'); return $1; end $$; +create temp table gist_join_fact as + select g as id, point(g % 101, g % 103) as p from generate_series(1, 1000) g; +create index on gist_join_fact using gist (gist_join_pt(p)); +create temp table gist_join_dim as select g as id from generate_series(1, 1000, 2) g; +create index on gist_join_dim (id); +vacuum analyze gist_join_fact; +vacuum analyze gist_join_dim; +set enable_seqscan = off; +set enable_bitmapscan = off; +set enable_indexonlyscan = off; +set enable_hashjoin = off; +set enable_mergejoin = off; +set enable_sort = off; +explain (verbose, costs off) +select f.id, gist_join_pt(f.p) <-> point(5,5) as d + from gist_join_fact f join gist_join_dim j on j.id = f.id + order by gist_join_pt(f.p) <-> point(5,5) limit 3; +select setval('gist_join_cnt', 1, false); +set extra_float_digits = 0; +select f.id, gist_join_pt(f.p) <-> point(5,5) as d + from gist_join_fact f join gist_join_dim j on j.id = f.id + order by gist_join_pt(f.p) <-> point(5,5) limit 3; +reset extra_float_digits; +select nextval('gist_join_cnt') - 1 as calls_during_join; +-- Nothing extra is emitted when the query doesn't need the value above the +-- scan. +explain (verbose, costs off) +select f.id from gist_join_fact f join gist_join_dim j on j.id = f.id + order by gist_join_pt(f.p) <-> point(5,5) limit 3; +reset enable_seqscan; +reset enable_bitmapscan; +reset enable_indexonlyscan; +reset enable_hashjoin; +reset enable_mergejoin; +reset enable_sort; +drop table gist_join_fact; +drop table gist_join_dim; +drop function gist_join_pt(point); +drop sequence gist_join_cnt; + -- (The tests above reset these to their defaults; the rest of this file -- wants them as set at the top.) set enable_seqscan = off; From f23832d75eb8a3cc5e5df772e1bbed83dd6df3d4 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Wed, 7 Oct 2026 01:50:14 +0000 Subject: [PATCH 09/12] Don't charge an index-only scan for index expressions it returns An index-only scan on an index with an expression column reads that expression's value from the index: set_indexonlyscan_references() replaces a target list entry equal() to a returnable index expression with a reference to that column. But the planner still charges the scan the full cost of evaluating the expression for every row, as if it computed it. Heikki Linnakangas showed this in the 2023 junk-column thread with an index on (i, expensive(i)) and a function of COST 1000000: the index-only scan was estimated at 25 million, like the Seq Scan that really does evaluate expensive(i). path_target_cost() now also handles index-only scan paths: a top-level target expression equal() to a returnable index expression costs nothing. cost_index() already goes through path_target_cost(), so the index-only scan's estimate drops to what it actually does. This corrects the estimate of an index-only scan path that is already in the running, e.g. one kept for its pathkeys. It does not make such a path win the base relation's add_path() comparison when its only merit is the cheaper expression: at that point every path of the rel carries rel->reltarget, which holds only Vars, and the expression is charged to all of them alike afterwards. Doing that would mean giving index-only paths their own targets, as the previous commit does for ordering Index Scans; that's left for later. Author: Greg Burd Discussion: https://postgr.es/m/8s5lT8erXzBMugXJQ6Wginbp_gc2B4hmCYhF0Q0GpnuK87eCAOcKs5K31AWCwraCNFIQt42wwkow_MPPubHg485MWv-zDBY4NL_JX9yJEEg=@burd.me Discussion: https://postgr.es/m/356d5811-d8f0-4c29-a601-b61334337b11@iki.fi --- src/backend/optimizer/util/pathnode.c | 59 ++++++++++++++++++++++++++- src/test/regress/expected/gist.out | 33 +++++++++++++++ src/test/regress/sql/gist.sql | 23 +++++++++++ 3 files changed, 113 insertions(+), 2 deletions(-) diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 616eedf2897f1..de05f2f400b40 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -2795,6 +2795,57 @@ orderby_tlist_match(Expr *expr, Expr *orderby) equal(lsecond(a->args), linitial(b->args)); } +/* + * indexonly_target_cost + * path_target_cost() for an index-only scan path. + * + * setrefs.c's set_indexonlyscan_references() matches target expressions + * against the index's returnable columns, expression columns included, and + * replaces a match with a reference to that column: the scan reads the + * value the index stored instead of computing it. So a top-level target + * expression equal() to a returnable index expression costs nothing. Only + * top-level entries are considered, as elsewhere; that's what's normally in + * a scan's target. + */ +static QualCost +indexonly_target_cost(PlannerInfo *root, IndexPath *ipath, PathTarget *target) +{ + QualCost cost = target->cost; + IndexOptInfo *index = ipath->indexinfo; + ListCell *lc; + + if (index->indexprs == NIL) + return cost; /* only Vars, which cost nothing anyway */ + + foreach(lc, target->exprs) + { + Node *expr = (Node *) lfirst(lc); + ListCell *lt; + int i = 0; + + if (IsA(expr, Var)) + continue; + + foreach(lt, index->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lt); + + if (index->canreturn[i++] && !IsA(tle->expr, Var) && + equal(expr, tle->expr)) + { + QualCost ecost; + + cost_qual_eval_node(&ecost, expr, root); + cost.startup -= ecost.startup; + cost.per_tuple -= ecost.per_tuple; + break; + } + } + } + + return cost; +} + /* * join_target_cost * path_target_cost() for a join path. @@ -2855,8 +2906,9 @@ join_target_cost(PlannerInfo *root, JoinPath *jpath, PathTarget *target) * Return the cost 'path' pays to evaluate 'target'. * * Normally that's just target->cost. A join path doesn't pay for entries - * an input already emits (see join_target_cost()). And a plain IndexScan - * takes any + * an input already emits (see join_target_cost()), nor an index-only scan + * for entries it reads from the index (see indexonly_target_cost()). And a + * plain IndexScan takes any * top-level target expression that is a returnable ORDER BY expression * (see orderby_tlist_match()) from its ORDER BY values instead of * evaluating it (setrefs.c rewrites it into a reference to them), so it @@ -2873,6 +2925,9 @@ path_target_cost(PlannerInfo *root, Path *path, PathTarget *target) if (IsA(path, NestPath) || IsA(path, MergePath) || IsA(path, HashPath)) return join_target_cost(root, (JoinPath *) path, target); + if (IsA(path, IndexPath) && path->pathtype == T_IndexOnlyScan) + return indexonly_target_cost(root, (IndexPath *) path, target); + if (!IsA(path, IndexPath) || path->pathtype != T_IndexScan) return cost; ipath = (IndexPath *) path; diff --git a/src/test/regress/expected/gist.out b/src/test/regress/expected/gist.out index 06213604df003..d9a580826e3c3 100644 --- a/src/test/regress/expected/gist.out +++ b/src/test/regress/expected/gist.out @@ -853,6 +853,39 @@ drop table gist_join_fact; drop table gist_join_dim; drop function gist_join_pt(point); drop sequence gist_join_cnt; +-- An index-only scan isn't charged for evaluating a target expression it +-- reads from a returnable index expression column: it reads the stored +-- value instead. (Not GiST-specific; btree returns expression columns.) +create function gist_ios_costly(int) returns int + language plpgsql immutable strict cost 100000 + as $$ begin return $1; end $$; +create temp table gist_ios_cost (i int); +insert into gist_ios_cost select g from generate_series(1, 10000) g; +create index on gist_ios_cost (i, gist_ios_costly(i)); +vacuum analyze gist_ios_cost; +create function gist_ios_total(q text) returns float8 language plpgsql as +$$ declare j json; begin + execute 'explain (format json) ' || q into j; + return (j->0->'Plan'->>'Total Cost')::float8; + end $$; +explain (verbose, costs off) +select gist_ios_costly(i) from gist_ios_cost order by i; + QUERY PLAN +-------------------------------------------------------------------------------------- + Index Only Scan using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost + Output: (gist_ios_costly(i)), i +(2 rows) + +select gist_ios_total('select gist_ios_costly(i) from gist_ios_cost order by i') + < 10000 as ios_expr_is_free; + ios_expr_is_free +------------------ + t +(1 row) + +drop function gist_ios_total(text); +drop table gist_ios_cost; +drop function gist_ios_costly(int); -- (The tests above reset these to their defaults; the rest of this file -- wants them as set at the top.) set enable_seqscan = off; diff --git a/src/test/regress/sql/gist.sql b/src/test/regress/sql/gist.sql index 3cea6f47557c4..137170aae90f7 100644 --- a/src/test/regress/sql/gist.sql +++ b/src/test/regress/sql/gist.sql @@ -435,6 +435,29 @@ drop table gist_join_dim; drop function gist_join_pt(point); drop sequence gist_join_cnt; +-- An index-only scan isn't charged for evaluating a target expression it +-- reads from a returnable index expression column: it reads the stored +-- value instead. (Not GiST-specific; btree returns expression columns.) +create function gist_ios_costly(int) returns int + language plpgsql immutable strict cost 100000 + as $$ begin return $1; end $$; +create temp table gist_ios_cost (i int); +insert into gist_ios_cost select g from generate_series(1, 10000) g; +create index on gist_ios_cost (i, gist_ios_costly(i)); +vacuum analyze gist_ios_cost; +create function gist_ios_total(q text) returns float8 language plpgsql as +$$ declare j json; begin + execute 'explain (format json) ' || q into j; + return (j->0->'Plan'->>'Total Cost')::float8; + end $$; +explain (verbose, costs off) +select gist_ios_costly(i) from gist_ios_cost order by i; +select gist_ios_total('select gist_ios_costly(i) from gist_ios_cost order by i') + < 10000 as ios_expr_is_free; +drop function gist_ios_total(text); +drop table gist_ios_cost; +drop function gist_ios_costly(int); + -- (The tests above reset these to their defaults; the rest of this file -- wants them as set at the top.) set enable_seqscan = off; From 9c22dc1ed9ae6b250256b7d48f26b418df7b391e Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Wed, 7 Oct 2026 14:34:42 -0400 Subject: [PATCH 10/12] Let an index-only scan win on the index expressions it returns The previous commit stopped charging an index-only scan for target expressions it reads from the index, but that only changes the plan when the index-only scan path is already in the running for some other reason, such as useful pathkeys. Heikki Linnakangas's example, an index on (i, expensive(i)) and SELECT expensive(i) FROM atab ORDER BY expensive(i); still got a Seq Scan and evaluated expensive() for every row. When the rel's paths are compared, they all carry rel->reltarget, which holds only Vars. The index-only scan costs more than the seq scan, and the expression it would have read for free is charged to the winner later. Give an index-only scan path its own PathTarget: rel->reltarget plus each returnable index expression that is a top-level entry of the query's targetlist. This is the same thing the earlier commit does for an ordering Index Scan's ORDER BY values. Teach add_path() and add_partial_path() to treat base-rel index paths with different targets as incomparable, the way they treat different pathkeys, so neither dominates the other on cost alone. When the scan/join target is applied, the seq scan pays for evaluating the expression and the index-only scan doesn't, so the index-only scan wins on cost when it should. A path with its own target also needs the projection even when the scan/join target's exprs equal the reltarget's, since its target has extra columns. Heikki's query now plans as Sort over an Index Only Scan: estimated 930 instead of 25000809. With a function that really costs something, it ran about 20 times faster. This is limited to the query's sole base relation, where the value goes straight to the scan/join target. Below a join, the join would have to credit the input for the value and the joinrel's add_path() would have to keep it; that isn't attempted here. Author: Greg Burd Discussion: https://postgr.es/m/8s5lT8erXzBMugXJQ6Wginbp_gc2B4hmCYhF0Q0GpnuK87eCAOcKs5K31AWCwraCNFIQt42wwkow_MPPubHg485MWv-zDBY4NL_JX9yJEEg=@burd.me Discussion: https://postgr.es/m/356d5811-d8f0-4c29-a601-b61334337b11@iki.fi --- src/backend/optimizer/plan/planner.c | 28 ++++++--- src/backend/optimizer/util/pathnode.c | 91 ++++++++++++++++++++++++++- src/test/regress/expected/gist.out | 46 ++++++++++++++ src/test/regress/sql/gist.sql | 14 +++++ 4 files changed, 169 insertions(+), 10 deletions(-) diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 567edfe03e2e0..f7fe5a66f2019 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -8191,6 +8191,7 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, bool tlist_same_exprs) { bool rel_is_partitioned = IS_PARTITIONED_REL(rel); + bool projected = false; PathTarget *scanjoin_target; ListCell *lc; @@ -8260,10 +8261,18 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, * If the tlist exprs are the same, we can just inject the sortgroupref * information into the existing pathtargets. Otherwise, replace each * path with a projection path that generates the SRF-free scan/join - * target. We modify the lists in place. That usually adds the same cost - * to every path, but a plain IndexScan doesn't pay for target entries it - * takes from its ORDER BY values, so it can become cheaper relative to - * the others; restore the cost ordering afterwards. + * target. We modify the lists in place. + * + * An index path may emit its own target, rel->reltarget plus a value it + * reads from the index (see indexonly_path_target() and + * index_path_orderby_target()). Such a path needs a projection even when + * the exprs are the same, since its target has more columns than + * scanjoin_target's sortgrouprefs describe. + * + * A projection usually adds the same cost to every path, but not always: + * an index path isn't charged for target entries it reads from the index, + * and a path that can't project gets a Result node. So when any path was + * replaced, restore the cost ordering afterwards. */ foreach(lc, rel->pathlist) { @@ -8272,7 +8281,7 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, /* Shouldn't have any parameterized paths anymore */ Assert(subpath->param_info == NULL); - if (tlist_same_exprs) + if (tlist_same_exprs && subpath->pathtarget == rel->reltarget) subpath->pathtarget->sortgrouprefs = scanjoin_target->sortgrouprefs; else @@ -8282,10 +8291,12 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, newpath = (Path *) create_projection_path(root, rel, subpath, scanjoin_target); lfirst(lc) = newpath; + projected = true; } } - if (!tlist_same_exprs) + if (projected) sort_pathlist_by_cost(rel->pathlist); + projected = false; /* Likewise adjust the targets for any partial paths. */ foreach(lc, rel->partial_pathlist) @@ -8295,7 +8306,7 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, /* Shouldn't have any parameterized paths anymore */ Assert(subpath->param_info == NULL); - if (tlist_same_exprs) + if (tlist_same_exprs && subpath->pathtarget == rel->reltarget) subpath->pathtarget->sortgrouprefs = scanjoin_target->sortgrouprefs; else @@ -8305,9 +8316,10 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, newpath = (Path *) create_projection_path(root, rel, subpath, scanjoin_target); lfirst(lc) = newpath; + projected = true; } } - if (!tlist_same_exprs) + if (projected) sort_pathlist_by_cost(rel->partial_pathlist); /* diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index de05f2f400b40..eb9dac861ad62 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -386,6 +386,27 @@ set_cheapest(RelOptInfo *parent_rel) parent_rel->cheapest_parameterized_paths = parameterized_paths; } +/* + * pathtargets_differ + * Do two paths of base relation 'rel' emit different targets? + * + * A base relation's paths normally all emit rel->reltarget, but an index + * path may also emit a value the query needs above the scan, one it has + * for free and other paths must compute (see indexonly_path_target() and + * index_path_orderby_target()). That cost isn't in either path's cost + * yet; it is charged when the expression is. So add_path() treats paths + * with different targets like paths with different pathkeys, and keeps + * both. + */ +static inline bool +pathtargets_differ(RelOptInfo *rel, Path *path1, Path *path2) +{ + return rel->reloptkind == RELOPT_BASEREL && + (IsA(path1, IndexPath) || IsA(path2, IndexPath)) && + path1->pathtarget != path2->pathtarget && + !equal(path1->pathtarget->exprs, path2->pathtarget->exprs); +} + /* * add_path * Consider a potential implementation path for the specified parent rel, @@ -513,6 +534,8 @@ add_path(RelOptInfo *parent_rel, Path *new_path) old_path_pathkeys = old_path->param_info ? NIL : old_path->pathkeys; keyscmp = compare_pathkeys(new_path_pathkeys, old_path_pathkeys); + if (pathtargets_differ(parent_rel, new_path, old_path)) + keyscmp = PATHKEYS_DIFFERENT; if (keyscmp != PATHKEYS_DIFFERENT) { switch (costcmp) @@ -818,8 +841,10 @@ add_partial_path(RelOptInfo *parent_rel, Path *new_path) bool remove_old = false; /* unless new proves superior */ PathKeysComparison keyscmp; - /* Compare pathkeys. */ + /* Compare pathkeys, and targets as in add_path(). */ keyscmp = compare_pathkeys(new_path->pathkeys, old_path->pathkeys); + if (pathtargets_differ(parent_rel, new_path, old_path)) + keyscmp = PATHKEYS_DIFFERENT; /* * Unless pathkeys are incompatible, see if one of the paths dominates @@ -1148,6 +1173,67 @@ index_path_orderby_target(PlannerInfo *root, IndexOptInfo *index, return set_pathtarget_cost_width(root, target); } +/* + * indexonly_path_target + * Choose the PathTarget for an index-only scan path. + * + * An index-only scan reads a returnable index expression column instead of + * computing it (set_indexonlyscan_references()), and isn't charged for it + * (indexonly_target_cost()). But that only helps if the path survives + * add_path(), and with rel->reltarget, which has only Vars, nothing + * distinguishes it from a seq scan that will have to compute the expression + * later; it loses on cost before the expression is charged to anyone. So + * if the query's targetlist has a top-level entry equal() to a returnable + * index expression, give the path a copy of rel->reltarget with that + * expression appended. add_path() keeps paths with different targets + * (see pathtargets_differ()), and once the scan/join target is applied, the + * paths that must compute the expression pay for it and this one doesn't. + * + * This is done only when the rel is the query's sole base relation, where + * the value goes straight to the scan/join target. Below a join, the join + * would have to credit it (as join_target_cost() does for ORDER BY values) + * and the joinrel's add_path() would have to keep it; not attempted yet. + */ +static PathTarget * +indexonly_path_target(PlannerInfo *root, IndexOptInfo *index) +{ + RelOptInfo *rel = index->rel; + PathTarget *target = NULL; + ListCell *lt; + int i = 0; + + if (index->indexprs == NIL || rel->reloptkind != RELOPT_BASEREL || + root->processed_tlist == NIL || + !bms_equal(rel->relids, root->all_query_rels)) + return rel->reltarget; + + foreach(lt, index->indextlist) + { + TargetEntry *itle = lfirst_node(TargetEntry, lt); + ListCell *lq; + + if (!index->canreturn[i++] || IsA(itle->expr, Var)) + continue; + + foreach(lq, root->processed_tlist) + { + if (equal(lfirst_node(TargetEntry, lq)->expr, itle->expr)) + { + if (target == NULL) + target = copy_pathtarget(rel->reltarget); + add_column_to_pathtarget(target, itle->expr, 0); + break; + } + } + } + + if (target == NULL) + return rel->reltarget; + + /* add_column_to_pathtarget doesn't maintain cost and width */ + return set_pathtarget_cost_width(root, target); +} + /* * index_path_emits_orderby * Does this plain IndexScan path's target hold a value it takes from its @@ -1223,7 +1309,8 @@ create_index_path(PlannerInfo *root, pathnode->path.pathtype = indexonly ? T_IndexOnlyScan : T_IndexScan; pathnode->path.parent = rel; - pathnode->path.pathtarget = indexonly ? rel->reltarget : + pathnode->path.pathtarget = indexonly ? + indexonly_path_target(root, index) : index_path_orderby_target(root, index, indexorderbys, indexorderbycols); pathnode->path.param_info = get_baserel_parampathinfo(root, rel, diff --git a/src/test/regress/expected/gist.out b/src/test/regress/expected/gist.out index d9a580826e3c3..9c06a949ea1d5 100644 --- a/src/test/regress/expected/gist.out +++ b/src/test/regress/expected/gist.out @@ -883,6 +883,52 @@ select gist_ios_total('select gist_ios_costly(i) from gist_ios_cost order by i') t (1 row) +-- With nothing else to recommend it (no qual, no useful order), an +-- index-only scan that reads the expression still beats a seq scan that has +-- to compute it: add_path() keeps both, since their targets differ. +explain (verbose, costs off) +select gist_ios_costly(i) from gist_ios_cost order by gist_ios_costly(i); + QUERY PLAN +-------------------------------------------------------------------------------------------- + Sort + Output: (gist_ios_costly(i)) + Sort Key: (gist_ios_costly(gist_ios_cost.i)) + -> Index Only Scan using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost + Output: (gist_ios_costly(i)) +(5 rows) + +explain (verbose, costs off) +select gist_ios_costly(i) from gist_ios_cost; + QUERY PLAN +-------------------------------------------------------------------------------------- + Index Only Scan using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost + Output: (gist_ios_costly(i)) +(2 rows) + +-- Ordered by the index, the scan emits the final targetlist directly and +-- still reads the expression from the index. +explain (verbose, costs off) +select gist_ios_costly(i) from gist_ios_cost order by i desc limit 3; + QUERY PLAN +----------------------------------------------------------------------------------------------------- + Limit + Output: (gist_ios_costly(i)), i + -> Index Only Scan Backward using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost + Output: (gist_ios_costly(i)), i +(4 rows) + +-- Below a join it isn't considered. +explain (costs off) +select gist_ios_costly(a.i) from gist_ios_cost a join gist_ios_cost b using (i); + QUERY PLAN +----------------------------------------- + Hash Join + Hash Cond: (a.i = b.i) + -> Seq Scan on gist_ios_cost a + -> Hash + -> Seq Scan on gist_ios_cost b +(5 rows) + drop function gist_ios_total(text); drop table gist_ios_cost; drop function gist_ios_costly(int); diff --git a/src/test/regress/sql/gist.sql b/src/test/regress/sql/gist.sql index 137170aae90f7..04402409b0886 100644 --- a/src/test/regress/sql/gist.sql +++ b/src/test/regress/sql/gist.sql @@ -454,6 +454,20 @@ explain (verbose, costs off) select gist_ios_costly(i) from gist_ios_cost order by i; select gist_ios_total('select gist_ios_costly(i) from gist_ios_cost order by i') < 10000 as ios_expr_is_free; +-- With nothing else to recommend it (no qual, no useful order), an +-- index-only scan that reads the expression still beats a seq scan that has +-- to compute it: add_path() keeps both, since their targets differ. +explain (verbose, costs off) +select gist_ios_costly(i) from gist_ios_cost order by gist_ios_costly(i); +explain (verbose, costs off) +select gist_ios_costly(i) from gist_ios_cost; +-- Ordered by the index, the scan emits the final targetlist directly and +-- still reads the expression from the index. +explain (verbose, costs off) +select gist_ios_costly(i) from gist_ios_cost order by i desc limit 3; +-- Below a join it isn't considered. +explain (costs off) +select gist_ios_costly(a.i) from gist_ios_cost a join gist_ios_cost b using (i); drop function gist_ios_total(text); drop table gist_ios_cost; drop function gist_ios_costly(int); From 457475ad9b1f3c1be5a647dae3dc89569bebd9c1 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Thu, 8 Oct 2026 07:55:02 -0400 Subject: [PATCH 11/12] Carry index-computed values up through joins and Appends The previous commit lets an index-only scan beat a seq scan when it reads an expression the query's targetlist needs, but only when the scan's rel is the only one in the query and the targetlist entry is that exact expression. Below a join the scan isn't chosen, and the expression is computed above it as before. Nor is it chosen when the expression is part of a larger one, such as expensive(i) + 1, or when the table is partitioned. Generalize the extra values an index path may emit, which 0003 added for an ordering IndexScan's ORDER BY values and the previous commit for an index-only scan's expression columns: * A join emits what its inputs emit, so the value reaches the scan/join target from any depth. fix_join_expr() already matches whole expressions in an input's targetlist, and path_target_cost() now credits them wherever they appear in an expression, as setrefs.c replaces them. Material and Memoize share their input's target. * An index-only scan emits a returnable index expression wherever it appears in the targetlist, including ORDER BY columns. Its qpquals aren't charged for one either, since set_indexonlyscan_references() rewrites them too. * For a partitioned table whose partitions all have the expression index, an unordered Append over child paths that all emit the value emits it too. create_append_child_plan() puts each child's values in the Append's order, translating them to the child, and drops any extra values a child emits for an Append that doesn't. * Hash and merge joins look only at each input's cheapest path, which rarely emits the value. Also try each input's cheapest path that does. * add_path_precheck() and add_partial_path_precheck() can't see the values, so they would reject a join path over such an input that add_path() keeps. New add_path_precheck_extras() and add_partial_path_precheck_extras() take the values the path will emit, and let an existing path dominate it only if that path emits them all as well. The old functions keep their signatures. add_path() and add_partial_path() compare the extra values like pathkeys: a path that emits a superset of another's is better, so it can dominate it, but not the other way round. The previous commit's rule kept both whenever the targets differed, which kept, for example, a seq scan that a cheaper, better-ordered kNN path dominates. The cost is planning time: every join path over an input with an extra value is kept alongside the one it would otherwise have replaced. The extras-aware prechecks reject most of the others before they are built. Still, a value is emitted only if it costs more than 10 times cpu_operator_cost per evaluation, the same test make_sort_input_target() uses to decide an expression is worth postponing past a sort. That applies to an ordering IndexScan's ORDER BY values too, so a plain point distance is computed again above a join. Queries without such an expression plan as before. A value from the nullable side of an outer join is never used above it: the query's Vars carry varnullingrels the index's don't, so they don't match. Partial paths emit only parallel-safe values. An ordered or parallel Append still computes the expression above it. Author: Greg Burd Discussion: https://postgr.es/m/8s5lT8erXzBMugXJQ6Wginbp_gc2B4hmCYhF0Q0GpnuK87eCAOcKs5K31AWCwraCNFIQt42wwkow_MPPubHg485MWv-zDBY4NL_JX9yJEEg=@burd.me Discussion: https://postgr.es/m/356d5811-d8f0-4c29-a601-b61334337b11@iki.fi --- src/backend/optimizer/path/allpaths.c | 118 ++++ src/backend/optimizer/path/costsize.c | 47 +- src/backend/optimizer/path/joinpath.c | 159 ++++- src/backend/optimizer/plan/createplan.c | 77 ++- src/backend/optimizer/plan/planner.c | 11 +- src/backend/optimizer/util/clauses.c | 21 + src/backend/optimizer/util/pathnode.c | 738 ++++++++++++++++++------ src/include/nodes/pathnodes.h | 5 + src/include/optimizer/clauses.h | 1 + src/include/optimizer/pathnode.h | 17 +- src/test/regress/expected/gist.out | 252 +++++++- src/test/regress/sql/gist.sql | 120 +++- src/tools/pgindent/typedefs.list | 1 + 13 files changed, 1322 insertions(+), 245 deletions(-) diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 72ad61e8c2493..047d197c68ce9 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -103,6 +103,8 @@ static void set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, static void set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte); static void create_plain_partial_paths(PlannerInfo *root, RelOptInfo *rel); +static void add_extras_append_path(PlannerInfo *root, RelOptInfo *rel, + List *live_childrels); static void set_rel_consider_parallel(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte); static void set_plain_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, @@ -1409,6 +1411,114 @@ set_grouped_rel_pathlist(PlannerInfo *root, RelOptInfo *rel) } +/* + * add_extras_append_path + * Build an unordered, unparameterized Append over child paths that all + * emit the same extra values, if there are any worth emitting. + * + * A partitioned table's index on an expression is an index on that + * expression in every partition, so each child may have an index-only scan + * that reads it (see indexonly_path_target()). The Append can pass such a + * value up, as a join does, but only if every child emits it: they must all + * return the Append's targetlist. The candidates are the parent's + * partitioned indexes' expressions that the query's targetlist uses; for + * each, translated to the child, we take the child's cheapest path that + * emits it. create_append_child_plan() puts each child's values in the + * Append's order. + * + * Only an unordered, non-partial Append is built this way; an ordered or + * parallel Append still computes the expression above it. + */ +static void +add_extras_append_path(PlannerInfo *root, RelOptInfo *rel, + List *live_childrels) +{ + List *extras = NIL; + List *tlist_exprs; + AppendPathInput input = {0}; + PathTarget *target; + AppendPath *apath; + ListCell *lc; + + if (rel->reloptkind != RELOPT_BASEREL || live_childrels == NIL || + root->processed_tlist == NIL) + return; + + tlist_exprs = query_tlist_exprs(root, rel); + foreach(lc, rel->indexlist) + { + IndexOptInfo *index = (IndexOptInfo *) lfirst(lc); + ListCell *le = list_head(index->indexprs); + + for (int i = 0; i < index->ncolumns; i++) + { + Node *expr; + ListCell *lt; + + if (index->indexkeys[i] != 0) + continue; + expr = (Node *) lfirst(le); + le = lnext(index->indexprs, le); + + /* + * Only a column the children's index-only scans can return (see + * indexonly_path_target()), and not one the rel's target already + * has: apply_scanjoin_target_to_paths() rebuilds the Appends once + * the target is the final one, and its children then emit nothing + * extra. + */ + if (!index->canreturn[i] || + list_member(rel->reltarget->exprs, expr) || + !expr_worth_emitting(root, expr)) + continue; + foreach(lt, tlist_exprs) + { + if (expr_contains(lfirst(lt), expr)) + { + extras = list_append_unique(extras, expr); + break; + } + } + } + } + if (extras == NIL) + return; + + foreach(lc, live_childrels) + { + RelOptInfo *childrel = (RelOptInfo *) lfirst(lc); + List *child_extras; + Path *best = NULL; + ListCell *lp; + + child_extras = (List *) + adjust_appendrel_attrs_multilevel(root, (Node *) extras, + childrel, childrel->top_parent); + foreach(lp, childrel->pathlist) + { + Path *path = (Path *) lfirst(lp); + + if (path->param_info == NULL && + path_covers_extras(path, child_extras)) + { + best = path; /* pathlist is sorted by total cost */ + break; + } + } + if (best == NULL) + return; + accumulate_append_subpath(best, &input.subpaths, NULL, + &input.child_append_relid_sets); + } + + apath = create_append_path(root, rel, input, NIL, NULL, 0, false, -1); + target = copy_pathtarget(rel->reltarget); + foreach(lc, extras) + add_new_column_to_pathtarget(target, (Expr *) lfirst(lc)); + apath->path.pathtarget = set_pathtarget_cost_width(root, target); + add_path(rel, (Path *) apath); +} + /* * add_paths_to_append_rel * Generate paths for the given append relation given the set of non-dummy @@ -1633,6 +1743,14 @@ add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, NIL, NULL, 0, false, -1)); + /* + * Also consider an Append that emits extra values (see + * path_extra_exprs()) its children read from their indexes, so a join + * above it doesn't compute them. + */ + if (unparameterized_valid) + add_extras_append_path(root, rel, live_childrels); + /* build an AppendPath for the cheap startup paths, if valid */ if (startup_valid) add_path(rel, (Path *) create_append_path(root, rel, startup, diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index a844701b1d11c..4bd912d0811d6 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -795,6 +795,13 @@ cost_index(IndexPath *path, PlannerInfo *root, double loop_count, */ cost_qual_eval(&qpqual_cost, qpquals, root); + /* + * An index-only scan reads returnable index expressions in its quals from + * the index, as it does in its targetlist; see path_target_cost(). + */ + if (indexonly) + qpqual_cost = indexonly_qual_cost(root, path, qpquals, qpqual_cost); + startup_cost += qpqual_cost.startup; cpu_per_tuple = cpu_tuple_cost + qpqual_cost.per_tuple; @@ -802,8 +809,8 @@ cost_index(IndexPath *path, PlannerInfo *root, double loop_count, /* * tlist eval costs are paid per output row, not per tuple scanned. An - * ordering IndexScan doesn't pay for target entries it takes from its - * ORDER BY values. + * index path doesn't pay for target entries it reads from the index; see + * path_target_cost(). */ tlist_cost = path_target_cost(root, (Path *) path, path->path.pathtarget); startup_cost += tlist_cost.startup; @@ -3652,9 +3659,15 @@ final_cost_nestloop(PlannerInfo *root, NestPath *path, cpu_per_tuple = cpu_tuple_cost + restrict_qual_cost.per_tuple; run_cost += cpu_per_tuple * ntuples; - /* tlist eval costs are paid per output row, not per tuple scanned */ - startup_cost += path->jpath.path.pathtarget->cost.startup; - run_cost += path->jpath.path.pathtarget->cost.per_tuple * path->jpath.path.rows; + /* + * tlist eval costs are paid per output row, not per tuple scanned. The + * join may also emit extra values its inputs emit (see + * join_path_target()), but it doesn't evaluate those, so charge only for + * the joinrel's reltarget. + */ + startup_cost += path->jpath.path.parent->reltarget->cost.startup; + run_cost += path->jpath.path.parent->reltarget->cost.per_tuple * + path->jpath.path.rows; path->jpath.path.startup_cost = startup_cost; path->jpath.path.total_cost = startup_cost + run_cost; @@ -4242,9 +4255,15 @@ final_cost_mergejoin(PlannerInfo *root, MergePath *path, cpu_per_tuple = cpu_tuple_cost + qp_qual_cost.per_tuple; run_cost += cpu_per_tuple * mergejointuples; - /* tlist eval costs are paid per output row, not per tuple scanned */ - startup_cost += path->jpath.path.pathtarget->cost.startup; - run_cost += path->jpath.path.pathtarget->cost.per_tuple * path->jpath.path.rows; + /* + * tlist eval costs are paid per output row, not per tuple scanned. The + * join may also emit extra values its inputs emit (see + * join_path_target()), but it doesn't evaluate those, so charge only for + * the joinrel's reltarget. + */ + startup_cost += path->jpath.path.parent->reltarget->cost.startup; + run_cost += path->jpath.path.parent->reltarget->cost.per_tuple * + path->jpath.path.rows; path->jpath.path.startup_cost = startup_cost; path->jpath.path.total_cost = startup_cost + run_cost; @@ -4761,9 +4780,15 @@ final_cost_hashjoin(PlannerInfo *root, HashPath *path, run_cost += cpu_per_tuple * hashjointuples; } - /* tlist eval costs are paid per output row, not per tuple scanned */ - startup_cost += path->jpath.path.pathtarget->cost.startup; - run_cost += path->jpath.path.pathtarget->cost.per_tuple * path->jpath.path.rows; + /* + * tlist eval costs are paid per output row, not per tuple scanned. The + * join may also emit extra values its inputs emit (see + * join_path_target()), but it doesn't evaluate those, so charge only for + * the joinrel's reltarget. + */ + startup_cost += path->jpath.path.parent->reltarget->cost.startup; + run_cost += path->jpath.path.parent->reltarget->cost.per_tuple * + path->jpath.path.rows; path->jpath.path.startup_cost = startup_cost; path->jpath.path.total_cost = startup_cost + run_cost; diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 9d9e6fad6a1de..178548d617d3a 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -45,6 +45,7 @@ join_path_setup_hook_type join_path_setup_hook = NULL; #define PATH_PARAM_BY_REL(path, rel) \ (PATH_PARAM_BY_REL_SELF(path, rel) || PATH_PARAM_BY_PARENT(path, rel)) +static Path *cheapest_extras_path(RelOptInfo *rel); static void try_partial_mergejoin_path(PlannerInfo *root, RelOptInfo *joinrel, Path *outer_path, @@ -152,6 +153,8 @@ add_paths_to_joinrel(PlannerInfo *root, extra.restrictlist = restrictlist; extra.mergeclause_list = NIL; extra.sjinfo = sjinfo; + extra.extras_outer = cheapest_extras_path(outerrel); + extra.extras_inner = cheapest_extras_path(innerrel); extra.param_source_rels = NULL; extra.pgs_mask = joinrel->pgs_mask; @@ -998,6 +1001,54 @@ get_memoize_path(PlannerInfo *root, RelOptInfo *innerrel, return NULL; } +/* + * join_input_extras + * The values beyond their reltargets that the input paths emit, which a + * join over them emits too (see join_path_target()), or NIL. + * + * add_path() counts them in the join path's favor like better pathkeys, so + * the prechecks are told about them. + */ +static List * +join_input_extras(Path *outer_path, Path *inner_path) +{ + List *extras = NIL; + + /* cheap test first: most paths share their rel's reltarget */ + if (outer_path->pathtarget != outer_path->parent->reltarget) + extras = path_extra_exprs(outer_path); + if (inner_path->pathtarget != inner_path->parent->reltarget) + extras = list_concat(extras, path_extra_exprs(inner_path)); + return extras; +} + +/* + * cheapest_extras_path + * The cheapest unparameterized path of 'rel' that emits values beyond its + * reltarget (see path_extra_exprs()), or NULL. + * + * Such a path is kept by add_path() but is rarely the rel's cheapest, and + * hash and sort-merge joins look only at the cheapest. It is worth trying + * as well: a join over it doesn't pay for evaluating those values. + * pathlist is sorted by total cost, so the first one found is the cheapest. + */ +static Path * +cheapest_extras_path(RelOptInfo *rel) +{ + ListCell *lc; + + foreach(lc, rel->pathlist) + { + Path *path = (Path *) lfirst(lc); + + if (path->param_info == NULL && path != rel->cheapest_total_path && + path->pathtarget != rel->reltarget && + path_emits_extras(path)) + return path; + } + return NULL; +} + /* * try_nestloop_path * Consider a nestloop join path; if it appears useful, push it into @@ -1096,9 +1147,10 @@ try_nestloop_path(PlannerInfo *root, nestloop_subtype | PGS_CONSIDER_NONPARTIAL, outer_path, inner_path, extra); - if (add_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, workspace.total_cost, - pathkeys, required_outer)) + if (add_path_precheck_extras(joinrel, workspace.disabled_nodes, + workspace.startup_cost, workspace.total_cost, + pathkeys, required_outer, + join_input_extras(outer_path, inner_path))) { add_path(joinrel, (Path *) create_nestloop_path(root, @@ -1181,9 +1233,11 @@ try_partial_nestloop_path(PlannerInfo *root, */ initial_cost_nestloop(root, &workspace, jointype, nestloop_subtype, outer_path, inner_path, extra); - if (!add_partial_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, - workspace.total_cost, pathkeys)) + if (!add_partial_path_precheck_extras(joinrel, workspace.disabled_nodes, + workspace.startup_cost, + workspace.total_cost, pathkeys, + join_input_extras(outer_path, + inner_path))) return; /* Might be good enough to be worth trying, so let's try it. */ @@ -1289,9 +1343,10 @@ try_mergejoin_path(PlannerInfo *root, outer_presorted_keys, extra); - if (add_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, workspace.total_cost, - pathkeys, required_outer)) + if (add_path_precheck_extras(joinrel, workspace.disabled_nodes, + workspace.startup_cost, workspace.total_cost, + pathkeys, required_outer, + join_input_extras(outer_path, inner_path))) { add_path(joinrel, (Path *) create_mergejoin_path(root, @@ -1371,9 +1426,11 @@ try_partial_mergejoin_path(PlannerInfo *root, outer_presorted_keys, extra); - if (!add_partial_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, - workspace.total_cost, pathkeys)) + if (!add_partial_path_precheck_extras(joinrel, workspace.disabled_nodes, + workspace.startup_cost, + workspace.total_cost, pathkeys, + join_input_extras(outer_path, + inner_path))) return; /* Might be good enough to be worth trying, so let's try it. */ @@ -1443,9 +1500,10 @@ try_hashjoin_path(PlannerInfo *root, initial_cost_hashjoin(root, &workspace, jointype, hashclauses, outer_path, inner_path, extra, false); - if (add_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, workspace.total_cost, - NIL, required_outer)) + if (add_path_precheck_extras(joinrel, workspace.disabled_nodes, + workspace.startup_cost, workspace.total_cost, + NIL, required_outer, + join_input_extras(outer_path, inner_path))) { add_path(joinrel, (Path *) create_hashjoin_path(root, @@ -1504,9 +1562,11 @@ try_partial_hashjoin_path(PlannerInfo *root, */ initial_cost_hashjoin(root, &workspace, jointype, hashclauses, outer_path, inner_path, extra, parallel_hash); - if (!add_partial_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, - workspace.total_cost, NIL)) + if (!add_partial_path_precheck_extras(joinrel, workspace.disabled_nodes, + workspace.startup_cost, + workspace.total_cost, NIL, + join_input_extras(outer_path, + inner_path))) return; /* Might be good enough to be worth trying, so let's try it. */ @@ -1545,6 +1605,8 @@ sort_inner_and_outer(PlannerInfo *root, { Path *outer_path; Path *inner_path; + Path *extras_outer; + Path *extras_inner; Path *cheapest_partial_outer = NULL; Path *cheapest_safe_inner = NULL; List *all_pathkeys; @@ -1579,6 +1641,8 @@ sort_inner_and_outer(PlannerInfo *root, if (PATH_PARAM_BY_REL(outer_path, innerrel) || PATH_PARAM_BY_REL(inner_path, outerrel)) return; + extras_outer = extra->extras_outer; + extras_inner = extra->extras_inner; /* * If the joinrel is parallel-safe, we may be able to consider a partial @@ -1687,6 +1751,16 @@ sort_inner_and_outer(PlannerInfo *root, extra, false); + /* Likewise with inputs that emit extra values; see hash join */ + if (extras_outer != NULL) + try_mergejoin_path(root, joinrel, extras_outer, inner_path, + merge_pathkeys, cur_mergeclauses, + outerkeys, innerkeys, jointype, extra, false); + if (extras_inner != NULL) + try_mergejoin_path(root, joinrel, outer_path, extras_inner, + merge_pathkeys, cur_mergeclauses, + outerkeys, innerkeys, jointype, extra, false); + /* * If we have partial outer and parallel safe inner path then try * partial mergejoin path. @@ -1971,6 +2045,7 @@ match_unsorted_outer(PlannerInfo *root, bool useallclauses; Path *inner_cheapest_total = innerrel->cheapest_total_path; Path *matpath = NULL; + Path *extras_matpath = NULL; ListCell *lc1; /* @@ -2043,10 +2118,21 @@ match_unsorted_outer(PlannerInfo *root, if ((extra->pgs_mask & (PGS_NESTLOOP_MATERIALIZE | PGS_CONSIDER_NONPARTIAL)) == (PGS_NESTLOOP_MATERIALIZE | PGS_CONSIDER_NONPARTIAL) && - inner_cheapest_total != NULL && - !ExecMaterializesOutput(inner_cheapest_total->pathtype)) - matpath = (Path *) - create_material_path(innerrel, inner_cheapest_total, true); + inner_cheapest_total != NULL) + { + Path *extras_inner = extra->extras_inner; + + if (!ExecMaterializesOutput(inner_cheapest_total->pathtype)) + matpath = (Path *) + create_material_path(innerrel, inner_cheapest_total, true); + + /* likewise the cheapest inner path that emits extra values */ + if (extras_inner != NULL && + !PATH_PARAM_BY_REL(extras_inner, outerrel) && + !ExecMaterializesOutput(extras_inner->pathtype)) + extras_matpath = (Path *) + create_material_path(innerrel, extras_inner, true); + } } foreach(lc1, outerrel->pathlist) @@ -2120,6 +2206,17 @@ match_unsorted_outer(PlannerInfo *root, jointype, PGS_NESTLOOP_MATERIALIZE, extra); + + /* And of an inner path that emits extra values; see hash join */ + if (extras_matpath != NULL) + try_nestloop_path(root, + joinrel, + outerpath, + extras_matpath, + merge_pathkeys, + jointype, + PGS_NESTLOOP_MATERIALIZE, + extra); } /* Can't do anything else if inner rel is parameterized by outer */ @@ -2409,6 +2506,24 @@ hash_inner_and_outer(PlannerInfo *root, jointype, extra); + /* + * Also try an input that emits values the query needs above the join, + * on either side; see cheapest_extras_path(). + */ + { + Path *extras_outer = extra->extras_outer; + Path *extras_inner = extra->extras_inner; + + if (extras_outer != NULL) + try_hashjoin_path(root, joinrel, + extras_outer, cheapest_total_inner, + hashclauses, jointype, extra); + if (extras_inner != NULL) + try_hashjoin_path(root, joinrel, + cheapest_total_outer, extras_inner, + hashclauses, jointype, extra); + } + foreach(lc1, outerrel->cheapest_parameterized_paths) { Path *outerpath = (Path *) lfirst(lc1); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 283043acfa0ce..22a976058b060 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -24,6 +24,7 @@ #include "nodes/extensible.h" #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" +#include "optimizer/appendinfo.h" #include "optimizer/clauses.h" #include "optimizer/cost.h" #include "optimizer/optimizer.h" @@ -83,6 +84,8 @@ static Plan *create_gating_plan(PlannerInfo *root, Path *path, Plan *plan, List *gating_quals); static Plan *create_join_plan(PlannerInfo *root, JoinPath *best_path); static bool mark_async_capable_plan(Plan *plan, Path *path); +static Plan *create_append_child_plan(PlannerInfo *root, Path *apath, + Path *subpath); static Plan *create_append_plan(PlannerInfo *root, AppendPath *best_path, int flags); static Plan *create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path, @@ -939,13 +942,25 @@ use_physical_tlist(PlannerInfo *root, Path *path, int flags) } /* - * Nor if a plain index scan path was asked to emit its ORDER BY values - * (see index_path_orderby_target()); a physical tlist has only Vars, so - * whatever needs them would compute them again. + * Nor if the path emits extra values (see path_extra_exprs()) that a + * physical tlist lacks, such as an index scan's ORDER BY values. An + * index-only scan's are its own expression columns (see + * indexonly_path_target()), which its physical tlist, the index's + * indextlist, already holds. */ - if (path->pathtype == T_IndexScan && - index_path_emits_orderby((IndexPath *) path)) - return false; + if (path_emits_extras(path)) + { + List *indextlist; + + if (path->pathtype != T_IndexOnlyScan) + return false; + indextlist = ((IndexPath *) path)->indexinfo->indextlist; + foreach(lc, path_extra_exprs(path)) + { + if (!tlist_member((Expr *) lfirst(lc), indextlist)) + return false; + } + } /* * For an index-only scan, the "physical tlist" is the index's indextlist. @@ -1217,6 +1232,52 @@ mark_async_capable_plan(Plan *plan, Path *path) return true; } +/* + * create_append_child_plan + * Build the plan for a child of an Append or MergeAppend. + * + * Every child must return the Append's tlist: its rel's reltarget, plus any + * extra values the Append emits (see add_extras_append_path()), translated + * to the child. A child path may emit extra values of its own beyond its + * reltarget (see path_extra_exprs()), in some order; keep those the Append + * emits, in the Append's order, and drop the rest. + */ +static Plan * +create_append_child_plan(PlannerInfo *root, Path *apath, Path *subpath) +{ + Plan *subplan = create_plan_recurse(root, subpath, CP_EXACT_TLIST); + List *append_extras = path_extra_exprs(apath); + List *child_extras; + List *tlist; + int nrel; + ListCell *lc; + + if (!path_emits_extras(subpath)) + { + Assert(append_extras == NIL); + return subplan; + } + + nrel = list_length(subpath->parent->reltarget->exprs); + tlist = list_copy_head(subplan->targetlist, nrel); + child_extras = (List *) + adjust_appendrel_attrs_multilevel(root, (Node *) append_extras, + subpath->parent, + apath->parent); + foreach(lc, child_extras) + { + TargetEntry *tle = tlist_member((Expr *) lfirst(lc), + subplan->targetlist); + + if (tle == NULL) + elog(ERROR, "child of Append doesn't emit one of its extra values"); + tlist = lappend(tlist, makeTargetEntry(tle->expr, + list_length(tlist) + 1, + NULL, false)); + } + return change_plan_targetlist(subplan, tlist, subplan->parallel_safe); +} + /* * create_append_plan * Create an Append plan for 'best_path' and (recursively) plans @@ -1318,7 +1379,7 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path, int flags) Plan *subplan; /* Must insist that all children return the same tlist */ - subplan = create_plan_recurse(root, subpath, CP_EXACT_TLIST); + subplan = create_append_child_plan(root, (Path *) best_path, subpath); /* * For ordered Appends, we must insert a Sort node if subplan isn't @@ -1534,7 +1595,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path, /* Build the child plan */ /* Must insist that all children return the same tlist */ - subplan = create_plan_recurse(root, subpath, CP_EXACT_TLIST); + subplan = create_append_child_plan(root, (Path *) best_path, subpath); /* Compute sort column info, and adjust subplan's tlist as needed */ subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index f7fe5a66f2019..1a610a923457b 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -8263,10 +8263,9 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, * path with a projection path that generates the SRF-free scan/join * target. We modify the lists in place. * - * An index path may emit its own target, rel->reltarget plus a value it - * reads from the index (see indexonly_path_target() and - * index_path_orderby_target()). Such a path needs a projection even when - * the exprs are the same, since its target has more columns than + * A path may emit extra values beyond rel->reltarget (see + * path_extra_exprs()). Such a path needs a projection even when the + * exprs are the same, since its target has more columns than * scanjoin_target's sortgrouprefs describe. * * A projection usually adds the same cost to every path, but not always: @@ -8281,7 +8280,7 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, /* Shouldn't have any parameterized paths anymore */ Assert(subpath->param_info == NULL); - if (tlist_same_exprs && subpath->pathtarget == rel->reltarget) + if (tlist_same_exprs && !path_emits_extras(subpath)) subpath->pathtarget->sortgrouprefs = scanjoin_target->sortgrouprefs; else @@ -8306,7 +8305,7 @@ apply_scanjoin_target_to_paths(PlannerInfo *root, /* Shouldn't have any parameterized paths anymore */ Assert(subpath->param_info == NULL); - if (tlist_same_exprs && subpath->pathtarget == rel->reltarget) + if (tlist_same_exprs && !path_emits_extras(subpath)) subpath->pathtarget->sortgrouprefs = scanjoin_target->sortgrouprefs; else diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c index 9cfba66b5a04f..af8a3166a3199 100644 --- a/src/backend/optimizer/util/clauses.c +++ b/src/backend/optimizer/util/clauses.c @@ -117,6 +117,7 @@ typedef struct static bool contain_agg_clause_walker(Node *node, void *context); static bool find_window_functions_walker(Node *node, WindowFuncLists *lists); static bool contain_subplans_walker(Node *node, void *context); +static bool expr_contains_walker(Node *node, void *context); static bool contain_mutable_functions_walker(Node *node, void *context); static bool xmlexpr_is_immutable(XmlExpr *xexpr); static bool contain_volatile_functions_walker(Node *node, void *context); @@ -377,6 +378,26 @@ contain_subplans_walker(Node *node, void *context) } +/* + * expr_contains + * Is 'expr', or some subexpression of it, equal() to 'sub'? + */ +bool +expr_contains(Node *expr, Node *sub) +{ + return expr_contains_walker(expr, sub); +} + +static bool +expr_contains_walker(Node *node, void *context) +{ + if (node == NULL) + return false; + if (equal(node, context)) + return true; + return expression_tree_walker(node, expr_contains_walker, context); +} + /***************************************************************************** * Check clauses for mutable functions *****************************************************************************/ diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index eb9dac861ad62..d0ff9c668cfe6 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -387,24 +387,193 @@ set_cheapest(RelOptInfo *parent_rel) } /* - * pathtargets_differ - * Do two paths of base relation 'rel' emit different targets? + * path_extras_start + * Where the values a path emits beyond its rel's reltarget start in its + * pathtarget's exprs, or NULL if it emits none; *nextras is their number. * - * A base relation's paths normally all emit rel->reltarget, but an index - * path may also emit a value the query needs above the scan, one it has - * for free and other paths must compute (see indexonly_path_target() and - * index_path_orderby_target()). That cost isn't in either path's cost - * yet; it is charged when the expression is. So add_path() treats paths - * with different targets like paths with different pathkeys, and keeps - * both. + * A path normally emits rel->reltarget, the Vars and PlaceHolderVars the + * rest of the query needs. An index path may also emit a non-Var + * expression the query needs that it has for free: an index-only scan reads + * a returnable index expression column (see indexonly_path_target()), and an + * ordering IndexScan has its ORDER BY values (see + * index_path_orderby_target()). A join emits whatever its inputs emit (see + * join_path_target()), and so may an Append whose children all emit the + * same values (see add_extras_append_path()). Such a target is a copy of + * rel->reltarget with the extra expressions appended, so they're its tail. + * + * Sort, Material and Memoize don't project and share their input's target, + * so they emit what it emits. Nothing else emits extra values: other path + * types build their own targets, and an upper rel's targets differ from its + * reltarget for reasons of their own. + */ +static ListCell * +path_extras_start(Path *path, int *nextras) +{ + RelOptInfo *rel = path->parent; + PathTarget *reltarget = rel->reltarget; + int nrel; + + *nextras = 0; + if (path->pathtarget == reltarget) + return NULL; + + switch (nodeTag(path)) + { + case T_SortPath: + case T_IncrementalSortPath: + case T_MaterialPath: + case T_MemoizePath: + { + Path *subpath; + + if (IsA(path, MaterialPath)) + subpath = ((MaterialPath *) path)->subpath; + else if (IsA(path, MemoizePath)) + subpath = ((MemoizePath *) path)->subpath; + else + subpath = ((SortPath *) path)->subpath; + if (subpath->parent != rel || + subpath->pathtarget != path->pathtarget) + return NULL; + return path_extras_start(subpath, nextras); + } + case T_IndexPath: + if (!IS_SIMPLE_REL(rel)) + return NULL; + break; + case T_NestPath: + case T_MergePath: + case T_HashPath: + if (!IS_JOIN_REL(rel)) + return NULL; + break; + case T_AppendPath: + /* see add_extras_append_path() */ + if (rel->reloptkind != RELOPT_BASEREL) + return NULL; + break; + default: + return NULL; + } + + /* + * Our targets are a copy of reltarget with the values appended. Anything + * else (say, a final target that apply_projection_to_path() put into an + * index path) isn't ours. + */ + nrel = list_length(reltarget->exprs); + if (list_length(path->pathtarget->exprs) <= nrel) + return NULL; + for (int i = 0; i < nrel; i++) + { + if (list_nth(path->pathtarget->exprs, i) != + list_nth(reltarget->exprs, i)) + return NULL; + } + *nextras = list_length(path->pathtarget->exprs) - nrel; + return list_nth_cell(path->pathtarget->exprs, nrel); +} + +/* + * path_emits_extras + * Does the path emit any values beyond its rel's reltarget? + */ +bool +path_emits_extras(Path *path) +{ + int n; + + return path_extras_start(path, &n) != NULL; +} + +/* + * path_extra_exprs + * The values a path emits beyond its rel's reltarget (see + * path_extras_start()), as a new list. + */ +List * +path_extra_exprs(Path *path) +{ + int n; + ListCell *start = path_extras_start(path, &n); + + if (start == NULL) + return NIL; + return list_copy_tail(path->pathtarget->exprs, + list_length(path->pathtarget->exprs) - n); +} + +/* + * extras_subset + * Is each of the 'n1' values from 'c1' among the 'n2' from 'c2'? + */ +static bool +extras_subset(List *l1, ListCell *c1, int n1, List *l2, ListCell *c2, int n2) +{ + int i1 = list_cell_number(l1, c1); + int i2 = list_cell_number(l2, c2); + + for (int a = 0; a < n1; a++) + { + Node *x = list_nth(l1, i1 + a); + bool found = false; + + for (int b = 0; b < n2 && !found; b++) + found = equal(x, list_nth(l2, i2 + b)); + if (!found) + return false; + } + return true; +} + +/* + * compare_path_extras + * Compare the extra values two paths emit (see path_extra_exprs()). + * + * A path that emits a superset of what the other emits is "better" in the + * same sense as better-sorted: whatever needs the other path's values can + * use this one's instead. The result uses the PathKeysComparison codes so + * add_path() can combine it with the pathkeys comparison. */ -static inline bool -pathtargets_differ(RelOptInfo *rel, Path *path1, Path *path2) +static PathKeysComparison +compare_path_extras(Path *path1, Path *path2) { - return rel->reloptkind == RELOPT_BASEREL && - (IsA(path1, IndexPath) || IsA(path2, IndexPath)) && - path1->pathtarget != path2->pathtarget && - !equal(path1->pathtarget->exprs, path2->pathtarget->exprs); + int n1, + n2; + ListCell *c1 = path_extras_start(path1, &n1); + ListCell *c2 = path_extras_start(path2, &n2); + List *l1 = path1->pathtarget->exprs; + List *l2 = path2->pathtarget->exprs; + bool sub12, + sub21; + + if (n1 == 0 && n2 == 0) + return PATHKEYS_EQUAL; + sub12 = n1 == 0 || (n2 > 0 && extras_subset(l1, c1, n1, l2, c2, n2)); + sub21 = n2 == 0 || (n1 > 0 && extras_subset(l2, c2, n2, l1, c1, n1)); + if (sub12 && sub21) + return PATHKEYS_EQUAL; + if (sub21) + return PATHKEYS_BETTER1; + if (sub12) + return PATHKEYS_BETTER2; + return PATHKEYS_DIFFERENT; +} + +/* + * combine_comparisons + * Combine two PathKeysComparison results into one. + * + * One path is better overall only if it's better or equal on both counts. + */ +static PathKeysComparison +combine_comparisons(PathKeysComparison a, PathKeysComparison b) +{ + if (a == PATHKEYS_EQUAL) + return b; + if (b == PATHKEYS_EQUAL || a == b) + return a; + return PATHKEYS_DIFFERENT; } /* @@ -415,7 +584,11 @@ pathtargets_differ(RelOptInfo *rel, Path *path1, Path *path2) * A path is worthy if it has a better sort order (better pathkeys) or * cheaper cost (as defined below), or generates fewer rows, than any * existing path that has the same or superset parameterization rels. We - * also consider parallel-safe paths more worthy than others. + * also consider parallel-safe paths more worthy than others. And a path + * that emits extra values the query needs (see path_extra_exprs()) is + * better, like a better-sorted one, than a path that emits a subset of + * them: whatever would compute those values above the other path can + * read them from this one. * * Cheaper cost can mean either a cheaper total cost or a cheaper startup * cost; if one path is cheaper in one of these aspects and another is @@ -534,8 +707,8 @@ add_path(RelOptInfo *parent_rel, Path *new_path) old_path_pathkeys = old_path->param_info ? NIL : old_path->pathkeys; keyscmp = compare_pathkeys(new_path_pathkeys, old_path_pathkeys); - if (pathtargets_differ(parent_rel, new_path, old_path)) - keyscmp = PATHKEYS_DIFFERENT; + keyscmp = combine_comparisons(keyscmp, + compare_path_extras(new_path, old_path)); if (keyscmp != PATHKEYS_DIFFERENT) { switch (costcmp) @@ -712,6 +885,44 @@ bool add_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, Cost startup_cost, Cost total_cost, List *pathkeys, Relids required_outer) +{ + return add_path_precheck_extras(parent_rel, disabled_nodes, startup_cost, + total_cost, pathkeys, required_outer, NIL); +} + +/* + * path_covers_extras + * Does 'path' emit each of 'extras' (see path_extra_exprs())? + */ +bool +path_covers_extras(Path *path, List *extras) +{ + int n; + ListCell *start; + + if (extras == NIL) + return true; + start = path_extras_start(path, &n); + if (start == NULL) + return false; + return extras_subset(extras, list_head(extras), list_length(extras), + path->pathtarget->exprs, start, n); +} + +/* + * add_path_precheck_extras + * add_path_precheck() for a path that will emit (at most) 'extras' beyond + * its rel's reltarget. + * + * add_path() counts such values in a path's favor like better pathkeys (see + * compare_path_extras()), so an old path dominates the new one only if it + * emits them all too. 'extras' may include values the new path won't end + * up emitting; that only makes the test stricter. + */ +bool +add_path_precheck_extras(RelOptInfo *parent_rel, int disabled_nodes, + Cost startup_cost, Cost total_cost, + List *pathkeys, Relids required_outer, List *extras) { List *new_path_pathkeys; bool consider_startup; @@ -764,7 +975,8 @@ add_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, keyscmp == PATHKEYS_BETTER2) { /* new path does not win on pathkeys... */ - if (bms_equal(required_outer, PATH_REQ_OUTER(old_path))) + if (bms_equal(required_outer, PATH_REQ_OUTER(old_path)) && + path_covers_extras(old_path, extras)) { /* Found an old path that dominates the new one */ return false; @@ -841,10 +1053,10 @@ add_partial_path(RelOptInfo *parent_rel, Path *new_path) bool remove_old = false; /* unless new proves superior */ PathKeysComparison keyscmp; - /* Compare pathkeys, and targets as in add_path(). */ + /* Compare pathkeys, and extra values as in add_path(). */ keyscmp = compare_pathkeys(new_path->pathkeys, old_path->pathkeys); - if (pathtargets_differ(parent_rel, new_path, old_path)) - keyscmp = PATHKEYS_DIFFERENT; + keyscmp = combine_comparisons(keyscmp, + compare_path_extras(new_path, old_path)); /* * Unless pathkeys are incompatible, see if one of the paths dominates @@ -939,6 +1151,21 @@ add_partial_path(RelOptInfo *parent_rel, Path *new_path) bool add_partial_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, Cost startup_cost, Cost total_cost, List *pathkeys) +{ + return add_partial_path_precheck_extras(parent_rel, disabled_nodes, + startup_cost, total_cost, + pathkeys, NIL); +} + +/* + * add_partial_path_precheck_extras + * add_partial_path_precheck() for a path that will emit (at most) + * 'extras'; see add_path_precheck_extras(). + */ +bool +add_partial_path_precheck_extras(RelOptInfo *parent_rel, int disabled_nodes, + Cost startup_cost, Cost total_cost, + List *pathkeys, List *extras) { bool consider_startup = parent_rel->consider_startup; ListCell *p1; @@ -1016,7 +1243,8 @@ add_partial_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, * If the existing path is cheaper and the pathkeys are equal or * worse, the new path is not interesting. */ - if (costcmp == COSTS_BETTER2 && keyscmp != PATHKEYS_BETTER1) + if (costcmp == COSTS_BETTER2 && keyscmp != PATHKEYS_BETTER1 && + path_covers_extras(old_path, extras)) return false; /* @@ -1033,8 +1261,8 @@ add_partial_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, * non-parallel plans. If it loses even before accounting for the cost of * the Gather node, we should definitely reject it. */ - if (!add_path_precheck(parent_rel, disabled_nodes, startup_cost, - total_cost, pathkeys, NULL)) + if (!add_path_precheck_extras(parent_rel, disabled_nodes, startup_cost, + total_cost, pathkeys, NULL, extras)) return false; return true; @@ -1113,13 +1341,14 @@ create_samplescan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer * give the path a copy of rel->reltarget with those expressions appended. * They cost the scan nothing (path_target_cost() exempts them), and * setrefs.c's fix_join_expr() matches them in the parent join's expressions, - * replacing its copy with a reference to the scan's output. + * replacing its copy with a reference to the scan's output. A join passes + * them on up (see join_path_target()). * - * Restricted to plain base relations: an appendrel child's paths are wrapped - * by Append/MergeAppend, which build their own targetlists, and the top-level - * scan/join target already handles the unjoined case. Returns - * rel->reltarget unchanged when there's nothing to add, which is the common - * case, so most paths share it as before. + * Restricted to plain base relations: an appendrel child's ORDER BY values + * reach the scan/join target through the child's own projection (see + * apply_scanjoin_target_to_paths()). Returns rel->reltarget unchanged when + * there's nothing to add, which is the common case, so most paths share it + * as before. */ static PathTarget * index_path_orderby_target(PlannerInfo *root, IndexOptInfo *index, @@ -1143,6 +1372,14 @@ index_path_orderby_target(PlannerInfo *root, IndexOptInfo *index, if (!index_orderby_returnable(index, lfirst_int(lcol), orderby)) continue; + /* + * Not if it's cheap: a join would drop it (see join_path_target()), + * and emitting it would still keep extra paths at this level (see + * expr_worth_emitting()). + */ + if (!expr_worth_emitting(root, (Node *) orderby)) + continue; + /* * Only if some top-level targetlist entry is this expression; an * expression that merely contains it would not be rewritten, and @@ -1173,40 +1410,88 @@ index_path_orderby_target(PlannerInfo *root, IndexOptInfo *index, return set_pathtarget_cost_width(root, target); } +/* + * expr_worth_emitting + * Is 'expr' expensive enough to be worth emitting as an extra value? + * + * A path that emits an extra value makes add_path() keep it, and every join + * path built on it, alongside paths it would otherwise have discarded (see + * compare_path_extras()). That costs planning time at every join level + * above. For a cheap expression, computing it again costs less than that, + * so emit only values that cost more than 10 times cpu_operator_cost per + * evaluation, the same test make_sort_input_target() uses to decide that an + * expression is worth postponing past a sort. + */ +bool +expr_worth_emitting(PlannerInfo *root, Node *expr) +{ + QualCost cost; + + cost_qual_eval_node(&cost, expr, root); + return cost.per_tuple > 10 * cpu_operator_cost; +} + +/* + * query_tlist_exprs + * The query's final targetlist expressions, in terms of 'rel'. + * + * root->processed_tlist is in terms of the parent of an appendrel; translate + * it for a child, so it can be compared with the child's index expressions. + */ +List * +query_tlist_exprs(PlannerInfo *root, RelOptInfo *rel) +{ + List *exprs = get_tlist_exprs(root->processed_tlist, true); + + if (rel->reloptkind == RELOPT_OTHER_MEMBER_REL) + exprs = (List *) adjust_appendrel_attrs_multilevel(root, (Node *) exprs, + rel, + rel->top_parent); + return exprs; +} + /* * indexonly_path_target * Choose the PathTarget for an index-only scan path. * * An index-only scan reads a returnable index expression column instead of * computing it (set_indexonlyscan_references()), and isn't charged for it - * (indexonly_target_cost()). But that only helps if the path survives + * (path_target_cost()). But that only helps if the path survives * add_path(), and with rel->reltarget, which has only Vars, nothing * distinguishes it from a seq scan that will have to compute the expression * later; it loses on cost before the expression is charged to anyone. So - * if the query's targetlist has a top-level entry equal() to a returnable - * index expression, give the path a copy of rel->reltarget with that - * expression appended. add_path() keeps paths with different targets - * (see pathtargets_differ()), and once the scan/join target is applied, the - * paths that must compute the expression pay for it and this one doesn't. + * if the query's targetlist contains a returnable index expression, give + * the path a copy of rel->reltarget with that expression appended, making + * it an extra value the path emits (see path_extra_exprs()). add_path() + * then won't let a path that lacks it dominate this one on cost alone. + * The value travels up through joins (see join_path_target()), and + * setrefs.c's fix_join_expr() and fix_upper_expr() replace each occurrence, + * at any depth, with a reference to it; the paths that must compute it + * instead pay for it when the expression is charged. + * + * An expression is matched with equal(), so a Var of a rel on the nullable + * side of an outer join, which carries varnullingrels in the targetlist, + * never matches the index's own: the value read below the join isn't the + * one the query asks for above it. * - * This is done only when the rel is the query's sole base relation, where - * the value goes straight to the scan/join target. Below a join, the join - * would have to credit it (as join_target_cost() does for ORDER BY values) - * and the joinrel's add_path() would have to keep it; not attempted yet. + * A partial path gets the extra values only if they're parallel-safe. */ static PathTarget * -indexonly_path_target(PlannerInfo *root, IndexOptInfo *index) +indexonly_path_target(PlannerInfo *root, IndexOptInfo *index, + bool partial_path) { RelOptInfo *rel = index->rel; PathTarget *target = NULL; + List *tlist_exprs; ListCell *lt; int i = 0; - if (index->indexprs == NIL || rel->reloptkind != RELOPT_BASEREL || - root->processed_tlist == NIL || - !bms_equal(rel->relids, root->all_query_rels)) + if (index->indexprs == NIL || !IS_SIMPLE_REL(rel) || + root->processed_tlist == NIL) return rel->reltarget; + tlist_exprs = query_tlist_exprs(root, rel); + foreach(lt, index->indextlist) { TargetEntry *itle = lfirst_node(TargetEntry, lt); @@ -1214,14 +1499,18 @@ indexonly_path_target(PlannerInfo *root, IndexOptInfo *index) if (!index->canreturn[i++] || IsA(itle->expr, Var)) continue; + if (partial_path && !is_parallel_safe(root, (Node *) itle->expr)) + continue; + if (!expr_worth_emitting(root, (Node *) itle->expr)) + continue; - foreach(lq, root->processed_tlist) + foreach(lq, tlist_exprs) { - if (equal(lfirst_node(TargetEntry, lq)->expr, itle->expr)) + if (expr_contains(lfirst(lq), (Node *) itle->expr)) { if (target == NULL) target = copy_pathtarget(rel->reltarget); - add_column_to_pathtarget(target, itle->expr, 0); + add_new_column_to_pathtarget(target, itle->expr); break; } } @@ -1230,46 +1519,10 @@ indexonly_path_target(PlannerInfo *root, IndexOptInfo *index) if (target == NULL) return rel->reltarget; - /* add_column_to_pathtarget doesn't maintain cost and width */ + /* add_new_column_to_pathtarget doesn't maintain cost and width */ return set_pathtarget_cost_width(root, target); } -/* - * index_path_emits_orderby - * Does this plain IndexScan path's target hold a value it takes from its - * ORDER BY values? - * - * use_physical_tlist() must not replace such a target with the scan's - * physical tlist, which has only Vars, or whatever needs the value would - * compute it again. - */ -bool -index_path_emits_orderby(IndexPath *ipath) -{ - ListCell *lc; - - if (ipath->path.pathtype != T_IndexScan || ipath->indexorderbys == NIL) - return false; - - foreach(lc, ipath->path.pathtarget->exprs) - { - Expr *expr = (Expr *) lfirst(lc); - ListCell *lo, - *lcol; - - if (IsA(expr, Var)) - continue; - forboth(lo, ipath->indexorderbys, lcol, ipath->indexorderbycols) - { - if (index_orderby_returnable(ipath->indexinfo, lfirst_int(lcol), - (Expr *) lfirst(lo)) && - orderby_tlist_match(expr, (Expr *) lfirst(lo))) - return true; - } - } - return false; -} - /* * create_index_path * Creates a path node for an index scan. @@ -1310,7 +1563,7 @@ create_index_path(PlannerInfo *root, pathnode->path.pathtype = indexonly ? T_IndexOnlyScan : T_IndexScan; pathnode->path.parent = rel; pathnode->path.pathtarget = indexonly ? - indexonly_path_target(root, index) : + indexonly_path_target(root, index, partial_path) : index_path_orderby_target(root, index, indexorderbys, indexorderbycols); pathnode->path.param_info = get_baserel_parampathinfo(root, rel, @@ -1927,7 +2180,8 @@ create_material_path(RelOptInfo *rel, Path *subpath, bool enabled) pathnode->path.pathtype = T_Material; pathnode->path.parent = rel; - pathnode->path.pathtarget = rel->reltarget; + /* Material doesn't project, so use source path's pathtarget */ + pathnode->path.pathtarget = subpath->pathtarget; pathnode->path.param_info = get_wrapper_parampathinfo(subpath); pathnode->path.parallel_aware = false; pathnode->path.parallel_safe = rel->consider_parallel && @@ -1963,7 +2217,8 @@ create_memoize_path(PlannerInfo *root, RelOptInfo *rel, Path *subpath, pathnode->path.pathtype = T_Memoize; pathnode->path.parent = rel; - pathnode->path.pathtarget = rel->reltarget; + /* Memoize doesn't project, so use source path's pathtarget */ + pathnode->path.pathtarget = subpath->pathtarget; pathnode->path.param_info = get_wrapper_parampathinfo(subpath); pathnode->path.parallel_aware = false; pathnode->path.parallel_safe = rel->consider_parallel && @@ -2545,6 +2800,86 @@ calc_non_nestloop_required_outer(Path *outer_path, Path *inner_path) return required_outer; } +/* + * join_path_target + * Choose the PathTarget for a join path. + * + * A join normally emits joinrel->reltarget. If an input emits extra values + * (see path_extra_exprs()), the join passes them up too, so that a higher + * join or the final scan/join target can use them instead of computing them + * (fix_join_expr() matches them in the inputs' targetlists, and + * join_free_exprs() credits them). Without that, an index-only scan or + * ordering IndexScan below the topmost join would read the value for + * nothing. + * + * Only values the query still needs above this join are passed up: a + * value that refers to a rel on the nullable side of an outer join this + * join forms would be wrong above it, but those never match the query's + * targetlist (its Vars carry varnullingrels the index's don't), so no input + * emits one; and every value we emit came from the final targetlist. + * + * Values must be parallel-safe if the join is, since a partial join path's + * target is computed in workers. + * + * Every value a scan emits passed expr_worth_emitting() there, so we don't + * look up its cost again for each join path. + */ +static PathTarget * +join_path_target(PlannerInfo *root, RelOptInfo *joinrel, + Path *outer_path, Path *inner_path) +{ + List *outer_extras = path_extra_exprs(outer_path); + List *inner_extras = path_extra_exprs(inner_path); + int nextras; + PathTarget *target; + bool all_kept = true; + ListCell *lc; + + if (outer_extras == NIL && inner_extras == NIL) + return joinrel->reltarget; + + target = copy_pathtarget(joinrel->reltarget); + nextras = list_length(outer_extras) + list_length(inner_extras); + foreach(lc, list_concat(outer_extras, inner_extras)) + { + Node *expr = (Node *) lfirst(lc); + + if (!joinrel->consider_parallel || is_parallel_safe(root, expr)) + add_new_column_to_pathtarget(target, (Expr *) expr); + else + all_kept = false; + } + if (list_length(target->exprs) == list_length(joinrel->reltarget->exprs)) + return joinrel->reltarget; + + /* + * If every value added a column, their cost and width are what they added + * to the inputs' targets. Working that out is much cheaper than + * set_pathtarget_cost_width(), which looks up each function's cost again. + * (They always do today: an input's values are distinct, the two inputs' + * refer to different rels, and a join rel's reltarget holds only Vars and + * PlaceHolderVars. But check, rather than miscount if that changes.) + */ + if (all_kept && + list_length(target->exprs) == + list_length(joinrel->reltarget->exprs) + nextras) + { + Path *inputs[2] = {outer_path, inner_path}; + + for (int i = 0; i < 2; i++) + { + PathTarget *pt = inputs[i]->pathtarget; + PathTarget *rt = inputs[i]->parent->reltarget; + + target->cost.startup += pt->cost.startup - rt->cost.startup; + target->cost.per_tuple += pt->cost.per_tuple - rt->cost.per_tuple; + target->width += pt->width - rt->width; + } + return target; + } + return set_pathtarget_cost_width(root, target); +} + /* * create_nestloop_path * Creates a pathnode corresponding to a nestloop join between two @@ -2613,7 +2948,9 @@ create_nestloop_path(PlannerInfo *root, pathnode->jpath.path.pathtype = T_NestLoop; pathnode->jpath.path.parent = joinrel; - pathnode->jpath.path.pathtarget = joinrel->reltarget; + pathnode->jpath.path.pathtarget = join_path_target(root, joinrel, + outer_path, + inner_path); pathnode->jpath.path.param_info = get_joinrel_parampathinfo(root, joinrel, @@ -2679,7 +3016,9 @@ create_mergejoin_path(PlannerInfo *root, pathnode->jpath.path.pathtype = T_MergeJoin; pathnode->jpath.path.parent = joinrel; - pathnode->jpath.path.pathtarget = joinrel->reltarget; + pathnode->jpath.path.pathtarget = join_path_target(root, joinrel, + outer_path, + inner_path); pathnode->jpath.path.param_info = get_joinrel_parampathinfo(root, joinrel, @@ -2744,7 +3083,9 @@ create_hashjoin_path(PlannerInfo *root, pathnode->jpath.path.pathtype = T_HashJoin; pathnode->jpath.path.parent = joinrel; - pathnode->jpath.path.pathtarget = joinrel->reltarget; + pathnode->jpath.path.pathtarget = join_path_target(root, joinrel, + outer_path, + inner_path); pathnode->jpath.path.param_info = get_joinrel_parampathinfo(root, joinrel, @@ -2882,125 +3223,154 @@ orderby_tlist_match(Expr *expr, Expr *orderby) equal(lsecond(a->args), linitial(b->args)); } +/* state for credit_free_exprs_walker() */ +typedef struct +{ + PlannerInfo *root; + List *free; + QualCost cost; +} credit_free_context; + /* - * indexonly_target_cost - * path_target_cost() for an index-only scan path. + * credit_free_exprs_walker + * Subtract from context->cost the cost of each outermost subexpression + * of 'node' equal() to one of context->free. * - * setrefs.c's set_indexonlyscan_references() matches target expressions - * against the index's returnable columns, expression columns included, and - * replaces a match with a reference to that column: the scan reads the - * value the index stored instead of computing it. So a top-level target - * expression equal() to a returnable index expression costs nothing. Only - * top-level entries are considered, as elsewhere; that's what's normally in - * a scan's target. + * setrefs.c's fix_upper_expr() and fix_join_expr() try to match every + * non-Var subexpression against the input's targetlist before descending + * into it, and replace a match with a reference to the input's column. So + * an expression costs nothing where an input emits it, wherever it appears + * in the expression, and its own subexpressions aren't evaluated either. + */ +static bool +credit_free_exprs_walker(Node *node, credit_free_context *context) +{ + if (node == NULL || IsA(node, Var) || IsA(node, Const)) + return false; + if (list_member(context->free, node)) + { + QualCost ecost; + + cost_qual_eval_node(&ecost, node, context->root); + context->cost.startup -= ecost.startup; + context->cost.per_tuple -= ecost.per_tuple; + return false; /* don't count its subexpressions twice */ + } + return expression_tree_walker(node, credit_free_exprs_walker, context); +} + +/* + * free_exprs_cost + * target->cost, less the cost of the values in 'free' that 'target' + * uses (see credit_free_exprs_walker()). */ static QualCost -indexonly_target_cost(PlannerInfo *root, IndexPath *ipath, PathTarget *target) +free_exprs_cost(PlannerInfo *root, PathTarget *target, List *free) +{ + credit_free_context context; + + context.root = root; + context.free = free; + context.cost = target->cost; + if (free != NIL) + (void) credit_free_exprs_walker((Node *) target->exprs, &context); + return context.cost; +} + +/* + * indexonly_free_exprs + * The returnable expression columns of an index-only scan's index. + * + * set_indexonlyscan_references() replaces each occurrence of one of these + * in the scan's targetlist and quals with a reference to the index column. + */ +static List * +indexonly_free_exprs(IndexPath *ipath) { - QualCost cost = target->cost; IndexOptInfo *index = ipath->indexinfo; - ListCell *lc; + List *result = NIL; + ListCell *lt; + int i = 0; if (index->indexprs == NIL) - return cost; /* only Vars, which cost nothing anyway */ - - foreach(lc, target->exprs) + return NIL; + foreach(lt, index->indextlist) { - Node *expr = (Node *) lfirst(lc); - ListCell *lt; - int i = 0; - - if (IsA(expr, Var)) - continue; + TargetEntry *tle = lfirst_node(TargetEntry, lt); - foreach(lt, index->indextlist) - { - TargetEntry *tle = lfirst_node(TargetEntry, lt); - - if (index->canreturn[i++] && !IsA(tle->expr, Var) && - equal(expr, tle->expr)) - { - QualCost ecost; - - cost_qual_eval_node(&ecost, expr, root); - cost.startup -= ecost.startup; - cost.per_tuple -= ecost.per_tuple; - break; - } - } + if (index->canreturn[i++] && !IsA(tle->expr, Var)) + result = lappend(result, tle->expr); } - - return cost; + return result; } /* - * join_target_cost - * path_target_cost() for a join path. + * indexonly_qual_cost + * 'cost', the cost of an index-only scan's qpquals, less the returnable + * index expressions they use. * - * A join takes any top-level target expression that one of its inputs - * already emits from that input's output instead of evaluating it again: - * setrefs.c's fix_join_expr() matches whole non-Var expressions against the - * input targetlists before descending into them. Join inputs ordinarily - * emit only Vars and PlaceHolderVars, so this changes nothing; but an - * ordering IndexScan input may emit the ORDER BY values it returns (see - * index_path_orderby_target()). Look through Material and Memoize, which - * pass their input's targetlist up unchanged. We look only at the join's - * immediate inputs; the value can't reach a higher join without passing - * through this one's targetlist, which is the joinrel's and has only Vars. + * set_indexonlyscan_references() rewrites the scan's quals as it does its + * targetlist. */ -static QualCost -join_target_cost(PlannerInfo *root, JoinPath *jpath, PathTarget *target) +QualCost +indexonly_qual_cost(PlannerInfo *root, IndexPath *ipath, List *qpquals, + QualCost cost) { - QualCost cost = target->cost; - Path *inputs[2] = {jpath->outerjoinpath, jpath->innerjoinpath}; + credit_free_context context; ListCell *lc; - foreach(lc, target->exprs) + context.root = root; + context.free = indexonly_free_exprs(ipath); + context.cost = cost; + if (context.free == NIL) + return cost; + foreach(lc, qpquals) { - Node *expr = (Node *) lfirst(lc); - - if (IsA(expr, Var) || IsA(expr, PlaceHolderVar)) - continue; + Node *qual = (Node *) lfirst(lc); - for (int i = 0; i < 2; i++) - { - Path *input = inputs[i]; - - while (IsA(input, MaterialPath) || IsA(input, MemoizePath)) - input = IsA(input, MaterialPath) ? - ((MaterialPath *) input)->subpath : - ((MemoizePath *) input)->subpath; - - if (IsA(input, IndexPath) && - index_path_emits_orderby((IndexPath *) input) && - list_member(input->pathtarget->exprs, expr)) - { - QualCost ecost; - - cost_qual_eval_node(&ecost, expr, root); - cost.startup -= ecost.startup; - cost.per_tuple -= ecost.per_tuple; - break; - } - } + if (IsA(qual, RestrictInfo)) + qual = (Node *) ((RestrictInfo *) qual)->clause; + (void) credit_free_exprs_walker(qual, &context); } + return context.cost; +} - return cost; +/* + * join_free_exprs + * The values a join's inputs emit beyond their reltargets. + * + * fix_join_expr() replaces each occurrence of one of these in the join's + * targetlist with a reference to the input's output. Material and Memoize + * pass their input's targetlist up unchanged; path_extra_exprs() already + * looks through them. + */ +static List * +join_free_exprs(JoinPath *jpath) +{ + return list_concat_unique(path_extra_exprs(jpath->outerjoinpath), + path_extra_exprs(jpath->innerjoinpath)); } /* * path_target_cost * Return the cost 'path' pays to evaluate 'target'. * - * Normally that's just target->cost. A join path doesn't pay for entries - * an input already emits (see join_target_cost()), nor an index-only scan - * for entries it reads from the index (see indexonly_target_cost()). And a - * plain IndexScan takes any - * top-level target expression that is a returnable ORDER BY expression - * (see orderby_tlist_match()) from its ORDER BY values instead of - * evaluating it (setrefs.c rewrites it into a reference to them), so it - * doesn't pay for those. Each target entry - * is counted at most once, as setrefs.c rewrites it once. + * Normally that's just target->cost. But a path doesn't pay for a value + * it has for free: + * + * - a join, for values one of its inputs emits (join_free_exprs()); + * - an index-only scan, for returnable index expression columns + * (indexonly_free_exprs()); + * + * in either case wherever the value appears in the target, since setrefs.c + * replaces every occurrence (see credit_free_exprs_walker()). And + * + * - a plain IndexScan, for a top-level target entry that is one of its + * returnable ORDER BY expressions (see orderby_tlist_match()), which + * setrefs.c's replace_orderby_tlist_refs() takes from its ORDER BY values. + * Only top-level entries of the scan's own targetlist are rewritten there, + * so only those count here; above the scan, a join or upper node that + * reads the value is credited for it at any depth, as above. */ QualCost path_target_cost(PlannerInfo *root, Path *path, PathTarget *target) @@ -3010,10 +3380,12 @@ path_target_cost(PlannerInfo *root, Path *path, PathTarget *target) ListCell *lc; if (IsA(path, NestPath) || IsA(path, MergePath) || IsA(path, HashPath)) - return join_target_cost(root, (JoinPath *) path, target); + return free_exprs_cost(root, target, + join_free_exprs((JoinPath *) path)); if (IsA(path, IndexPath) && path->pathtype == T_IndexOnlyScan) - return indexonly_target_cost(root, (IndexPath *) path, target); + return free_exprs_cost(root, target, + indexonly_free_exprs((IndexPath *) path)); if (!IsA(path, IndexPath) || path->pathtype != T_IndexScan) return cost; diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index ca2a024b915c0..0877d24c762f7 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -3644,6 +3644,9 @@ typedef struct SemiAntiJoinFactors * RIGHT_ANTI/inner_unique joins) * param_source_rels are OK targets for parameterization of result paths * pgs_mask is a bitmask of PGS_* constants to limit the join strategy + * extras_outer and extras_inner are each input rel's cheapest unparameterized + * path that emits extra values, if that isn't its cheapest path anyway + * (see cheapest_extras_path() in joinpath.c), else NULL */ typedef struct JoinPathExtraData { @@ -3654,6 +3657,8 @@ typedef struct JoinPathExtraData SemiAntiJoinFactors semifactors; Relids param_source_rels; uint64 pgs_mask; + Path *extras_outer; + Path *extras_inner; } JoinPathExtraData; /* diff --git a/src/include/optimizer/clauses.h b/src/include/optimizer/clauses.h index e929789762be9..e0196edf109bb 100644 --- a/src/include/optimizer/clauses.h +++ b/src/include/optimizer/clauses.h @@ -45,6 +45,7 @@ extern bool contain_subplans(Node *clause); extern char max_parallel_hazard(Query *parse); extern bool is_parallel_safe(PlannerInfo *root, Node *node); extern bool contain_nonstrict_functions(Node *clause); +extern bool expr_contains(Node *expr, Node *sub); extern bool contain_exec_param(Node *clause, List *param_ids); extern bool contain_leaked_vars(Node *clause); diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index 9dade3b9416bf..88e1b8f9179ba 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -63,6 +63,15 @@ extern void add_partial_path(RelOptInfo *parent_rel, Path *new_path); extern bool add_partial_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, Cost startup_cost, Cost total_cost, List *pathkeys); +extern bool add_path_precheck_extras(RelOptInfo *parent_rel, + int disabled_nodes, Cost startup_cost, + Cost total_cost, List *pathkeys, + Relids required_outer, List *extras); +extern bool add_partial_path_precheck_extras(RelOptInfo *parent_rel, + int disabled_nodes, + Cost startup_cost, + Cost total_cost, List *pathkeys, + List *extras); extern Path *create_seqscan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer, int parallel_workers); @@ -231,7 +240,13 @@ extern bool index_orderby_returnable(IndexOptInfo *index, int indexcol, extern bool orderby_tlist_match(Expr *expr, Expr *orderby); extern QualCost path_target_cost(PlannerInfo *root, Path *path, PathTarget *target); -extern bool index_path_emits_orderby(IndexPath *ipath); +extern List *path_extra_exprs(Path *path); +extern bool path_emits_extras(Path *path); +extern bool expr_worth_emitting(PlannerInfo *root, Node *expr); +extern List *query_tlist_exprs(PlannerInfo *root, RelOptInfo *rel); +extern bool path_covers_extras(Path *path, List *extras); +extern QualCost indexonly_qual_cost(PlannerInfo *root, IndexPath *ipath, + List *qpquals, QualCost cost); extern ProjectionPath *create_projection_path(PlannerInfo *root, RelOptInfo *rel, Path *subpath, diff --git a/src/test/regress/expected/gist.out b/src/test/regress/expected/gist.out index 9c06a949ea1d5..1770c96118094 100644 --- a/src/test/regress/expected/gist.out +++ b/src/test/regress/expected/gist.out @@ -843,6 +843,27 @@ select f.id from gist_join_fact f join gist_join_dim j on j.id = f.id Index Cond: (j.id = f.id) (11 rows) +-- Nor for a cheap ORDER BY expression: the plain point distance is computed +-- again above the join, which costs less than carrying it up. +create index on gist_join_fact using gist (p); +explain (verbose, costs off) +select f.id, f.p <-> point(5,5) as d + from gist_join_fact f join gist_join_dim j on j.id = f.id + order by f.p <-> point(5,5) limit 3; + QUERY PLAN +------------------------------------------------------------------------------- + Limit + Output: f.id, ((f.p <-> '(5,5)'::point)) + -> Nested Loop + Output: f.id, (f.p <-> '(5,5)'::point) + -> Index Scan using gist_join_fact_p_idx on pg_temp.gist_join_fact f + Output: f.id, f.p + Order By: (f.p <-> '(5,5)'::point) + -> Index Scan using gist_join_dim_id_idx on pg_temp.gist_join_dim j + Output: j.id + Index Cond: (j.id = f.id) +(10 rows) + reset enable_seqscan; reset enable_bitmapscan; reset enable_indexonlyscan; @@ -885,7 +906,7 @@ select gist_ios_total('select gist_ios_costly(i) from gist_ios_cost order by i') -- With nothing else to recommend it (no qual, no useful order), an -- index-only scan that reads the expression still beats a seq scan that has --- to compute it: add_path() keeps both, since their targets differ. +-- to compute it: add_path() keeps the path that emits the value. explain (verbose, costs off) select gist_ios_costly(i) from gist_ios_cost order by gist_ios_costly(i); QUERY PLAN @@ -917,18 +938,229 @@ select gist_ios_costly(i) from gist_ios_cost order by i desc limit 3; Output: (gist_ios_costly(i)), i (4 rows) --- Below a join it isn't considered. -explain (costs off) -select gist_ios_costly(a.i) from gist_ios_cost a join gist_ios_cost b using (i); - QUERY PLAN ------------------------------------------ +-- The expression may be part of a larger one, or appear in a qual. +explain (verbose, costs off) +select gist_ios_costly(i) + 1 from gist_ios_cost + where gist_ios_costly(i) % 3 = 0; + QUERY PLAN +-------------------------------------------------------------------------------------- + Index Only Scan using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost + Output: ((gist_ios_costly(i)) + 1) + Filter: (((gist_ios_costly(gist_ios_cost.i)) % 3) = 0) +(3 rows) + +-- Below a join, the scan emits it and the join passes it up, through more +-- than one join level. +create temp table gist_ios_dim (i int, tag text); +insert into gist_ios_dim select g, 't' || g from generate_series(1, 10000, 7) g; +create index on gist_ios_dim (i); +create temp table gist_ios_dim2 (k int, i int); +insert into gist_ios_dim2 select g, g * 3 from generate_series(1, 3000) g; +vacuum analyze gist_ios_dim, gist_ios_dim2; +-- (Each join method is pinned; it's the value passing up that's tested.) +set enable_hashjoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(a.i), d.tag from gist_ios_cost a + join gist_ios_dim d on d.i = a.i; + QUERY PLAN +---------------------------------------------------------------------------------------------- + Merge Join + Output: (gist_ios_costly(a.i)), d.tag + Merge Cond: (a.i = d.i) + -> Index Only Scan using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost a + Output: a.i, (gist_ios_costly(a.i)) + -> Index Scan using gist_ios_dim_i_idx on pg_temp.gist_ios_dim d + Output: d.i, d.tag +(7 rows) + +reset enable_nestloop; +set enable_mergejoin = off; +explain (verbose, costs off) +select gist_ios_costly(a.i), d.tag, n.k from gist_ios_cost a + join gist_ios_dim d on d.i = a.i join gist_ios_dim2 n on n.i = a.i; + QUERY PLAN +---------------------------------------------------------------------------------------------- + Nested Loop + Output: (gist_ios_costly(a.i)), d.tag, n.k + Join Filter: (a.i = d.i) + -> Nested Loop + Output: d.tag, d.i, n.k, n.i + -> Seq Scan on pg_temp.gist_ios_dim2 n + Output: n.k, n.i + -> Index Scan using gist_ios_dim_i_idx on pg_temp.gist_ios_dim d + Output: d.i, d.tag + Index Cond: (d.i = n.i) + -> Index Only Scan using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost a + Output: a.i, (gist_ios_costly(a.i)) + Index Cond: (a.i = n.i) +(13 rows) + +reset enable_hashjoin; +reset enable_mergejoin; +-- When everything the scan emits is an index column, it returns the +-- index's own tlist without projecting (here including k, which nothing +-- needs); the value is still read above the join. +create temp table gist_ios_wide (k int, i int); +insert into gist_ios_wide select g, g from generate_series(1, 10000) g; +create index on gist_ios_wide (k, gist_ios_costly(i)) include (i); +vacuum analyze gist_ios_wide; +set enable_mergejoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(w.i), d.tag from gist_ios_wide w + join gist_ios_dim d on d.i = w.i; + QUERY PLAN +------------------------------------------------------------------------------------------------ Hash Join - Hash Cond: (a.i = b.i) - -> Seq Scan on gist_ios_cost a + Output: (gist_ios_costly(w.i)), d.tag + Hash Cond: (w.i = d.i) + -> Index Only Scan using gist_ios_wide_k_gist_ios_costly_i_i_idx on pg_temp.gist_ios_wide w + Output: w.k, (gist_ios_costly(w.i)), w.i -> Hash - -> Seq Scan on gist_ios_cost b -(5 rows) + Output: d.tag, d.i + -> Seq Scan on pg_temp.gist_ios_dim d + Output: d.tag, d.i +(9 rows) + +reset enable_mergejoin; +reset enable_nestloop; +drop table gist_ios_wide; +-- But not from the nullable side of an outer join: there the query's Var +-- differs from the index's (it may be null), so the value read below the +-- join isn't the one asked for. +explain (verbose, costs off) +select coalesce(gist_ios_costly(a.i), -1) from gist_ios_dim d + left join gist_ios_cost a on a.i = d.i * 2; + QUERY PLAN +--------------------------------------------------------- + Hash Left Join + Output: COALESCE(gist_ios_costly(a.i), '-1'::integer) + Hash Cond: ((d.i * 2) = a.i) + -> Seq Scan on pg_temp.gist_ios_dim d + Output: d.i, d.tag + -> Hash + Output: a.i + -> Seq Scan on pg_temp.gist_ios_cost a + Output: a.i +(9 rows) + +-- A partitioned table whose partitions all have the index: each child reads +-- the value and the Append passes it up, even when a partition's columns +-- are in a different order. +create temp table gist_ios_part (i int, pad text) partition by range (i); +create temp table gist_ios_part1 (pad text, i int); +alter table gist_ios_part attach partition gist_ios_part1 + for values from (1) to (5001); +create temp table gist_ios_part2 partition of gist_ios_part + for values from (5001) to (10001); +insert into gist_ios_part select g, 'p' from generate_series(1, 10000) g; +create index on gist_ios_part (i, gist_ios_costly(i)); +vacuum analyze gist_ios_part; +set enable_mergejoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(p.i), d.tag from gist_ios_part p + join gist_ios_dim d on d.i = p.i; + QUERY PLAN +-------------------------------------------------------------------------------------------------------- + Hash Join + Output: (gist_ios_costly(p.i)), d.tag + Hash Cond: (p.i = d.i) + -> Append + -> Index Only Scan using gist_ios_part1_i_gist_ios_costly_i_idx on pg_temp.gist_ios_part1 p_1 + Output: p_1.i, (gist_ios_costly(p_1.i)) + -> Index Only Scan using gist_ios_part2_i_gist_ios_costly_i_idx on pg_temp.gist_ios_part2 p_2 + Output: p_2.i, (gist_ios_costly(p_2.i)) + -> Hash + Output: d.tag, d.i + -> Seq Scan on pg_temp.gist_ios_dim d + Output: d.tag, d.i +(12 rows) + +-- An index the children can't return a value from (hash indexes return +-- nothing) doesn't stop the Append passing up the one they can. +create function gist_ios_costly2(int) returns int + language plpgsql immutable cost 100000 + as $$ begin return $1 * 3; end $$; +create index on gist_ios_part using hash (gist_ios_costly2(i)); +explain (verbose, costs off) +select gist_ios_costly(p.i), gist_ios_costly2(p.i), d.tag from gist_ios_part p + join gist_ios_dim d on d.i = p.i; + QUERY PLAN +-------------------------------------------------------------------------------------------------------- + Hash Join + Output: (gist_ios_costly(p.i)), gist_ios_costly2(p.i), d.tag + Hash Cond: (p.i = d.i) + -> Append + -> Index Only Scan using gist_ios_part1_i_gist_ios_costly_i_idx on pg_temp.gist_ios_part1 p_1 + Output: p_1.i, (gist_ios_costly(p_1.i)) + -> Index Only Scan using gist_ios_part2_i_gist_ios_costly_i_idx on pg_temp.gist_ios_part2 p_2 + Output: p_2.i, (gist_ios_costly(p_2.i)) + -> Hash + Output: d.tag, d.i + -> Seq Scan on pg_temp.gist_ios_dim d + Output: d.tag, d.i +(12 rows) + +drop function gist_ios_costly2(int) cascade; +NOTICE: drop cascades to index gist_ios_part_gist_ios_costly2_i_idx +reset enable_mergejoin; +reset enable_nestloop; +-- An index created on only one partition doesn't count: every child must +-- emit the value for the Append to. +create temp table gist_ios_part_x (i int) partition by range (i); +create temp table gist_ios_part_x1 partition of gist_ios_part_x + for values from (1) to (5001); +create temp table gist_ios_part_x2 partition of gist_ios_part_x + for values from (5001) to (10001); +insert into gist_ios_part_x select g from generate_series(1, 10000) g; +create index on gist_ios_part_x1 (i, gist_ios_costly(i)); +vacuum analyze gist_ios_part_x; +set enable_mergejoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(p.i), d.tag from gist_ios_part_x p + join gist_ios_dim d on d.i = p.i; + QUERY PLAN +------------------------------------------------------ + Hash Join + Output: gist_ios_costly(p.i), d.tag + Hash Cond: (p.i = d.i) + -> Append + -> Seq Scan on pg_temp.gist_ios_part_x1 p_1 + Output: p_1.i + -> Seq Scan on pg_temp.gist_ios_part_x2 p_2 + Output: p_2.i + -> Hash + Output: d.tag, d.i + -> Seq Scan on pg_temp.gist_ios_dim d + Output: d.tag, d.i +(12 rows) +reset enable_mergejoin; +reset enable_nestloop; +-- Each of these returns what the same query does without index-only scans. +create temp table gist_ios_check as +select gist_ios_costly(a.i) as e, d.tag from gist_ios_cost a + join gist_ios_dim d on d.i = a.i; +set enable_indexonlyscan = off; +select count(*) as mismatches from ( + (select gist_ios_costly(a.i) as e, d.tag from gist_ios_cost a + join gist_ios_dim d on d.i = a.i + except all select * from gist_ios_check) + union all + (select * from gist_ios_check + except all select gist_ios_costly(a.i), d.tag from gist_ios_cost a + join gist_ios_dim d on d.i = a.i)) s; + mismatches +------------ + 0 +(1 row) + +reset enable_indexonlyscan; +drop table gist_ios_check, gist_ios_part, gist_ios_part_x, gist_ios_dim, + gist_ios_dim2; drop function gist_ios_total(text); drop table gist_ios_cost; drop function gist_ios_costly(int); diff --git a/src/test/regress/sql/gist.sql b/src/test/regress/sql/gist.sql index 04402409b0886..2a402dee3f773 100644 --- a/src/test/regress/sql/gist.sql +++ b/src/test/regress/sql/gist.sql @@ -424,6 +424,13 @@ select nextval('gist_join_cnt') - 1 as calls_during_join; explain (verbose, costs off) select f.id from gist_join_fact f join gist_join_dim j on j.id = f.id order by gist_join_pt(f.p) <-> point(5,5) limit 3; +-- Nor for a cheap ORDER BY expression: the plain point distance is computed +-- again above the join, which costs less than carrying it up. +create index on gist_join_fact using gist (p); +explain (verbose, costs off) +select f.id, f.p <-> point(5,5) as d + from gist_join_fact f join gist_join_dim j on j.id = f.id + order by f.p <-> point(5,5) limit 3; reset enable_seqscan; reset enable_bitmapscan; reset enable_indexonlyscan; @@ -456,7 +463,7 @@ select gist_ios_total('select gist_ios_costly(i) from gist_ios_cost order by i') < 10000 as ios_expr_is_free; -- With nothing else to recommend it (no qual, no useful order), an -- index-only scan that reads the expression still beats a seq scan that has --- to compute it: add_path() keeps both, since their targets differ. +-- to compute it: add_path() keeps the path that emits the value. explain (verbose, costs off) select gist_ios_costly(i) from gist_ios_cost order by gist_ios_costly(i); explain (verbose, costs off) @@ -465,9 +472,114 @@ select gist_ios_costly(i) from gist_ios_cost; -- still reads the expression from the index. explain (verbose, costs off) select gist_ios_costly(i) from gist_ios_cost order by i desc limit 3; --- Below a join it isn't considered. -explain (costs off) -select gist_ios_costly(a.i) from gist_ios_cost a join gist_ios_cost b using (i); +-- The expression may be part of a larger one, or appear in a qual. +explain (verbose, costs off) +select gist_ios_costly(i) + 1 from gist_ios_cost + where gist_ios_costly(i) % 3 = 0; +-- Below a join, the scan emits it and the join passes it up, through more +-- than one join level. +create temp table gist_ios_dim (i int, tag text); +insert into gist_ios_dim select g, 't' || g from generate_series(1, 10000, 7) g; +create index on gist_ios_dim (i); +create temp table gist_ios_dim2 (k int, i int); +insert into gist_ios_dim2 select g, g * 3 from generate_series(1, 3000) g; +vacuum analyze gist_ios_dim, gist_ios_dim2; +-- (Each join method is pinned; it's the value passing up that's tested.) +set enable_hashjoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(a.i), d.tag from gist_ios_cost a + join gist_ios_dim d on d.i = a.i; +reset enable_nestloop; +set enable_mergejoin = off; +explain (verbose, costs off) +select gist_ios_costly(a.i), d.tag, n.k from gist_ios_cost a + join gist_ios_dim d on d.i = a.i join gist_ios_dim2 n on n.i = a.i; +reset enable_hashjoin; +reset enable_mergejoin; +-- When everything the scan emits is an index column, it returns the +-- index's own tlist without projecting (here including k, which nothing +-- needs); the value is still read above the join. +create temp table gist_ios_wide (k int, i int); +insert into gist_ios_wide select g, g from generate_series(1, 10000) g; +create index on gist_ios_wide (k, gist_ios_costly(i)) include (i); +vacuum analyze gist_ios_wide; +set enable_mergejoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(w.i), d.tag from gist_ios_wide w + join gist_ios_dim d on d.i = w.i; +reset enable_mergejoin; +reset enable_nestloop; +drop table gist_ios_wide; +-- But not from the nullable side of an outer join: there the query's Var +-- differs from the index's (it may be null), so the value read below the +-- join isn't the one asked for. +explain (verbose, costs off) +select coalesce(gist_ios_costly(a.i), -1) from gist_ios_dim d + left join gist_ios_cost a on a.i = d.i * 2; +-- A partitioned table whose partitions all have the index: each child reads +-- the value and the Append passes it up, even when a partition's columns +-- are in a different order. +create temp table gist_ios_part (i int, pad text) partition by range (i); +create temp table gist_ios_part1 (pad text, i int); +alter table gist_ios_part attach partition gist_ios_part1 + for values from (1) to (5001); +create temp table gist_ios_part2 partition of gist_ios_part + for values from (5001) to (10001); +insert into gist_ios_part select g, 'p' from generate_series(1, 10000) g; +create index on gist_ios_part (i, gist_ios_costly(i)); +vacuum analyze gist_ios_part; +set enable_mergejoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(p.i), d.tag from gist_ios_part p + join gist_ios_dim d on d.i = p.i; +-- An index the children can't return a value from (hash indexes return +-- nothing) doesn't stop the Append passing up the one they can. +create function gist_ios_costly2(int) returns int + language plpgsql immutable cost 100000 + as $$ begin return $1 * 3; end $$; +create index on gist_ios_part using hash (gist_ios_costly2(i)); +explain (verbose, costs off) +select gist_ios_costly(p.i), gist_ios_costly2(p.i), d.tag from gist_ios_part p + join gist_ios_dim d on d.i = p.i; +drop function gist_ios_costly2(int) cascade; +reset enable_mergejoin; +reset enable_nestloop; +-- An index created on only one partition doesn't count: every child must +-- emit the value for the Append to. +create temp table gist_ios_part_x (i int) partition by range (i); +create temp table gist_ios_part_x1 partition of gist_ios_part_x + for values from (1) to (5001); +create temp table gist_ios_part_x2 partition of gist_ios_part_x + for values from (5001) to (10001); +insert into gist_ios_part_x select g from generate_series(1, 10000) g; +create index on gist_ios_part_x1 (i, gist_ios_costly(i)); +vacuum analyze gist_ios_part_x; +set enable_mergejoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(p.i), d.tag from gist_ios_part_x p + join gist_ios_dim d on d.i = p.i; +reset enable_mergejoin; +reset enable_nestloop; +-- Each of these returns what the same query does without index-only scans. +create temp table gist_ios_check as +select gist_ios_costly(a.i) as e, d.tag from gist_ios_cost a + join gist_ios_dim d on d.i = a.i; +set enable_indexonlyscan = off; +select count(*) as mismatches from ( + (select gist_ios_costly(a.i) as e, d.tag from gist_ios_cost a + join gist_ios_dim d on d.i = a.i + except all select * from gist_ios_check) + union all + (select * from gist_ios_check + except all select gist_ios_costly(a.i), d.tag from gist_ios_cost a + join gist_ios_dim d on d.i = a.i)) s; +reset enable_indexonlyscan; +drop table gist_ios_check, gist_ios_part, gist_ios_part_x, gist_ios_dim, + gist_ios_dim2; drop function gist_ios_total(text); drop table gist_ios_cost; drop function gist_ios_costly(int); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index a9275933ed5a2..68cf1538df727 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3689,6 +3689,7 @@ count_param_references_context cp_hash_func create_upper_paths_hook_type createdb_failure_params +credit_free_context crosstab_HashEnt crosstab_cat_desc curl_infotype From e748c83f47e2777136118c8b5546f18c12b620d3 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Tue, 20 Oct 2026 16:33:05 -0400 Subject: [PATCH 12/12] Carry index-computed values through outer joins, Gather and ordered Appends The previous commit passes an index's extra values (see path_extra_exprs()) up through inner joins and an unordered Append, but stops at three places, where the expression is then computed above them: * The nullable side of an outer join. Above the join, the query's copy of the expression has the join's bit in its Vars' nullingrels, and the scan's copy doesn't. The join null-extends the scan's value along with the Vars, so the two agree on every row exactly when the expression yields NULL for NULL inputs. So for a strict expression, join_path_target() now relabels the value with the joinrel's Vars (their nullingrels) as it passes it up, and setrefs.c's fix_join_expr() matches it against the input's copy at an outer join the way it already matches Vars there: equal but for nullingrels, with the upper copy's a superset of the input's (search_indexed_tlist_for_nulled_expr()); path_target_cost() credits it the same way. A non-strict expression is still computed above the join. An index-only scan matches the query's targetlist without nullingrels, so it emits the value in the first place; an ordering IndexScan's ORDER BY values still aren't emitted on the nullable side. * Partial paths, which emitted only parallel-safe values. A worker reads the stored value; nothing evaluates the expression, so a parallel-restricted one is fine. Gather and Gather Merge now pass a partial path's values up (gather_target()), and a Gather over the cheapest partial path that emits values is considered as well as one over the cheapest partial path. * An ordered Append or MergeAppend, and a parallel Append. add_extras_append_path() now also builds those, over child paths that emit the values and have the ordering, and over the children's partial paths, collecting the children as generate_orderedappend_paths() does. The expression also has to be compared without nullingrels when a scan decides what to emit. The new strip_nullingrels() in clauses.c does that; remove_nulling_relids() can't, as it errors out on the negative special varnos (ROWID_VAR in a MERGE). Author: Greg Burd Discussion: https://postgr.es/m/8s5lT8erXzBMugXJQ6Wginbp_gc2B4hmCYhF0Q0GpnuK87eCAOcKs5K31AWCwraCNFIQt42wwkow_MPPubHg485MWv-zDBY4NL_JX9yJEEg=@burd.me --- src/backend/optimizer/path/allpaths.c | 260 ++++++++++++++++++++++---- src/backend/optimizer/plan/setrefs.c | 127 ++++++++++++- src/backend/optimizer/util/clauses.c | 46 +++++ src/backend/optimizer/util/pathnode.c | 225 +++++++++++++++++++--- src/include/optimizer/clauses.h | 1 + src/test/regress/expected/gist.out | 214 ++++++++++++++++++++- src/test/regress/sql/gist.sql | 106 ++++++++++- src/tools/pgindent/typedefs.list | 1 + 8 files changed, 907 insertions(+), 73 deletions(-) diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 047d197c68ce9..ef94dc5b746cf 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -103,6 +103,7 @@ static void set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, static void set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte); static void create_plain_partial_paths(PlannerInfo *root, RelOptInfo *rel); +static PathTarget *gather_target(RelOptInfo *rel, Path *subpath); static void add_extras_append_path(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); static void set_rel_consider_parallel(PlannerInfo *root, RelOptInfo *rel, @@ -1411,10 +1412,39 @@ set_grouped_rel_pathlist(PlannerInfo *root, RelOptInfo *rel) } +/* + * extras_child_path + * The cheapest path of 'childrel' from 'paths' that emits every value + * in 'extras' (in the parent's terms) and is ordered by 'pathkeys' + * (NIL for no requirement), or NULL. Partial paths are sorted by total + * cost too, so the same search serves partial_pathlist. + */ +static Path * +extras_child_path(PlannerInfo *root, RelOptInfo *childrel, List *paths, + List *extras, List *pathkeys) +{ + List *child_extras; + ListCell *lp; + + child_extras = (List *) + adjust_appendrel_attrs_multilevel(root, (Node *) extras, + childrel, childrel->top_parent); + foreach(lp, paths) + { + Path *path = (Path *) lfirst(lp); + + if (path->param_info == NULL && + pathkeys_contained_in(pathkeys, path->pathkeys) && + path_covers_extras(path, child_extras)) + return path; + } + return NULL; +} + /* * add_extras_append_path - * Build an unordered, unparameterized Append over child paths that all - * emit the same extra values, if there are any worth emitting. + * Build Appends over child paths that all emit the same extra values, + * if there are any worth emitting. * * A partitioned table's index on an expression is an index on that * expression in every partition, so each child may have an index-only scan @@ -1426,8 +1456,10 @@ set_grouped_rel_pathlist(PlannerInfo *root, RelOptInfo *rel) * emits it. create_append_child_plan() puts each child's values in the * Append's order. * - * Only an unordered, non-partial Append is built this way; an ordered or - * parallel Append still computes the expression above it. + * We build an unordered Append, a MergeAppend for each ordering some child + * provides (or an ordered Append if the partitions are ordered that way, as + * generate_orderedappend_paths() does), and a parallel Append over the + * children's partial paths. */ static void add_extras_append_path(PlannerInfo *root, RelOptInfo *rel, @@ -1435,9 +1467,10 @@ add_extras_append_path(PlannerInfo *root, RelOptInfo *rel, { List *extras = NIL; List *tlist_exprs; - AppendPathInput input = {0}; + List *orderings = NIL; + List *partition_pathkeys = NIL; + bool partition_pathkeys_partial = true; PathTarget *target; - AppendPath *apath; ListCell *lc; if (rel->reloptkind != RELOPT_BASEREL || live_childrels == NIL || @@ -1484,39 +1517,159 @@ add_extras_append_path(PlannerInfo *root, RelOptInfo *rel, if (extras == NIL) return; - foreach(lc, live_childrels) + target = copy_pathtarget(rel->reltarget); + foreach(lc, extras) + add_new_column_to_pathtarget(target, (Expr *) lfirst(lc)); + target = set_pathtarget_cost_width(root, target); + + /* + * Unordered Append; also collect the orderings children offer. If some + * child has no path that emits the values, we can't build this or an + * ordered one, but may still build the parallel Append below. + */ { - RelOptInfo *childrel = (RelOptInfo *) lfirst(lc); - List *child_extras; - Path *best = NULL; - ListCell *lp; - - child_extras = (List *) - adjust_appendrel_attrs_multilevel(root, (Node *) extras, - childrel, childrel->top_parent); - foreach(lp, childrel->pathlist) + AppendPathInput input = {0}; + AppendPath *apath; + bool ok = true; + + foreach(lc, live_childrels) { - Path *path = (Path *) lfirst(lp); + RelOptInfo *childrel = (RelOptInfo *) lfirst(lc); + Path *best = extras_child_path(root, childrel, + childrel->pathlist, + extras, NIL); + ListCell *lp; - if (path->param_info == NULL && - path_covers_extras(path, child_extras)) + if (best == NULL) { - best = path; /* pathlist is sorted by total cost */ + ok = false; + orderings = NIL; break; } + accumulate_append_subpath(best, &input.subpaths, NULL, + &input.child_append_relid_sets); + foreach(lp, childrel->pathlist) + { + Path *path = (Path *) lfirst(lp); + + if (path->pathkeys != NIL && path_emits_extras(path) && + !list_member(orderings, path->pathkeys)) + orderings = lappend(orderings, path->pathkeys); + } + } + if (ok) + { + apath = create_append_path(root, rel, input, NIL, NULL, 0, false, + -1); + apath->path.pathtarget = target; + add_path(rel, (Path *) apath); } - if (best == NULL) - return; - accumulate_append_subpath(best, &input.subpaths, NULL, - &input.child_append_relid_sets); } - apath = create_append_path(root, rel, input, NIL, NULL, 0, false, -1); - target = copy_pathtarget(rel->reltarget); - foreach(lc, extras) - add_new_column_to_pathtarget(target, (Expr *) lfirst(lc)); - apath->path.pathtarget = set_pathtarget_cost_width(root, target); - add_path(rel, (Path *) apath); + /* Ordered: MergeAppend, or Append if the partitions are in that order */ + if (rel->part_scheme != NULL && + partitions_are_ordered(rel->boundinfo, rel->live_parts)) + partition_pathkeys = build_partition_pathkeys(root, rel, + ForwardScanDirection, + &partition_pathkeys_partial); + foreach(lc, orderings) + { + List *pathkeys = (List *) lfirst(lc); + AppendPathInput input = {0}; + bool match_partition_order; + bool ok = true; + ListCell *lcr; + + /* only orderings the parent's paths could be using */ + pathkeys = truncate_useless_pathkeys(root, rel, pathkeys); + if (pathkeys == NIL) + continue; + match_partition_order = + pathkeys_contained_in(pathkeys, partition_pathkeys) || + (!partition_pathkeys_partial && + pathkeys_contained_in(partition_pathkeys, pathkeys)); + + foreach(lcr, live_childrels) + { + RelOptInfo *childrel = (RelOptInfo *) lfirst(lcr); + Path *best = extras_child_path(root, childrel, + childrel->pathlist, + extras, pathkeys); + + if (best == NULL) + { + ok = false; + break; + } + + /* as generate_orderedappend_paths() collects them */ + if (match_partition_order) + input.subpaths = lappend(input.subpaths, + get_singleton_append_subpath(best, + &input.child_append_relid_sets)); + else + accumulate_append_subpath(best, &input.subpaths, NULL, + &input.child_append_relid_sets); + } + if (!ok) + continue; + + if (match_partition_order) + { + AppendPath *apath; + + apath = create_append_path(root, rel, input, pathkeys, NULL, 0, + false, -1); + apath->path.pathtarget = target; + add_path(rel, (Path *) apath); + } + else + { + MergeAppendPath *mapath; + + mapath = create_merge_append_path(root, rel, input.subpaths, + input.child_append_relid_sets, + pathkeys, NULL); + mapath->path.pathtarget = target; + add_path(rel, (Path *) mapath); + } + } + + /* Parallel Append over the children's partial paths */ + if (rel->consider_parallel) + { + AppendPathInput input = {0}; + AppendPath *apath; + int parallel_workers = 0; + + foreach(lc, live_childrels) + { + RelOptInfo *childrel = (RelOptInfo *) lfirst(lc); + Path *best = extras_child_path(root, childrel, + childrel->partial_pathlist, + extras, NIL); + + if (best == NULL) + return; + parallel_workers = Max(parallel_workers, best->parallel_workers); + accumulate_append_subpath(best, &input.partial_subpaths, NULL, + &input.child_append_relid_sets); + } + if (enable_parallel_append) + { + parallel_workers = Max(parallel_workers, + pg_leftmost_one_pos32(list_length(live_childrels)) + 1); + parallel_workers = Min(parallel_workers, + max_parallel_workers_per_gather); + } + if (parallel_workers <= 0) + return; + apath = create_append_path(root, rel, input, NIL, NULL, + parallel_workers, enable_parallel_append, + -1); + apath->path.pathtarget = target; + add_partial_path(rel, (Path *) apath); + } } /* @@ -3398,10 +3551,32 @@ generate_gather_paths(PlannerInfo *root, RelOptInfo *rel, bool override_rows) cheapest_partial_path = linitial(rel->partial_pathlist); rows = compute_gather_rows(cheapest_partial_path); simple_gather_path = (Path *) - create_gather_path(root, rel, cheapest_partial_path, rel->reltarget, + create_gather_path(root, rel, cheapest_partial_path, + gather_target(rel, cheapest_partial_path), NULL, rowsp); add_path(rel, simple_gather_path); + /* + * A partial path that emits extra values (see path_extra_exprs()) is + * rarely the cheapest; gather the cheapest such path too, passing its + * values up, so whatever is above can use them. + */ + foreach(lc, rel->partial_pathlist) + { + Path *subpath = (Path *) lfirst(lc); + + if (subpath == cheapest_partial_path) + continue; + if (path_emits_extras(subpath)) + { + rows = compute_gather_rows(subpath); + add_path(rel, (Path *) + create_gather_path(root, rel, subpath, + subpath->pathtarget, NULL, rowsp)); + break; + } + } + /* * For each useful ordering, we can consider an order-preserving Gather * Merge. @@ -3415,12 +3590,31 @@ generate_gather_paths(PlannerInfo *root, RelOptInfo *rel, bool override_rows) continue; rows = compute_gather_rows(subpath); - path = create_gather_merge_path(root, rel, subpath, rel->reltarget, + path = create_gather_merge_path(root, rel, subpath, + gather_target(rel, subpath), subpath->pathkeys, NULL, rowsp); add_path(rel, &path->path); } } +/* + * gather_target + * The target for a Gather or Gather Merge over 'subpath': the rel's + * reltarget, or the subpath's own if it emits extra values (see + * path_extra_exprs()), so they pass up through the Gather. + * + * A Gather over a projection to the final scan/join target, as built by + * apply_scanjoin_target_to_paths(), takes rel->reltarget, which by then is + * that target. + */ +static PathTarget * +gather_target(RelOptInfo *rel, Path *subpath) +{ + if (subpath->pathtarget != rel->reltarget && path_emits_extras(subpath)) + return subpath->pathtarget; + return rel->reltarget; +} + /* * get_useful_pathkeys_for_relation * Determine which orderings of a relation might be useful. @@ -3609,7 +3803,7 @@ generate_useful_gather_paths(PlannerInfo *root, RelOptInfo *rel, bool override_r rows = compute_gather_rows(subpath); path = create_gather_merge_path(root, rel, subpath, - rel->reltarget, + gather_target(rel, subpath), subpath->pathkeys, NULL, rowsp); diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index e6b6439ff2d01..ed8bdaefcae49 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -19,6 +19,7 @@ #include "catalog/pg_type.h" #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" +#include "optimizer/clauses.h" #include "optimizer/optimizer.h" #include "optimizer/pathnode.h" #include "optimizer/planmain.h" @@ -51,6 +52,12 @@ typedef struct int num_vars; /* number of plain Var tlist entries */ bool has_ph_vars; /* are there PlaceHolderVar entries? */ bool has_non_vars; /* are there other entries? */ + + /* + * non-Var entries without nullingrels, built on demand by + * search_indexed_tlist_for_nulled_expr(); not freed with the struct + */ + List *nulled_tles; tlist_vinfo vars[FLEXIBLE_ARRAY_MEMBER]; /* has num_vars entries */ } indexed_tlist; @@ -180,6 +187,10 @@ static Var *search_indexed_tlist_for_phv(PlaceHolderVar *phv, static Var *search_indexed_tlist_for_non_var(Expr *node, indexed_tlist *itlist, int newvarno); +static Var *search_indexed_tlist_for_nulled_expr(PlannerInfo *root, + Expr *node, + indexed_tlist *itlist, + int newvarno); static Var *search_indexed_tlist_for_sortgroupref(Expr *node, Index sortgroupref, indexed_tlist *itlist, @@ -2923,7 +2934,8 @@ set_dummy_tlist_references(Plan *plan, int rtoffset) * * The result of this function is an indexed_tlist struct to pass to * search_indexed_tlist_for_var() and siblings. - * When done, the indexed_tlist may be freed with a single pfree(). + * When done, the indexed_tlist may be freed with a single pfree(); a + * nulled_tles list built for it later stays in the planner's context. */ static indexed_tlist * build_tlist_index(List *tlist) @@ -2940,6 +2952,7 @@ build_tlist_index(List *tlist) itlist->tlist = tlist; itlist->has_ph_vars = false; itlist->has_non_vars = false; + itlist->nulled_tles = NIL; /* Find the Vars and fill in the index array */ vinfo = itlist->vars; @@ -2991,6 +3004,7 @@ build_tlist_index_other_vars(List *tlist, int ignore_rel) itlist->tlist = tlist; itlist->has_ph_vars = false; itlist->has_non_vars = false; + itlist->nulled_tles = NIL; /* Find the desired Vars and fill in the index array */ vinfo = itlist->vars; @@ -3177,6 +3191,107 @@ search_indexed_tlist_for_non_var(Expr *node, return NULL; /* no match */ } +/* + * expr_nullingrels + * The union of the nullingrels of the Vars and PlaceHolderVars in 'node'. + */ +static Relids +expr_nullingrels(Node *node) +{ + Relids result = NULL; + ListCell *lc; + + foreach(lc, pull_var_clause(node, PVC_INCLUDE_PLACEHOLDERS | + PVC_RECURSE_AGGREGATES | + PVC_RECURSE_WINDOWFUNCS)) + { + Node *v = (Node *) lfirst(lc); + + if (IsA(v, Var)) + result = bms_add_members(result, ((Var *) v)->varnullingrels); + else + result = bms_add_members(result, + ((PlaceHolderVar *) v)->phnullingrels); + } + return result; +} + +/* + * search_indexed_tlist_for_nulled_expr + * Like search_indexed_tlist_for_non_var(), for an expression above an + * outer join whose Vars may carry nullingrels the input's don't. + * + * An input of an outer join may emit a non-Var expression it read from an + * index (see path_extra_exprs()). Above the join, the query's copy of that + * expression has the join's bit added to its Vars' nullingrels, so equal() + * won't match it; the join adds the bit because it null-extends those Vars. + * It null-extends the input's copy of the expression too, so the two agree + * on every row if the expression yields NULL when its Vars are NULL, that + * is if it is strict. Then match it as we match Vars here (NRM_SUPERSET): + * equal but for nullingrels, with the query's copy's nullingrels a superset + * of the input's. The planner relies on this when it carries such a value + * up through an outer join (see join_path_target()). + * + * The first call for an itlist saves its non-Var entries without + * nullingrels, each paired with the nullingrels it had, so that later calls + * don't strip them again. + */ +static Var * +search_indexed_tlist_for_nulled_expr(PlannerInfo *root, Expr *node, + indexed_tlist *itlist, int newvarno) +{ + Node *stripped = NULL; + Relids nullingrels = NULL; + ListCell *lc; + + Assert(itlist->has_non_vars); + if (IsA(node, Const) || IsA(node, Param) || + bms_is_empty(root->outer_join_rels)) + return NULL; + + if (itlist->nulled_tles == NIL) + { + foreach(lc, itlist->tlist) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + TargetEntry *ntle; + + if (tle->expr == NULL || + IsA(tle->expr, Var) || IsA(tle->expr, PlaceHolderVar)) + continue; + ntle = flatCopyTargetEntry(tle); + ntle->expr = (Expr *) strip_nullingrels((Node *) tle->expr); + itlist->nulled_tles = lappend(itlist->nulled_tles, ntle); + itlist->nulled_tles = lappend(itlist->nulled_tles, + expr_nullingrels((Node *) tle->expr)); + } + } + + for (int i = 0; i < list_length(itlist->nulled_tles); i += 2) + { + TargetEntry *tle = list_nth_node(TargetEntry, itlist->nulled_tles, i); + + if (stripped == NULL) + { + if (contain_nonstrict_functions((Node *) node) || + !contain_var_clause((Node *) node)) + return NULL; + stripped = strip_nullingrels((Node *) node); + nullingrels = expr_nullingrels((Node *) node); + } + if (equal(stripped, tle->expr) && + bms_is_subset(list_nth(itlist->nulled_tles, i + 1), nullingrels)) + { + Var *newvar = makeVarFromTargetEntry(newvarno, tle); + + newvar->varnosyn = 0; /* wasn't ever a plain Var */ + newvar->varattnosyn = 0; + return newvar; + } + } + return NULL; +} + /* * search_indexed_tlist_for_sortgroupref --- find a sort/group expression * @@ -3384,6 +3499,11 @@ fix_join_expr_mutator(Node *node, fix_join_expr_context *context) newvar = search_indexed_tlist_for_non_var((Expr *) node, context->outer_itlist, OUTER_VAR); + if (newvar == NULL && context->nrm_match == NRM_SUPERSET) + newvar = search_indexed_tlist_for_nulled_expr(context->root, + (Expr *) node, + context->outer_itlist, + OUTER_VAR); if (newvar) return (Node *) newvar; } @@ -3392,6 +3512,11 @@ fix_join_expr_mutator(Node *node, fix_join_expr_context *context) newvar = search_indexed_tlist_for_non_var((Expr *) node, context->inner_itlist, INNER_VAR); + if (newvar == NULL && context->nrm_match == NRM_SUPERSET) + newvar = search_indexed_tlist_for_nulled_expr(context->root, + (Expr *) node, + context->inner_itlist, + INNER_VAR); if (newvar) return (Node *) newvar; } diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c index af8a3166a3199..da2249db69ee4 100644 --- a/src/backend/optimizer/util/clauses.c +++ b/src/backend/optimizer/util/clauses.c @@ -118,6 +118,7 @@ static bool contain_agg_clause_walker(Node *node, void *context); static bool find_window_functions_walker(Node *node, WindowFuncLists *lists); static bool contain_subplans_walker(Node *node, void *context); static bool expr_contains_walker(Node *node, void *context); +static Node *strip_nullingrels_mutator(Node *node, void *context); static bool contain_mutable_functions_walker(Node *node, void *context); static bool xmlexpr_is_immutable(XmlExpr *xexpr); static bool contain_volatile_functions_walker(Node *node, void *context); @@ -398,6 +399,51 @@ expr_contains_walker(Node *node, void *context) return expression_tree_walker(node, expr_contains_walker, context); } +/* + * strip_nullingrels + * A copy of 'node' with every Var's and PlaceHolderVar's nullingrels + * cleared. + * + * Used to compare an expression read below an outer join with the query's + * copy above it (see query_tlist_exprs() in pathnode.c and + * search_indexed_tlist_for_nulled_expr() in setrefs.c). + * remove_nulling_relids() can't do this: it tests each Var's varno with + * bms_is_member(), which errors out on the negative special varnos, such as + * the ROWID_VAR a MERGE's targetlist contains. + */ +Node * +strip_nullingrels(Node *node) +{ + return strip_nullingrels_mutator(node, NULL); +} + +static Node * +strip_nullingrels_mutator(Node *node, void *context) +{ + if (node == NULL) + return NULL; + if (IsA(node, Var)) + { + Var *var = (Var *) node; + + if (bms_is_empty(var->varnullingrels)) + return node; + var = copyObject(var); + var->varnullingrels = NULL; + return (Node *) var; + } + if (IsA(node, PlaceHolderVar)) + { + PlaceHolderVar *phv; + + phv = (PlaceHolderVar *) + expression_tree_mutator(node, strip_nullingrels_mutator, context); + phv->phnullingrels = NULL; + return (Node *) phv; + } + return expression_tree_mutator(node, strip_nullingrels_mutator, context); +} + /***************************************************************************** * Check clauses for mutable functions *****************************************************************************/ diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index d0ff9c668cfe6..6ce6145978bdf 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -402,9 +402,10 @@ set_cheapest(RelOptInfo *parent_rel) * rel->reltarget with the extra expressions appended, so they're its tail. * * Sort, Material and Memoize don't project and share their input's target, - * so they emit what it emits. Nothing else emits extra values: other path - * types build their own targets, and an upper rel's targets differ from its - * reltarget for reasons of their own. + * so they emit what it emits; so may a Gather or Gather Merge given its + * input's target (see generate_gather_paths()). Nothing else emits extra + * values: other path types build their own targets, and an upper rel's + * targets differ from its reltarget for reasons of their own. */ static ListCell * path_extras_start(Path *path, int *nextras) @@ -423,6 +424,8 @@ path_extras_start(Path *path, int *nextras) case T_IncrementalSortPath: case T_MaterialPath: case T_MemoizePath: + case T_GatherPath: + case T_GatherMergePath: { Path *subpath; @@ -430,6 +433,10 @@ path_extras_start(Path *path, int *nextras) subpath = ((MaterialPath *) path)->subpath; else if (IsA(path, MemoizePath)) subpath = ((MemoizePath *) path)->subpath; + else if (IsA(path, GatherPath)) + subpath = ((GatherPath *) path)->subpath; + else if (IsA(path, GatherMergePath)) + subpath = ((GatherMergePath *) path)->subpath; else subpath = ((SortPath *) path)->subpath; if (subpath->parent != rel || @@ -448,6 +455,7 @@ path_extras_start(Path *path, int *nextras) return NULL; break; case T_AppendPath: + case T_MergeAppendPath: /* see add_extras_append_path() */ if (rel->reloptkind != RELOPT_BASEREL) return NULL; @@ -1447,6 +1455,14 @@ query_tlist_exprs(PlannerInfo *root, RelOptInfo *rel) exprs = (List *) adjust_appendrel_attrs_multilevel(root, (Node *) exprs, rel, rel->top_parent); + + /* + * Above an outer join that null-extends the rel, the query's Vars carry + * nullingrels that a scan's don't. The scan may still emit the value; + * whether it can be used above the join is join_path_target()'s call. + */ + if (!bms_is_empty(root->outer_join_rels)) + exprs = (List *) strip_nullingrels((Node *) exprs); return exprs; } @@ -1469,16 +1485,16 @@ query_tlist_exprs(PlannerInfo *root, RelOptInfo *rel) * at any depth, with a reference to it; the paths that must compute it * instead pay for it when the expression is charged. * - * An expression is matched with equal(), so a Var of a rel on the nullable - * side of an outer join, which carries varnullingrels in the targetlist, - * never matches the index's own: the value read below the join isn't the - * one the query asks for above it. + * The query's targetlist is compared without nullingrels (see + * query_tlist_exprs()); whether a value read below an outer join can be used + * above it is decided as it's passed up (see relabel_extra_for_join()). * - * A partial path gets the extra values only if they're parallel-safe. + * A partial path may emit a value that isn't parallel-safe: a worker reads + * the stored value, it never evaluates the expression, and nothing above + * the scan does either, since it references the scan's output column. */ static PathTarget * -indexonly_path_target(PlannerInfo *root, IndexOptInfo *index, - bool partial_path) +indexonly_path_target(PlannerInfo *root, IndexOptInfo *index) { RelOptInfo *rel = index->rel; PathTarget *target = NULL; @@ -1499,8 +1515,6 @@ indexonly_path_target(PlannerInfo *root, IndexOptInfo *index, if (!index->canreturn[i++] || IsA(itle->expr, Var)) continue; - if (partial_path && !is_parallel_safe(root, (Node *) itle->expr)) - continue; if (!expr_worth_emitting(root, (Node *) itle->expr)) continue; @@ -1563,7 +1577,7 @@ create_index_path(PlannerInfo *root, pathnode->path.pathtype = indexonly ? T_IndexOnlyScan : T_IndexScan; pathnode->path.parent = rel; pathnode->path.pathtarget = indexonly ? - indexonly_path_target(root, index, partial_path) : + indexonly_path_target(root, index) : index_path_orderby_target(root, index, indexorderbys, indexorderbycols); pathnode->path.param_info = get_baserel_parampathinfo(root, rel, @@ -2800,6 +2814,99 @@ calc_non_nestloop_required_outer(Path *outer_path, Path *inner_path) return required_outer; } +/* state for relabel_extra_for_join()'s walker and mutator */ +typedef struct +{ + List *joinvars; + bool changed; + bool missing; +} relabel_context; + +static Var * +relabel_find_joinvar(Var *var, List *joinvars) +{ + ListCell *lc; + + foreach(lc, joinvars) + { + Var *jv = (Var *) lfirst(lc); + + if (IsA(jv, Var) && jv->varno == var->varno && + jv->varattno == var->varattno && jv->varlevelsup == 0) + return jv; + } + return NULL; +} + +/* First pass: does anything need relabeling, or is a Var missing? */ +static bool +relabel_check_walker(Node *node, relabel_context *context) +{ + if (node == NULL) + return false; + if (IsA(node, Var) && ((Var *) node)->varlevelsup == 0) + { + Var *var = (Var *) node; + Var *jv = relabel_find_joinvar(var, context->joinvars); + + if (jv == NULL) + context->missing = true; + else if (!bms_equal(jv->varnullingrels, var->varnullingrels)) + context->changed = true; + return context->missing; + } + return expression_tree_walker(node, relabel_check_walker, context); +} + +static Node * +relabel_extra_mutator(Node *node, relabel_context *context) +{ + if (node == NULL) + return NULL; + if (IsA(node, Var) && ((Var *) node)->varlevelsup == 0) + { + Var *var = copyObject((Var *) node); + Var *jv = relabel_find_joinvar(var, context->joinvars); + + var->varnullingrels = bms_copy(jv->varnullingrels); + return (Node *) var; + } + return expression_tree_mutator(node, relabel_extra_mutator, context); +} + +/* + * relabel_extra_for_join + * Express an input's extra value in terms of the joinrel's Vars. + * + * Above an outer join, the joinrel's reltarget has each Var of the nullable + * side with the join's bit added to its nullingrels, and the query's copy of + * an expression over those Vars has it too. The input's copy doesn't. The + * join null-extends the input's value along with the Vars, which gives the + * expression's value for null-extended rows only if the expression is + * strict, so for a non-strict one we can't pass the value up (return NULL). + * Otherwise return the expression with each Var replaced by the joinrel's + * version of it, which setrefs.c matches with the input's + * (search_indexed_tlist_for_nulled_expr()). If some Var isn't in the + * joinrel's reltarget, the value isn't needed above, so return NULL too. + */ +static Node * +relabel_extra_for_join(RelOptInfo *joinrel, Node *expr) +{ + relabel_context context; + + context.joinvars = joinrel->reltarget->exprs; + context.changed = false; + context.missing = false; + (void) relabel_check_walker(expr, &context); + if (context.missing) + return NULL; + if (!context.changed) + return expr; + if (contain_nonstrict_functions(expr)) + return NULL; + return relabel_extra_mutator(expr, &context); +} + /* * join_path_target * Choose the PathTarget for a join path. @@ -2812,17 +2919,14 @@ calc_non_nestloop_required_outer(Path *outer_path, Path *inner_path) * ordering IndexScan below the topmost join would read the value for * nothing. * - * Only values the query still needs above this join are passed up: a - * value that refers to a rel on the nullable side of an outer join this - * join forms would be wrong above it, but those never match the query's - * targetlist (its Vars carry varnullingrels the index's don't), so no input - * emits one; and every value we emit came from the final targetlist. + * Above an outer join a value from the nullable side is relabeled with the + * join's nullingrels, and passed up only if that gives the right answer for + * null-extended rows; see relabel_extra_for_join(). * - * Values must be parallel-safe if the join is, since a partial join path's - * target is computed in workers. - * - * Every value a scan emits passed expr_worth_emitting() there, so we don't - * look up its cost again for each join path. + * A value needn't be parallel-safe for a partial join path: the join only + * references its input's output column. Every value a scan emits passed + * expr_worth_emitting() there, so we don't look up its cost again for each + * join path. */ static PathTarget * join_path_target(PlannerInfo *root, RelOptInfo *joinrel, @@ -2842,9 +2946,9 @@ join_path_target(PlannerInfo *root, RelOptInfo *joinrel, nextras = list_length(outer_extras) + list_length(inner_extras); foreach(lc, list_concat(outer_extras, inner_extras)) { - Node *expr = (Node *) lfirst(lc); + Node *expr = relabel_extra_for_join(joinrel, (Node *) lfirst(lc)); - if (!joinrel->consider_parallel || is_parallel_safe(root, expr)) + if (expr != NULL) add_new_column_to_pathtarget(target, (Expr *) expr); else all_kept = false; @@ -3228,9 +3332,39 @@ typedef struct { PlannerInfo *root; List *free; + bool nulled_ok; /* see free_expr_match() */ + List *free_stripped; /* 'free' without nullingrels, on demand */ QualCost cost; } credit_free_context; +/* + * free_expr_match + * Is 'node' one of context->free? + * + * At an outer join (context->nulled_ok), also accept a strict 'node' that is + * one of them but for nullingrels, as setrefs.c does. + */ +static bool +free_expr_match(Node *node, credit_free_context *context) +{ + Node *stripped; + ListCell *lc; + + if (list_member(context->free, node)) + return true; + if (!context->nulled_ok || bms_is_empty(context->root->outer_join_rels)) + return false; + if (context->free_stripped == NIL) + context->free_stripped = (List *) strip_nullingrels((Node *) context->free); + stripped = strip_nullingrels(node); + foreach(lc, context->free_stripped) + { + if (equal(stripped, lfirst(lc))) + return !contain_nonstrict_functions(node); + } + return false; +} + /* * credit_free_exprs_walker * Subtract from context->cost the cost of each outermost subexpression @@ -3247,7 +3381,7 @@ credit_free_exprs_walker(Node *node, credit_free_context *context) { if (node == NULL || IsA(node, Var) || IsA(node, Const)) return false; - if (list_member(context->free, node)) + if (free_expr_match(node, context)) { QualCost ecost; @@ -3265,12 +3399,15 @@ credit_free_exprs_walker(Node *node, credit_free_context *context) * uses (see credit_free_exprs_walker()). */ static QualCost -free_exprs_cost(PlannerInfo *root, PathTarget *target, List *free) +free_exprs_cost(PlannerInfo *root, PathTarget *target, List *free, + bool nulled_ok) { credit_free_context context; context.root = root; context.free = free; + context.nulled_ok = nulled_ok; + context.free_stripped = NIL; context.cost = target->cost; if (free != NIL) (void) credit_free_exprs_walker((Node *) target->exprs, &context); @@ -3321,6 +3458,8 @@ indexonly_qual_cost(PlannerInfo *root, IndexPath *ipath, List *qpquals, context.root = root; context.free = indexonly_free_exprs(ipath); + context.nulled_ok = false; + context.free_stripped = NIL; context.cost = cost; if (context.free == NIL) return cost; @@ -3361,6 +3500,7 @@ join_free_exprs(JoinPath *jpath) * - a join, for values one of its inputs emits (join_free_exprs()); * - an index-only scan, for returnable index expression columns * (indexonly_free_exprs()); + * - a Gather or Gather Merge, for values its input emits; * * in either case wherever the value appears in the target, since setrefs.c * replaces every occurrence (see credit_free_exprs_walker()). And @@ -3380,12 +3520,37 @@ path_target_cost(PlannerInfo *root, Path *path, PathTarget *target) ListCell *lc; if (IsA(path, NestPath) || IsA(path, MergePath) || IsA(path, HashPath)) - return free_exprs_cost(root, target, - join_free_exprs((JoinPath *) path)); + { + JoinPath *jpath = (JoinPath *) path; + bool forms_oj; + + /* + * A join that forms outer joins matches nulled copies too, as + * set_join_references() does when the Join's ojrelids isn't empty. + * The inputs' relids are disjoint and both within the joinrel's, so + * it forms one exactly when the joinrel has more relids than they do. + */ + forms_oj = bms_num_members(path->parent->relids) > + bms_num_members(jpath->outerjoinpath->parent->relids) + + bms_num_members(jpath->innerjoinpath->parent->relids); + return free_exprs_cost(root, target, join_free_exprs(jpath), + forms_oj); + } if (IsA(path, IndexPath) && path->pathtype == T_IndexOnlyScan) return free_exprs_cost(root, target, - indexonly_free_exprs((IndexPath *) path)); + indexonly_free_exprs((IndexPath *) path), + false); + + /* set_upper_references() matches them in a Gather's subplan's tlist */ + if (IsA(path, GatherPath)) + return free_exprs_cost(root, target, + path_extra_exprs(((GatherPath *) path)->subpath), + false); + if (IsA(path, GatherMergePath)) + return free_exprs_cost(root, target, + path_extra_exprs(((GatherMergePath *) path)->subpath), + false); if (!IsA(path, IndexPath) || path->pathtype != T_IndexScan) return cost; diff --git a/src/include/optimizer/clauses.h b/src/include/optimizer/clauses.h index e0196edf109bb..f88278f10d98f 100644 --- a/src/include/optimizer/clauses.h +++ b/src/include/optimizer/clauses.h @@ -46,6 +46,7 @@ extern char max_parallel_hazard(Query *parse); extern bool is_parallel_safe(PlannerInfo *root, Node *node); extern bool contain_nonstrict_functions(Node *clause); extern bool expr_contains(Node *expr, Node *sub); +extern Node *strip_nullingrels(Node *node); extern bool contain_exec_param(Node *clause, List *param_ids); extern bool contain_leaked_vars(Node *clause); diff --git a/src/test/regress/expected/gist.out b/src/test/regress/expected/gist.out index 1770c96118094..1584cf415a8bc 100644 --- a/src/test/regress/expected/gist.out +++ b/src/test/regress/expected/gist.out @@ -1026,16 +1026,76 @@ select gist_ios_costly(w.i), d.tag from gist_ios_wide w reset enable_mergejoin; reset enable_nestloop; drop table gist_ios_wide; --- But not from the nullable side of an outer join: there the query's Var --- differs from the index's (it may be null), so the value read below the --- join isn't the one asked for. +-- Also from the nullable side of an outer join, because the function is +-- strict: a null-extended row gets NULL for the value, which is what the +-- function returns for a NULL argument. The value is used inside a larger, +-- non-strict expression above the join. +set enable_hashjoin = off; +set enable_nestloop = off; explain (verbose, costs off) select coalesce(gist_ios_costly(a.i), -1) from gist_ios_dim d left join gist_ios_cost a on a.i = d.i * 2; - QUERY PLAN ---------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------------- + Merge Left Join + Output: COALESCE((gist_ios_costly(a.i)), '-1'::integer) + Merge Cond: (((d.i * 2)) = a.i) + -> Sort + Output: d.i, ((d.i * 2)) + Sort Key: ((d.i * 2)) + -> Seq Scan on pg_temp.gist_ios_dim d + Output: d.i, (d.i * 2) + -> Index Only Scan using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost a + Output: a.i, (gist_ios_costly(a.i)) +(10 rows) + +reset enable_hashjoin; +reset enable_nestloop; +-- And from both sides' nullable rows of a FULL JOIN. +set enable_hashjoin = off; +explain (verbose, costs off) +select gist_ios_costly(a.i), d.tag from gist_ios_dim d + full join gist_ios_cost a on a.i = d.i * 2; + QUERY PLAN +---------------------------------------------------------------------------------------------- + Merge Full Join + Output: (gist_ios_costly(a.i)), d.tag + Merge Cond: (((d.i * 2)) = a.i) + -> Sort + Output: d.tag, d.i, ((d.i * 2)) + Sort Key: ((d.i * 2)) + -> Seq Scan on pg_temp.gist_ios_dim d + Output: d.tag, d.i, (d.i * 2) + -> Index Only Scan using gist_ios_cost_i_gist_ios_costly_i_idx on pg_temp.gist_ios_cost a + Output: a.i, (gist_ios_costly(a.i)) +(10 rows) + +reset enable_hashjoin; +select count(*) as total, count(v) as with_value, + count(*) filter (where v = i) as same + from (select gist_ios_costly(a.i) as v, a.i from gist_ios_dim d + full join gist_ios_cost a on a.i = d.i * 2) s; + total | with_value | same +-------+------------+------- + 10714 | 10000 | 10000 +(1 row) + +-- A function that isn't strict may return something else for NULL, so it's +-- computed above the join. +create function gist_ios_nonstrict(int) returns int + language plpgsql immutable called on null input cost 100000 + as $$ begin return coalesce($1, -1); end $$; +create index on gist_ios_cost (i, gist_ios_nonstrict(i)); +vacuum analyze gist_ios_cost; +set enable_mergejoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_nonstrict(a.i) from gist_ios_dim d + left join gist_ios_cost a on a.i = d.i * 2; + QUERY PLAN +------------------------------------------------- Hash Left Join - Output: COALESCE(gist_ios_costly(a.i), '-1'::integer) + Output: gist_ios_nonstrict(a.i) Hash Cond: ((d.i * 2) = a.i) -> Seq Scan on pg_temp.gist_ios_dim d Output: d.i, d.tag @@ -1045,6 +1105,126 @@ select coalesce(gist_ios_costly(a.i), -1) from gist_ios_dim d Output: a.i (9 rows) +reset enable_mergejoin; +reset enable_nestloop; +select count(*) filter (where v = -1) as null_extended + from (select gist_ios_nonstrict(a.i) as v from gist_ios_dim d + left join gist_ios_cost a on a.i = d.i * 2) s; + null_extended +--------------- + 714 +(1 row) + +drop function gist_ios_nonstrict(int) cascade; +NOTICE: drop cascades to index gist_ios_cost_i_gist_ios_nonstrict_i_idx +-- A parallel-restricted function: the value is read from the index inside +-- the workers, not evaluated there, and passed up through the Gather. (Not +-- a temp table, which can't be scanned in parallel.) +create function gist_ios_restricted(int) returns int + language plpgsql immutable strict parallel restricted cost 100000 + as $$ begin return $1; end $$; +create table gist_ios_par (i int); +insert into gist_ios_par select g from generate_series(1, 10000) g; +create index on gist_ios_par (i, gist_ios_restricted(i)); +vacuum analyze gist_ios_par; +set parallel_setup_cost = 0; +set parallel_tuple_cost = 0; +set min_parallel_index_scan_size = 0; +set max_parallel_workers_per_gather = 2; +set parallel_leader_participation = off; +-- (Pin the scan type: these tables aren't temp, so VACUUM may not get to +-- mark them all-visible while other sessions hold snapshots.) +set enable_seqscan = off; +set enable_bitmapscan = off; +explain (verbose, costs off) +select gist_ios_restricted(i) from gist_ios_par; + QUERY PLAN +------------------------------------------------------------------------------------------------------ + Gather + Output: (gist_ios_restricted(i)) + Workers Planned: 2 + -> Parallel Index Only Scan using gist_ios_par_i_gist_ios_restricted_i_idx on public.gist_ios_par + Output: i, (gist_ios_restricted(i)) +(5 rows) + +select count(*), sum(v) from (select gist_ios_restricted(i) as v from gist_ios_par) s; + count | sum +-------+---------- + 10000 | 50005000 +(1 row) + +-- Also through a Gather Merge, which sorts the workers' rows by a column +-- the index doesn't lead with. +create table gist_ios_gm (k int, i int); +insert into gist_ios_gm select g % 97, g from generate_series(1, 20000) g; +create index on gist_ios_gm (i, gist_ios_restricted(i)) include (k); +vacuum analyze gist_ios_gm; +explain (verbose, costs off) +select k, gist_ios_restricted(i) from gist_ios_gm where i > 100 order by k; + QUERY PLAN +------------------------------------------------------------------------------------------------------------ + Gather Merge + Output: k, (gist_ios_restricted(i)) + Workers Planned: 2 + -> Sort + Output: k, i, (gist_ios_restricted(i)) + Sort Key: gist_ios_gm.k + -> Parallel Index Only Scan using gist_ios_gm_i_gist_ios_restricted_i_k_idx on public.gist_ios_gm + Output: k, i, (gist_ios_restricted(i)) + Index Cond: (gist_ios_gm.i > 100) +(9 rows) + +select count(*) filter (where v = i) as same, count(*) as total + from (select k, i, gist_ios_restricted(i) as v from gist_ios_gm + where i > 100 order by k) s; + same | total +-------+------- + 19900 | 19900 +(1 row) + +drop table gist_ios_gm; +-- And through a Parallel Append over partitions that all have the index. +-- (Different sizes, so that the children's cost order, which is the plan's +-- order, doesn't depend on how many pages VACUUM could mark all-visible +-- when other sessions hold snapshots.) +create table gist_ios_ppart (i int) partition by range (i); +create table gist_ios_ppart1 partition of gist_ios_ppart + for values from (1) to (2001); +create table gist_ios_ppart2 partition of gist_ios_ppart + for values from (2001) to (10001); +insert into gist_ios_ppart select g from generate_series(1, 10000) g; +create index on gist_ios_ppart (i, gist_ios_restricted(i)); +vacuum analyze gist_ios_ppart; +explain (verbose, costs off) +select gist_ios_restricted(i) from gist_ios_ppart; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------------- + Gather + Output: (gist_ios_restricted(gist_ios_ppart.i)) + Workers Planned: 2 + -> Parallel Append + -> Parallel Index Only Scan using gist_ios_ppart2_i_gist_ios_restricted_i_idx on public.gist_ios_ppart2 gist_ios_ppart_2 + Output: gist_ios_ppart_2.i, (gist_ios_restricted(gist_ios_ppart_2.i)) + -> Parallel Index Only Scan using gist_ios_ppart1_i_gist_ios_restricted_i_idx on public.gist_ios_ppart1 gist_ios_ppart_1 + Output: gist_ios_ppart_1.i, (gist_ios_restricted(gist_ios_ppart_1.i)) +(8 rows) + +select count(*), sum(v) from (select gist_ios_restricted(i) as v from gist_ios_ppart) s; + count | sum +-------+---------- + 10000 | 50005000 +(1 row) + +drop table gist_ios_ppart; +reset enable_seqscan; +reset enable_bitmapscan; +reset parallel_setup_cost; +reset parallel_tuple_cost; +reset min_parallel_index_scan_size; +reset max_parallel_workers_per_gather; +reset parallel_leader_participation; +drop table gist_ios_par; +drop function gist_ios_restricted(int); -- A partitioned table whose partitions all have the index: each child reads -- the value and the Append passes it up, even when a partition's columns -- are in a different order. @@ -1107,6 +1287,28 @@ drop function gist_ios_costly2(int) cascade; NOTICE: drop cascades to index gist_ios_part_gist_ios_costly2_i_idx reset enable_mergejoin; reset enable_nestloop; +-- An ordered Append passes the value up as well, here into a merge join. +set enable_hashjoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(p.i), d.tag from gist_ios_part p + join gist_ios_dim d on d.i = p.i order by p.i; + QUERY PLAN +-------------------------------------------------------------------------------------------------------- + Merge Join + Output: (gist_ios_costly(p.i)), d.tag, p.i + Merge Cond: (p.i = d.i) + -> Append + -> Index Only Scan using gist_ios_part1_i_gist_ios_costly_i_idx on pg_temp.gist_ios_part1 p_1 + Output: p_1.i, (gist_ios_costly(p_1.i)) + -> Index Only Scan using gist_ios_part2_i_gist_ios_costly_i_idx on pg_temp.gist_ios_part2 p_2 + Output: p_2.i, (gist_ios_costly(p_2.i)) + -> Index Scan using gist_ios_dim_i_idx on pg_temp.gist_ios_dim d + Output: d.i, d.tag +(10 rows) + +reset enable_hashjoin; +reset enable_nestloop; -- An index created on only one partition doesn't count: every child must -- emit the value for the Append to. create temp table gist_ios_part_x (i int) partition by range (i); diff --git a/src/test/regress/sql/gist.sql b/src/test/regress/sql/gist.sql index 2a402dee3f773..b0686fc868758 100644 --- a/src/test/regress/sql/gist.sql +++ b/src/test/regress/sql/gist.sql @@ -512,12 +512,104 @@ select gist_ios_costly(w.i), d.tag from gist_ios_wide w reset enable_mergejoin; reset enable_nestloop; drop table gist_ios_wide; --- But not from the nullable side of an outer join: there the query's Var --- differs from the index's (it may be null), so the value read below the --- join isn't the one asked for. +-- Also from the nullable side of an outer join, because the function is +-- strict: a null-extended row gets NULL for the value, which is what the +-- function returns for a NULL argument. The value is used inside a larger, +-- non-strict expression above the join. +set enable_hashjoin = off; +set enable_nestloop = off; explain (verbose, costs off) select coalesce(gist_ios_costly(a.i), -1) from gist_ios_dim d left join gist_ios_cost a on a.i = d.i * 2; +reset enable_hashjoin; +reset enable_nestloop; +-- And from both sides' nullable rows of a FULL JOIN. +set enable_hashjoin = off; +explain (verbose, costs off) +select gist_ios_costly(a.i), d.tag from gist_ios_dim d + full join gist_ios_cost a on a.i = d.i * 2; +reset enable_hashjoin; +select count(*) as total, count(v) as with_value, + count(*) filter (where v = i) as same + from (select gist_ios_costly(a.i) as v, a.i from gist_ios_dim d + full join gist_ios_cost a on a.i = d.i * 2) s; +-- A function that isn't strict may return something else for NULL, so it's +-- computed above the join. +create function gist_ios_nonstrict(int) returns int + language plpgsql immutable called on null input cost 100000 + as $$ begin return coalesce($1, -1); end $$; +create index on gist_ios_cost (i, gist_ios_nonstrict(i)); +vacuum analyze gist_ios_cost; +set enable_mergejoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_nonstrict(a.i) from gist_ios_dim d + left join gist_ios_cost a on a.i = d.i * 2; +reset enable_mergejoin; +reset enable_nestloop; +select count(*) filter (where v = -1) as null_extended + from (select gist_ios_nonstrict(a.i) as v from gist_ios_dim d + left join gist_ios_cost a on a.i = d.i * 2) s; +drop function gist_ios_nonstrict(int) cascade; +-- A parallel-restricted function: the value is read from the index inside +-- the workers, not evaluated there, and passed up through the Gather. (Not +-- a temp table, which can't be scanned in parallel.) +create function gist_ios_restricted(int) returns int + language plpgsql immutable strict parallel restricted cost 100000 + as $$ begin return $1; end $$; +create table gist_ios_par (i int); +insert into gist_ios_par select g from generate_series(1, 10000) g; +create index on gist_ios_par (i, gist_ios_restricted(i)); +vacuum analyze gist_ios_par; +set parallel_setup_cost = 0; +set parallel_tuple_cost = 0; +set min_parallel_index_scan_size = 0; +set max_parallel_workers_per_gather = 2; +set parallel_leader_participation = off; +-- (Pin the scan type: these tables aren't temp, so VACUUM may not get to +-- mark them all-visible while other sessions hold snapshots.) +set enable_seqscan = off; +set enable_bitmapscan = off; +explain (verbose, costs off) +select gist_ios_restricted(i) from gist_ios_par; +select count(*), sum(v) from (select gist_ios_restricted(i) as v from gist_ios_par) s; +-- Also through a Gather Merge, which sorts the workers' rows by a column +-- the index doesn't lead with. +create table gist_ios_gm (k int, i int); +insert into gist_ios_gm select g % 97, g from generate_series(1, 20000) g; +create index on gist_ios_gm (i, gist_ios_restricted(i)) include (k); +vacuum analyze gist_ios_gm; +explain (verbose, costs off) +select k, gist_ios_restricted(i) from gist_ios_gm where i > 100 order by k; +select count(*) filter (where v = i) as same, count(*) as total + from (select k, i, gist_ios_restricted(i) as v from gist_ios_gm + where i > 100 order by k) s; +drop table gist_ios_gm; +-- And through a Parallel Append over partitions that all have the index. +-- (Different sizes, so that the children's cost order, which is the plan's +-- order, doesn't depend on how many pages VACUUM could mark all-visible +-- when other sessions hold snapshots.) +create table gist_ios_ppart (i int) partition by range (i); +create table gist_ios_ppart1 partition of gist_ios_ppart + for values from (1) to (2001); +create table gist_ios_ppart2 partition of gist_ios_ppart + for values from (2001) to (10001); +insert into gist_ios_ppart select g from generate_series(1, 10000) g; +create index on gist_ios_ppart (i, gist_ios_restricted(i)); +vacuum analyze gist_ios_ppart; +explain (verbose, costs off) +select gist_ios_restricted(i) from gist_ios_ppart; +select count(*), sum(v) from (select gist_ios_restricted(i) as v from gist_ios_ppart) s; +drop table gist_ios_ppart; +reset enable_seqscan; +reset enable_bitmapscan; +reset parallel_setup_cost; +reset parallel_tuple_cost; +reset min_parallel_index_scan_size; +reset max_parallel_workers_per_gather; +reset parallel_leader_participation; +drop table gist_ios_par; +drop function gist_ios_restricted(int); -- A partitioned table whose partitions all have the index: each child reads -- the value and the Append passes it up, even when a partition's columns -- are in a different order. @@ -547,6 +639,14 @@ select gist_ios_costly(p.i), gist_ios_costly2(p.i), d.tag from gist_ios_part p drop function gist_ios_costly2(int) cascade; reset enable_mergejoin; reset enable_nestloop; +-- An ordered Append passes the value up as well, here into a merge join. +set enable_hashjoin = off; +set enable_nestloop = off; +explain (verbose, costs off) +select gist_ios_costly(p.i), d.tag from gist_ios_part p + join gist_ios_dim d on d.i = p.i order by p.i; +reset enable_hashjoin; +reset enable_nestloop; -- An index created on only one partition doesn't count: every child must -- emit the value for the Append to. create temp table gist_ios_part_x (i int) partition by range (i); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 68cf1538df727..0f56d4e90a587 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -4222,6 +4222,7 @@ regexp regexp_matches_ctx registered_buffer regproc +relabel_context relopt_bool relopt_enum relopt_enum_elt_def