Repository navigation
Keep complete results in an Arrow result cache - #16
Merged
Merged
Conversation
Small results stay in memory; larger ones are written as Arrow IPC files in a locked per-process session directory, renamed into place once complete. Limits per result and for the whole cache keep a readable prefix, evicting the oldest other results first. The C ABI in sidequery_http.h exchanges batches through the Arrow C Data Interface.
sqlnb_result_tee passes a statement's first rows through in full chunks while storing every row, and can stop reading once storing stops. sqlnb_results, sqlnb_result and sqlnb_result_drop list, read back through DuckDB's Arrow scan, and remove results. The MCP agent connection cannot reach any of them.
The Arrow crates read the local time zone's name (chrono's clock feature, through iana-time-zone), which calls CoreFoundation on Apple platforms. The extension only loaded after another extension that links the framework.
sqlnb_result_tee no longer limits the statement's last pipeline to one thread. It wraps the statement's result collector: every thread stores the rows it produces as Arrow batches keyed by their executor batch, the file's footer lists them in that order, and only the first rows are passed on to the client, exactly the first rows stored. The runtime smoke test now loads SQLNB alone in a fresh process and checks the stored order of a filtered scan, a sort, several Parquet files and a join, and the prefix kept when a verified read stops at the limit.
sqlnb_result_tee takes ttl_seconds (24 hours unless named, NULL for as long as DuckDB runs). A result expires that long after its statement finished or sqlnb_result last read it, and is then gone like an evicted one. There is no timer: binding or beginning a tee, listing, reading and dropping remove what expired. sqlnb_results() reports last_used_ms and expires_ms. The cache keeps 8 GiB unless a tee names another limit, and a tee whose limit is below what is kept evicts the oldest results when it begins instead of when it first stores a batch. Nothing more is written to a result file once its volume would have less than 5 GiB free, or a tenth of a volume smaller than 50 GiB. Storing stops with limit_reason 'disk-space'; the statement runs on. The defaults now live with the cache in Rust and reach the function through sidequery_result_defaults.
The thread that passes the head on created and released the wrapped collector's shared state without the lock under which a thread leaving the head tests it.
A statement that stops reading once storing has stopped now completes its head first; threads that stopped earlier left the head with rows from wherever each of them was. The documentation no longer promises that every returned row is stored: both are the first rows of the result, and a cache that stops storing keeps fewer.
Quack sends a result's last batches from an event it inserts while it finalizes. The tee made the result readable as soon as that finalize returned, so a statement that failed in the event, with the client gone or the statement interrupted, left its result behind. The result is now published from an event that runs after those. Also say that the wrapped collector's own sink_state stays unset.
Every batch that ended before the lowest one left up to a head's worth of rows waiting, so a slow or unstarted lowest batch let them grow with the number of batches. Only the lowest ended batches now wait, up to the one with which they fill the head; later batches keep no copy.
DuckDB does not interrupt a pipeline's combine and finalize: a cancel that arrives then is raised afterwards, when the result has already been made readable. Over Quack a client that stopped reading and cancelled left the result of its failed statement listed and on disk. The statement now drops the result it published when it ends in error.
Brings in the upload endpoint and the Instant SQL changes. The README lists both the Instant SQL read policy and the result cache.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a result cache so the notebook can show a statement's first rows at once and later page through, sort, filter, aggregate or export the rest without running the statement again.
select * from sqlnb_result_tee((select ...), id := ...)returns the statement's first rows and stores every row. It wraps the statement's result collector, so the statement runs on all its threads as it would without the tee and each thread stores the Arrow batches it produces, labelled with their executor batch; the footer lists them in the statement's order. In a local measurement the tee cost no more than the statement alone (0.077 s against 0.079 s).sqlnb_result(id, first_row, row_count, row_number)reads a window or the whole result back and reads only the batches and columns a query needs;sqlnb_results()lists results andsqlnb_result_drop(id)removes one.Small results stay in memory. Larger ones are written as uncompressed Arrow IPC files under the user's cache directory, in a directory per process that is locked while the process runs and removed when it exits or, after a crash, by the next process. Limits default to 4 GiB per result and 8 GiB for the cache, results expire 24 hours after their last use, and nothing more is written once the cache's volume would drop below a free-space floor. A limit or a failure to store never fails or shortens the statement: the result keeps a prefix and reports why. A statement that fails or is interrupted leaves nothing behind. The MCP agent connection cannot use these functions.
The cache is written in Rust on the
arrow-*60.1.0 crates. On Apple platforms the extension now links CoreFoundation, which those crates need throughchrono, so it still loads on its own.docs/results.mdis the reference.The 109 Rust tests, the macOS release build and its runtime smoke pass locally.
test/sql/results.testhas not been run under DuckDB's test runner locally, and Linux and Windows have not been built; this pull request's workflows are their first check. The README line conflicts with the Instant SQL pull request's, so whichever lands second needsmainmerged in.