Skip to content

feat(artifacts): store a block's DataFrame result automatically - #131

Draft
tkislan wants to merge 1 commit into
mainfrom
feat/dataframe-storage
Draft

tkislan wants to merge 1 commit into
mainfrom
feat/dataframe-storage

Conversation

@tkislan

@tkislan tkislan commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

@coderabbitai ignore

Summary

Phase 2 of the "Full Dataframe storage" RFC: storing a block's result automatically.

When the webapp asks for it, a post_run_cell hook stores a Python or SQL block's DataFrame result as that block's frame. It writes .deepnote/dataframes/<notebookId>/<blockId>/, which is where the webapp's dataframe reader serves block frames from:

  • BLOCK_DATAFRAMES_DIRECTORY in deepnote-internal libs/dataframe-storage-node;
  • the endpoint GET /v2/blocks/:blockId/dataframe in deepnote/deepnote-internal#21080.

The frame is written by the same writer as artifacts.store_dataframe (#135, #142), so the Arrow file and the manifest are the same as for a named frame.

  • deepnote_toolkit/artifacts/_block_frames.py (new) holds:
    • the hook, registered as one more guarded step in init_deepnote_runtime();
    • the request metadata reader;
    • the error reports.
  • deepnote_toolkit/artifacts/_dataframe_storage.py:
    • store_frame takes the frame's directory instead of (name, root);
    • store_dataframe and the hook share is_storable_frame.
  • Kernel contract: the webapp asks per execution with execute_request.metadata.deepnote.dataframeStorage = {notebookId, blockId}. In a detached run, these are the source notebook and block. The hook runs only when DEEPNOTE_DATAFRAME_STORAGE_ENABLED is on (1, true, yes, on, in any case, as _to_bool in deepnote_core/config/loader.py). The variable doesn't affect store_dataframe.
  • No code is added to user cells.

Out of scope, in other repos:

  • the deepnote_store_dataframe block setting;
  • the webapp and executor sending the metadata;
  • the feature flag that sets the environment variable;
  • deleting a block's frames when its setting is cleared or the block is deleted.

Behaviour worth reviewing

  • What is stored: only the cell's result value, and only a pandas or polars DataFrame. Skipped:
    • SQL query previews;
    • PySpark and pandas-on-Spark frames;
    • non-DataFrame results;
    • failed or interrupted cells;
    • helper executions, recognised by result.info.store_history, because IPython 9 sets execution_count even when no history is stored;
    • a read-only mount, whose first write fails with EROFS before anything is converted.
  • A run that stores nothing keeps the block's previous frame. If a re-run raises, returns something other than a DataFrame, returns a frame Arrow refuses, or fails to write, the manifest keeps naming the last frame the block stored, and the block endpoint keeps serving it. The webapp should read a block's frame only after that run's execute_reply: the execute_result reaches the client before the hook writes.
  • Never raises, never prints into the cell. Warnings are suppressed.
    • Failures are reported to the webapp once per kernel session per errno or exception type. The report is TOOLKIT_RUNTIME_ERROR with code: DATAFRAME_STORAGE_WRITE_FAILED, the type toolkit/errors sends to Sentry grouped by code.
    • The report carries the exception type, never its message, because pyarrow quotes the failing value.
    • Reporting is one attempt with a 2 s timeout. A failed report goes to the file logger only.
  • Stop during a write. IPython would catch a re-raised KeyboardInterrupt, print "Error in callback" and keep the cell successful. So the hook sets result.error_in_exec and calls showtraceback(). The reply then carries KeyboardInterrupt and the executor stops its queue.
    • No SIGINT handler is installed. For a cell with top-level await, ipykernel runs post_run_cell under a handler that only cancels the finished cell future. Stop during that write is lost and the run goes on, as with any blocking code at the end of such a cell.
  • Untrusted inputs. notebookId and blockId must each match [A-Za-z0-9_-]{1,128}, because they become directories.

Changes since the previous revision of this PR

Not verified here

  • Python 3.11, 3.12 and 3.14, and IPython 9. Python 3.13 (pyarrow 24.0) and 3.10 (pyarrow 16.1) ran, both with IPython 8.12.3.
  • A real s3fs mount on a pod, and whether .deepnote/dataframes/ sits under the project mount there.
  • The webapp side end to end, because it doesn't send the metadata yet.

Test plan

  • tests/unit/artifacts and tests/unit/test_runtime_initialization.py: 184 passed on Python 3.13 and on 3.10, with the same warnings as main on each.
  • Full tests/unit: 1463 passed, 10 skipped. The 13 warnings are the same as on main.
  • Broke the hook on purpose in 17 places, and every break failed a test:
    • the directory, and the order of the two ids;
    • validation of each id;
    • the flag;
    • the helper-execution, failed-cell and query-preview checks;
    • EROFS handling;
    • report de-duplication, and the exception message leaking into the report;
    • raise_for_status and the timeout;
    • warning suppression;
    • interrupt handling;
    • errno handling;
    • registration at startup.
  • black, isort, flake8 clean on the touched files; mypy deepnote_toolkit/ clean.
  • CI across the Python / pyarrow matrix
  • Pod test once the webapp sends the metadata

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

📦 Python package built successfully!

  • Version: 2.9.1.dev3+b822c79
  • Wheel: deepnote_toolkit-2.9.1.dev3+b822c79-py3-none-any.whl
  • Install:
    pip install "deepnote-toolkit @ https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/2.9.1.dev3%2Bb822c79/deepnote_toolkit-2.9.1.dev3%2Bb822c79-py3-none-any.whl"

@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.16%. Comparing base (6523327) to head (c22b963).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #131      +/-   ##
==========================================
+ Coverage   78.43%   79.16%   +0.72%     
==========================================
  Files         121      122       +1     
  Lines        6920     7013      +93     
  Branches     1005     1017      +12     
==========================================
+ Hits         5428     5552     +124     
+ Misses       1182     1151      -31     
  Partials      310      310              
Flag Coverage Δ
combined 79.16% <100.00%> (+0.72%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@deepnote-bot

deepnote-bot commented Oct 6, 2026 •

Copy link
Copy Markdown

🚀 Review App Deployment Started

📝 Description 🌐 Link / Info
🌍 Review application ra-131
🔑 Sign-in URL Click to sign-in
📊 Application logs View logs
🔄 Actions Click to redeploy
🚀 ArgoCD deployment View deployment
⏰ Last deployed 2026-11-10 19:18:36 (UTC)
📜 Deployed commit 757e063397983ec99c3b118562817f3908df17f6
🛠️ Toolkit version b822c79

@tkislan
tkislan force-pushed the feat/dataframe-storage branch from 91ffced to 3b63336 Compare October 8, 2026 12:55
@tkislan tkislan changed the title feat(dataframe-storage): store the full DataFrame of flagged blocks feat(dataframe-storage): store a block's result automatically Oct 8, 2026
@tkislan
tkislan changed the base branch from main to feat/artifacts-store-dataframe October 8, 2026 12:55
@tkislan
tkislan force-pushed the feat/dataframe-storage branch 2 times, most recently from 0d1a11a to b832280 Compare October 14, 2026 09:51
Base automatically changed from feat/artifacts-store-dataframe to main October 19, 2026 17:36
@tkislan
tkislan force-pushed the feat/dataframe-storage branch from 5e6b392 to dee4404 Compare November 10, 2026 17:19
When the platform asks for it, a post_run_cell hook stores a Python or SQL
block's DataFrame result as the block's frame, in
.deepnote/dataframes/<notebookId>/<blockId>/, where the webapp's dataframe
reader serves block frames from. The frame is written by the same writer as
artifacts.store_dataframe, so it has the same Arrow file and manifest.

The webapp asks per execution through the execute_request metadata
deepnote.dataframeStorage = {notebookId, blockId}, and the hook runs only
when DEEPNOTE_DATAFRAME_STORAGE_ENABLED is on. It skips helper executions,
failed or interrupted cells, SQL query previews, PySpark and pandas-on-Spark
frames, non-DataFrame results and read-only mounts.

The hook never raises into the cell and suppresses warnings. A failed write
is reported to the webapp as TOOLKIT_RUNTIME_ERROR with code
DATAFRAME_STORAGE_WRITE_FAILED, once per kernel session per errno or
exception type, carrying the exception type but never its message. A Stop
during the write fails the execution with KeyboardInterrupt, so the
executor stops its queue.

store_frame now takes the frame's directory, and store_dataframe and the
hook share is_storable_frame.
@tkislan
tkislan force-pushed the feat/dataframe-storage branch from dee4404 to c22b963 Compare November 10, 2026 19:00
@tkislan tkislan changed the title feat(dataframe-storage): store a block's result automatically feat(artifacts): store a block's DataFrame result automatically Nov 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants