Repository navigation
Conversation
|
📦 Python package built successfully!
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #131 +/- ##
==========================================
+ Coverage 78.43% 79.16% +0.72%
==========================================
Files 121 122 +1
Lines 6920 7013 +93
Branches 1005 1017 +12
==========================================
+ Hits 5428 5552 +124
+ Misses 1182 1151 -31
Partials 310 310
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
|
🚀 Review App Deployment Started
|
tkislan
force-pushed
the
feat/dataframe-storage
branch
from
October 8, 2026 12:55
91ffced to
3b63336
Compare
tkislan
force-pushed
the
feat/dataframe-storage
branch
2 times, most recently
from
October 14, 2026 09:51
0d1a11a to
b832280
Compare
tkislan
force-pushed
the
feat/dataframe-storage
branch
from
November 10, 2026 17:19
5e6b392 to
dee4404
Compare
When the platform asks for it, a post_run_cell hook stores a Python or SQL
block's DataFrame result as the block's frame, in
.deepnote/dataframes/<notebookId>/<blockId>/, where the webapp's dataframe
reader serves block frames from. The frame is written by the same writer as
artifacts.store_dataframe, so it has the same Arrow file and manifest.
The webapp asks per execution through the execute_request metadata
deepnote.dataframeStorage = {notebookId, blockId}, and the hook runs only
when DEEPNOTE_DATAFRAME_STORAGE_ENABLED is on. It skips helper executions,
failed or interrupted cells, SQL query previews, PySpark and pandas-on-Spark
frames, non-DataFrame results and read-only mounts.
The hook never raises into the cell and suppresses warnings. A failed write
is reported to the webapp as TOOLKIT_RUNTIME_ERROR with code
DATAFRAME_STORAGE_WRITE_FAILED, once per kernel session per errno or
exception type, carrying the exception type but never its message. A Stop
during the write fails the execution with KeyboardInterrupt, so the
executor stops its queue.
store_frame now takes the frame's directory, and store_dataframe and the
hook share is_storable_frame.
tkislan
force-pushed
the
feat/dataframe-storage
branch
from
November 10, 2026 19:00
dee4404 to
c22b963
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai ignore
Summary
Phase 2 of the "Full Dataframe storage" RFC: storing a block's result automatically.
When the webapp asks for it, a
post_run_cellhook stores a Python or SQL block's DataFrame result as that block's frame. It writes.deepnote/dataframes/<notebookId>/<blockId>/, which is where the webapp's dataframe reader serves block frames from:BLOCK_DATAFRAMES_DIRECTORYin deepnote-internallibs/dataframe-storage-node;GET /v2/blocks/:blockId/dataframein deepnote/deepnote-internal#21080.The frame is written by the same writer as
artifacts.store_dataframe(#135, #142), so the Arrow file and the manifest are the same as for a named frame.deepnote_toolkit/artifacts/_block_frames.py(new) holds:init_deepnote_runtime();deepnote_toolkit/artifacts/_dataframe_storage.py:store_frametakes the frame's directory instead of(name, root);store_dataframeand the hook shareis_storable_frame.execute_request.metadata.deepnote.dataframeStorage = {notebookId, blockId}. In a detached run, these are the source notebook and block. The hook runs only whenDEEPNOTE_DATAFRAME_STORAGE_ENABLEDis on (1,true,yes,on, in any case, as_to_boolindeepnote_core/config/loader.py). The variable doesn't affectstore_dataframe.Out of scope, in other repos:
deepnote_store_dataframeblock setting;Behaviour worth reviewing
result.info.store_history, because IPython 9 setsexecution_counteven when no history is stored;EROFSbefore anything is converted.execute_reply: theexecute_resultreaches the client before the hook writes.TOOLKIT_RUNTIME_ERRORwithcode: DATAFRAME_STORAGE_WRITE_FAILED, the typetoolkit/errorssends to Sentry grouped bycode.KeyboardInterrupt, print "Error in callback" and keep the cell successful. So the hook setsresult.error_in_execand callsshowtraceback(). The reply then carriesKeyboardInterruptand the executor stops its queue.await, ipykernel runspost_run_cellunder a handler that only cancels the finished cell future. Stop during that write is lost and the run goes on, as with any blocking code at the end of such a cell.notebookIdandblockIdmust each match[A-Za-z0-9_-]{1,128}, because they become directories.Changes since the previous revision of this PR
mainas one commit, after feat(artifacts): add store_dataframe and delete_dataframe #135 was squash-merged and fix(artifacts): make stored DataFrames servable a page at a time #142 changed the writer.{notebookId, blockId}instead of{name, blockId}..deepnote/dataframes/<notebookId>/<blockId>/instead ofdeepnote_dataframes/<name>/.block_idfield._block_frames.py, sostore_dataframedoesn't import IPython orrequeststhrough the writer.RecordBatchFileWriter.write_batch,_to_arrow_table), instead of throughwrite_feather, which the writer no longer calls.tests/unit/artifacts/were removed.Not verified here
.deepnote/dataframes/sits under the project mount there.Test plan
tests/unit/artifactsandtests/unit/test_runtime_initialization.py: 184 passed on Python 3.13 and on 3.10, with the same warnings asmainon each.tests/unit: 1463 passed, 10 skipped. The 13 warnings are the same as onmain.raise_for_statusand the timeout;black,isort,flake8clean on the touched files;mypy deepnote_toolkit/clean.