Skip to content

feat(dsh): make the DeepSeek Harness store writable - #33

Open
zzyking wants to merge 7 commits into
skillsynchq:mainfrom
zzyking:feat/dsh-writable
Open

zzyking wants to merge 7 commits into
skillsynchq:mainfrom
zzyking:feat/dsh-writable

Conversation

@zzyking

@zzyking zzyking commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #19 — until that merges, the diff here includes its commits. The
new commits are feat: make the DeepSeek Harness store writable, fix(dsh): keep the previous copy until a save succeeds, confine delete, and feat(dsh): write session format v4.

Companion to #32, which does the same for Kimi. Both follow from the read-only
question raised on #18/#19: the bar I set there (a harness needs an official
import command before txcript writes its storage) was stricter than the one the
rest of the tree holds — of the twelve writable harnesses, only opencode goes
through a vendor import command.

Why it works

dsh discovers sessions by walking <root>/<project>/<session>/. No index, no
import command — writing the layout it scans for is enough to be found.

What makes it exact rather than approximate

dsh validates what it finds, and three of its checks fail the entire
listing rather than skipping the one bad session. Each is a hard requirement on
the writer:

Check Requirement
assertZstdHeaderFrame first Zstandard frame decodes to exactly one line — the header
assertStoredIdentity the header's id and cwd name the path the log was found at
checkRootEncoding one root never mixes .jsonl with .jsonl.zstd

The first is the interesting one. Compressing the whole log as a single frame
still round-trips through txcript's own reader, because that decodes across
frame boundaries — so no test written against txcript catches it. Only the
official reader does.

The third is the most destructive: writing the default zstd encoding into a
root that dsh configured for plaintext breaks that root under both
configurations, taking down the user's whole session list. So the root decides
the encoding, and only an empty root falls back to dsh's own default.

A duplicate id across two project directories is rejected the same way, so
re-saving a session whose cwd changed retires the copy filed under the old one.

Writing format v4

dsh 0.2.0-rc.2 reads the current format (v4) with full validation, so Common
sessions are written as v4 directly, laid out as the turn/step lifecycle that
validation enforces: each tool call is started by tool/call and answered by a
tool-role tool/result before its step ends. A few events exist only because
dsh requires them:

  • System head: an empty system/message heads the surface. dsh appends its
    system prompt when it resumes a session and accepts that only behind a system
    head; without one, the first resume works and the log dsh wrote fails its
    next load.
  • Unanswered calls are closed with dsh's own TOOL_NOT_STARTED repair, and
    that turn ends interrupted.
  • Model source: every assistant message names a provider (txcript) and
    model; its embedded stream is empty, the form dsh skips when replaying.

A native dsh body keeps its version, and save names the file for it
(session[.vN].jsonl[.zstd]), since dsh rejects a file name and header that
disagree. Other generations in the directory are removed.

Path derivation

Follows the official encodeSegment and projectKey: separators collapse to
-, [A-Za-z0-9._-] survives, every other UTF-16 code unit becomes ~XXXX,
and the project key is wrapped in --. Escaping over code units rather than
Unicode scalars is what keeps it injective over lone surrogates.

Escaping the separators is also what contains a traversing id: ../../evil
becomes the literal directory ..~002F..~002Fevil. That matches dsh, which
encodes rather than rejects — only a segment that is entirely . or .. is
special-cased.

Verification

dsh's own format catalog under its strictest settings (recovery: 'strict',
validation: 'current') accepts sessions converted from Claude Code, Codex,
and OpenCode (the largest about 14k events). A converted session was resumed
twice with dsh --profile headless --session-id; dsh answered from the
imported history, and the log it extended passed the same check after each
turn.

The official persistence backend's own listArtifacts and loadStored, run
against a txcript-written root:

  • it lists both a normal session and a _no-cwd one;
  • it decodes all 1257 event records of the former through decodeStorageRecord.

Re-introducing the single-frame bug makes that same check fail with dsh's own
corrupt Zstandard session log: first frame is not exactly one header line,
which is what confirms the oracle has teeth.

The derived project-directory names also reproduce those of the real local
session store exactly, including its non-ASCII path. Real dsh data was never
written to.

save writes to a temporary file and renames it into place, retiring other
copies only afterwards; delete only accepts a path that resolves (symlinks
included) to <root>/<project>/<session>.

dsh joins the path_safety and store_delete suites, and the CLI's refusal to
continue a dsh session in place is removed.

Full matrix green locally: fmt, clippy -D warnings, workspace tests,
no-default-features, both wasm32 invocations, MSRV 1.96.

@NishantJoshi00

Copy link
Copy Markdown
Collaborator

save deletes previous copies before writing the replacement. If that write fails, the old session is already gone. Can we retain it until the replacement succeeds? delete also needs to reject paths outside the configured session root.

Discover, list, search, view, export, and convert from dsh sessions
under $DSH_HOME/sessions (default ~/.dsh/sessions). The store is
read-only: dsh has no documented session import command and its
persistence seam does not delete logs.

Logs are append-only JSONL, Zstandard-framed by default. The first
line is a type:session header; the Common projection rebuilds the
ordered surface (user/message, assistant/message, tool/result) and
leaves packed chunk rows and unknown events in the native body.
…ption

Four review findings on the dsh harness, all confirmed against the official
`@deepseek-ai/dsh-session` sources:

- `surfaceOp` replace endpoints are event seqs, not surface-vector indexes.
  Log-only events sit between surface nodes, so the two diverge immediately
  and compaction shadowed the wrong range. The surface now tracks each node's
  seq and resolves both endpoints by seq, matching `replacementRange` in
  dsh's own `surface.ts`. The previous test masked this by using seqs that
  happened to equal vector indexes.
- Durable `assistant/message` `usage` and `interrupted` were dropped. Map
  `interrupted: true` to `StopReason::Aborted` and translate the `TokenUsage`
  counts, in both directions. `reasoningTokens` has no Common counterpart and
  stays in the native body.
- `std::io::Read` was imported unconditionally but only used by the native
  `decode_zstd`, failing the wasm32 job under `-D warnings`.
- Zstandard frames were split by scanning for frame magic, which also occurs
  inside compressed payloads and checksums. The streaming decoder already
  walks frame structure and continues across boundaries, so decode in one
  pass.

Also fixes an invariant violation in the writer: `tool/result` emitted
`sourceEventSeqs: []`, which dsh's `assertProvenance` rejects — empty arrays
are legal only on `assistant/message`. Verified by feeding a regenerated log
from a real session to the official `foldSurface`, which now passes.
The dsh store pulled zstd in as an unconditional non-wasm dependency.
Codex now declares the same crate as optional behind `codex_zstd`, and
Cargo merges the two declarations, so the unconditional one would turn
zstd on for every native build and make that feature meaningless.

Follow the Codex pattern instead: `dsh_zstd = ["dep:zstd"]`, on by
default. Without it — including the wasm build — `session.jsonl.zstd`
logs fail to load and drop out of discovery, while `session.jsonl` logs
read as before.
dsh 0.2 publishes each format generation as its own file
(`session[.vN].jsonl[.zstd]`) and keeps older generations beside it.
Discovery now picks the highest generation, and the reader understands
v0 through v4 directly: `startSeq`/`endSeq` replace endpoints, the
`system/message` and `developer/message` surface nodes, and tool-role
`tool/result` messages.

Environment context dsh injects as user messages (runtime snapshot,
skill catalog, instruction files, clocks, ...) is no longer projected
into Common. `parentSession` becomes a Fork lineage, or Spawn for a
subagent run.

Checked against sessions recorded by dsh 0.2.0-rc.2, and against a v0
session read both as-is and after the official catalog migrated it to
v4.
dsh ships no session import command, but it does not need one: it finds
sessions by walking its root, so writing one is a matter of reproducing
the layout it scans for.

What makes that exact rather than approximate is dsh's validation. Three
of its checks fail the *entire* listing rather than skipping the one bad
session, which makes each a hard requirement on the writer:

- `assertZstdHeaderFrame` — the first Zstandard frame must decode to
  exactly one line. Compressing the whole log as a single frame still
  round-trips through txcript's own reader, which decodes across frame
  boundaries, so only the official reader rejects it.
- `assertStoredIdentity` — the header's `id` and `cwd` must name the path
  the log was found at. A copy given a new identity would otherwise keep
  pointing at the original's, and take the listing down with it.
- `checkRootEncoding` — one root must not mix `.jsonl` with
  `.jsonl.zstd`. Writing the default encoding into a plaintext root
  breaks that root under *both* configurations, so the root decides the
  encoding and only an empty one falls back to dsh's `zstd` default.

A duplicate id across two project directories is rejected the same way,
so re-saving a session whose cwd changed retires the copy under the old
project directory.

Path derivation follows the official `encodeSegment` and `projectKey`:
separators collapse to `-`, `[A-Za-z0-9._-]` survives, every other UTF-16
code unit becomes `~XXXX`, and the project key is wrapped in `--`.
Escaping over code units rather than scalars is what keeps it injective
over lone surrogates. Escaping the separators is also what contains a
traversing id — `../../evil` becomes the literal directory
`..~002F..~002Fevil`, matching dsh, which encodes rather than rejects.

Verified by running the official persistence backend's own `listArtifacts`
and `loadStored` against a txcript-written root: it lists both a normal
session and a `_no-cwd` one, and decodes all 1257 event records of the
former. Re-introducing the single-frame bug makes that same check fail
with dsh's own "first frame is not exactly one header line". The derived
project-directory names reproduce those of the real local session store
exactly, including its non-ASCII path.

`delete` removes the session directory. dsh's persistence seam has no
delete API, but it keeps no index either — an absent directory is simply
not scanned.
`save` retired copies of the session under other project directories
before writing the replacement, so a write that failed afterwards left
no copy at all. It now encodes the log first, writes it to a temporary
file and renames it into place, and only then removes the stale copies.

`delete` called `remove_dir_all` on whatever path it was handed. It now
requires a session log, resolves the path with `canonicalize`, and
refuses anything that isn't exactly `<root>/<project>/<session>` — the
same guard the grok store applies. Tests cover a failed save under a
new cwd, a session-shaped directory outside the root, a project
directory, a directory with no log, and a symlink escaping the root.
dsh 0.2 migrates older logs when it opens them, but it reads only the
current format with full validation, and a log it has to migrate is
kept beside its successor. Write v4 directly from Common:

- the turn/step lifecycle the official reader checks, with each tool
  call started by `tool/call` and answered by a tool-role `tool/result`;
- an empty `system/message` heading the surface, without which the log
  dsh extends on resume fails its next load;
- dsh's own not-started repair for a call with no result;
- a provider/model on every model source and an empty embedded stream.

A native body keeps its version, and `save` names the file for it
(`session[.vN].jsonl[.zstd]`), retiring other generations beside it.

Converted Claude Code, Codex, and OpenCode sessions pass the official
catalog's strict current-format validation, and a converted session
resumed twice with `dsh --profile headless --session-id`.
@zzyking

zzyking commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Both fixed in c5b491d:

  • save: the replacement is written to a temporary file beside the log and renamed over it. Only after that succeeds does it remove a copy filed under a different project directory (left behind when a session's cwd changes). If the write fails, the temporary file is removed and the previous copy is untouched. There's a test that forces the write to fail and checks the old copy survives.
  • delete: same guard as feat(kimi): make the Kimi Code store writable #32. It requires a session log, canonicalizes the path and root (so symlinks resolve first), and only accepts exactly <root>/<project>/<session>.

Separately, dsh moved on while this was open: 0.2.0-rc.2 writes session format v4, so I've updated the writer to match (a764222). This doesn't depend on our reader. Every log is checked with dsh's own format catalog under its strictest settings (recovery: 'strict', validation: 'current'), and real dsh resumed a converted session end to end:

  • Common sessions are written as v4, using the turn/step lifecycle dsh's reader enforces. Each tool call is recorded as a tool/call, and its answer is a tool-role tool/result. A call that never got a result is closed with dsh's own TOOL_NOT_STARTED repair.
  • The surface starts with an empty system/message. I found this requirement the hard way: without it, the first dsh --profile headless --session-id <id> resume works, but dsh then appends its system prompt mid-surface, and the log it wrote fails its next load. With the head, a converted session resumed twice and passed the strict check after each turn.
  • A native dsh body keeps its version, and the file is named after it (session[.vN].jsonl[.zstd]). dsh rejects a log whose file name and header disagree. A save also removes any other generation left in the directory.
  • I converted Claude Code, Codex, and OpenCode sessions (the largest is about 14k events) and all of them pass the strict check.

The details and provenance are in docs/formats/dsh.md. The branch is still stacked on #19.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants