[experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 - #2159
Open
filimonov wants to merge 30 commits into
Open
[experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6#2159filimonov wants to merge 30 commits into
filimonov wants to merge 30 commits into
Conversation
filimonov
force-pushed
the
feature/antalya-26.6/CAS
branch
from
August 4, 2026 14:15
0ab537f to
9ec80cb
Compare
ReadBufferFromFileView assumed the inner buffer keeps its working buffer across setReadUntilPosition; ReadBufferFromS3 resets it, so getPosition lied and seek logic could re-read a stale decompressed block. Recompute the offset from the inner buffer after every right-bound change. Includes the ReadBufferFromMemory counterpart and gtest batteries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
processException kept retrying transient errors after KILL QUERY; check CurrentThread::get().isQueryCanceled() in the outer retry loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A borrowed child ThreadGroup parents its trackers at the parent group via raw pointers; background work outliving the query produced a use-after-free in parent counters. The child now holds a shared_ptr to the parent group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
addPart/dropPart guarded current_writer only with chassert (a release no-op), so a missing writer meant a null dereference; throw LOGICAL_ERROR instead. Also treats a missing logs_dir as normal for storages that do not materialize empty directories (carries a CAS-related hunk; wired later). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
HTTP 412 on a conditional request is deterministic: never retry it (S3Exception::isPreconditionFailed, RetryStrategy). Adds conditional PUT/COPY (If-Match / If-None-Match) through the client, WriteBufferFromS3 and copyS3File, and the token-conditional operations on S3ObjectStorage. Also carries the copyS3File message_format_string fix (PreformattedMessage instead of a preformatted string) and CAS-facing write-settings plumbing (wired later). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A conditional PUT that is doomed to 412 should not stream its whole body; send Expect: 100-continue above a size threshold and peek the response. Prevents mid-upload connection resets and retry storms on S3-compatible stores. Carries the GCS dialect integration points (wired later). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
GCS XML API needs native GOOG4 signing and x-goog-if-generation-match for generation-safe conditional writes; AWS SigV4 If-Match semantics are not enough, and conditional multipart complete is silently ignored. Adds the signer, the header dialect, and fixed-vector tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Emulate object-store behavior under concurrent removal: a file vanishing between listing and stat is skipped, not a filesystem_error; non-recursive explicit-stack walk with error_code overloads and a symlink guard; fail closed on an embedded-NUL path (AST fuzzer). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Projection sub-parts ride the parent whole-part transaction instead of committing early; read-your-writes (in-flight resolve) becomes part of the IDiskTransaction contract so staged state is visible before commit; clone/freeze/restore wrap the whole part in one transaction; staged operation order is explicit. Mixed-file note: these files also carry the content-addressed capability surface and eager-dispatch branches that are wired by the later CAS integration commits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Single-table SYSTEM commands (SYNC/RESTORE/RESTART/DROP REPLICA, WAIT LOADING PARTS, PREWARM, ...) cast the storage directly and missed tables behind a proxy; unwrap the proxy before the cast. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
clickhouse-disks --query always exited 0 and reported failures only on
stderr, so scripts, cron jobs and CI could not gate on it at all. Record
each command's error code in processQueryText and return it as the
process exit code for non-interactive runs; interactive REPL sessions
keep exiting 0. Within one semicolon-separated batch a later success does
not clear an earlier failure.
Carries the two test fixes the contract exposed: a 2024 typo in
test_disks_app_func ("d/a" instead of "a/d/a" always failed inside the
tool and was swallowed), and test_replicated_table_structure_alter
reading metadata_path from system.tables after DETACH DATABASE (empty
path, the remove never ran, the recovery scenario was never exercised).
Mixed-file note: DisksApp.cpp also carries the cas-* command registration
and CA pool initialization, wired by the later CAS integration commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Core value types of the content-addressed storage subsystem: identifiers, hashes, tokens, namespaces. No dependencies on other CAS layers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
On-wire/on-disk encodings: manifests, ref-log records, GC state, seals, codecs. Depends only on Primitives. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Object-storage access layer: the backend interface, the object-storage adapter, the in-memory backend for tests, capability probing and request control. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Pool identity and runtime: server root, mount lifecycle, pool metadata, the ref ledger and ref protocol (publish/confirm, recovery, snapshots), the part-write transaction (dedup gate, conditional create, promote), staging and plain objects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Part-path parsing and the part-folder access facade over manifests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The garbage-collection round: fold, in-degree settlement, lease and heartbeat, prune, baseline rebuild, ack-floor fencing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
fsck (integrity checking), inspect, and pool-member decommission. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The content-addressed metadata storage, its disk transaction and part staging (the top-level subsystem glue), registration of the content_addressed metadata storage type, capability predicates on IDisk/DiskObjectStorage/IMetadataStorage, the conditional-object-storage API on IObjectStorage, the FileView read-pipeline stage, write-ETag surfacing, cache-over-CA, and the atomic-file-write short-circuit for txn_version.txt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The CAS system logs (definitions, SystemLog registration, Context getters), the CAS mounts system table, and the per-disk GC-health asynchronous metrics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
SYSTEM CONTENT ADDRESSED GARBAGE COLLECTION / GC REBUILD / DROP POOL MEMBER: grammar, AST, interpreter handlers, access checks, and the parser round-trip test. Includes the magic_enum range widening required once ASTSystemQuery::Type outgrew the default range. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
ProfileEvents/CurrentMetrics/AccessType/FailPoint/ServerSettings entries, build wiring, the fetch-by-relink replication protocol extension in DataPartsExchange (same-pool fetch publishes a local ref over shared blobs; gated by pool_uuid, non-CA fetches unchanged), the clickhouse-disks CA commands (inspect, fsck, gc-dryrun, gc-rebuild, drop-member), and server/local entry-point wiring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The full CAS gtest battery: per-layer unit tests, protocol state-machine tests, wiring/assembly tests, and the shared test helpers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
New stateless tests for content-addressed storage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Integration suites: shared pool, replicated relink, GC (sharded and S3), insert fault recovery, lazy-load recovery, file cache, drop pool member, ref snaplog, disks-app; plus shared helpers and compose files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
CA-default stateless/integration lanes in praktika and workflows, the content-addressed default-disk test configs, clickhouse-test support, and tag edits to pre-existing tests (no-content-addressed-storage and friends). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The dedicated documentation set under docs/en/antalya/cas/: architecture (storage layout, manifests and refs, blob protocol, part lifecycle, read path, replication, mounts and leases, namespaces, garbage collection, correctness, design history) and operations (configuration, bucket requirements, monitoring, debugging, troubleshooting, migration). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
User-facing documentation: content-addressed storage in storing-data, the SYSTEM statements, and the three system-table pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
force-pushed
the
feature/antalya-26.6/CAS
branch
from
August 4, 2026 22:09
9ec80cb to
056488b
Compare
filimonov
added a commit
that referenced
this pull request
Aug 5, 2026
…cessor test Upstream commit bfe5988 ("Iceberg type 'time' support") changed the getSimpleType mapping of the Iceberg `time` type from Int64 to Time64(6), but the test kept the old expectation. The mismatch is inherited from altinity/antalya-26.6 and fails `Unit tests (asan_ubsan)` on every PR against that base. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2159&sha=056488b47a0b3f32ea2ebf83d508b697ff4c0c17&name_0=PR&name_1=Unit%20tests%20%28asan_ubsan%29 PR: #2159 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A new opt-in metadata storage type
casfor object-storage disks: MergeTree parts are storedas content-addressed blobs in a shared S3/GCS bucket pool, deduplicated by content hash, with
part manifests and named refs on top — "git for MergeTree". Multiple servers mount the same
pool and share identical data blocks without re-uploading them; same-pool replication fetches
publish a local ref over the shared blobs instead of copying bytes (fetch-by-relink). Garbage
collection runs as a lease-coordinated background round with full audit trail; integrity is
checkable online with
SYSTEM CAS FSCKand theclickhouse-diskscas-*applets.User-facing surface: the
casdisk/metadata-storage type,SYSTEM CAS GC RUN / GC REBUILD / FSCK / DROP POOL MEMBER, system tablessystem.cas_log,system.cas_gc_log,system.cas_mounts, and theclickhouse-diskscommandscas-inspect,cas-fsck,cas-gc-dryrun,cas-gc-rebuild,cas-drop-member. Full documentation ships in this PR underdocs/en/antalya/cas/(architecture and operations), plus the system-table pages.How the series is structured
29 commits, ordered for review; the subsystem lands dark and is switched on by one registration
line.
reviewable on its own:
ReadBufferFromFileViewposition tracking,ReadBufferFromS3retry-after-cancel,
ThreadGroupparent lifetime,MergeTreeDeduplicationLogfail-closednull-writer, S3 conditional writes + 412 no-retry policy,
Expect: 100-continuefor largeconditional uploads, GCS conditional-write dialect and GOOG4 signer,
LocalObjectStoragesnapshot listing semantics, the disk-transaction contract (one logical part = one
transaction, read-your-writes), SYSTEM-on-proxy unwrap, and
clickhouse-disksnon-zero exitcode for failed non-interactive commands.
src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/, bottom-up:Primitives → Formats → Backend → Pool → Parts → Gc → Tools. No layer is reachable untilthe wiring lands.
predicates, the registration line, system logs and tables, SYSTEM commands, registries and
entry points, fetch-by-relink in
DataPartsExchange.on CAS, and the documentation set.
Behavior changes to shared code (reviewers, look here)
Everything CAS-specific is gated behind the
casdisk type; a server without acasdisk getsidentical behavior except for these deliberate fixes:
clickhouse-disks --querynow exits non-zero when a command fails (previously always 0);interactive sessions unaffected. This exposed and fixes two latent test defects
(
test_disks_app_funcpath typo,test_replicated_databasereadingmetadata_pathafterDETACH).pre-existing Iceberg conditional writes).
whole-part transaction and makes staged state readable before commit — behavior-preserving on
local disks, stricter on object-storage metadata.
ReadBufferFromFileViewposition fix also affects the pre-existing packed skip-index reader.Verification
src/Disks/tests/, 134 files) plus stateless and integrationsuites; CI runs a CAS-default-disk lane over the general stateless set.
invariants) has been green on this tree; the harness itself is intentionally not part of this
PR.
and the development branch, restricted to shipped paths, is empty by construction.
Developed with AI assistance (Claude); every commit carries
Co-Authored-ByandSigned-off-by.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added experimental content-addressed storage (CAS) for MergeTree: a new
casmetadata storagetype for object-storage disks that stores parts as content-hash-deduplicated blobs in a shared
S3/GCS pool with manifests and refs, allowing multiple servers to share identical data without
re-uploading it, replicate same-pool parts without transferring bytes, and reclaim unreferenced
data with lease-coordinated garbage collection. Includes
SYSTEM CAS GC RUN / GC REBUILD / FSCK / DROP POOL MEMBER, system tablessystem.cas_log,system.cas_gc_log,system.cas_mounts, andclickhouse-diskscommandscas-inspect,cas-fsck,cas-gc-dryrun,cas-gc-rebuild,cas-drop-member.Documentation entry for user-facing changes
Documentation is included in this pull request:
docs/en/antalya/cas/(architecture andoperations),
docs/en/operations/storing-data.md,docs/en/sql-reference/statements/system.md,and the three system-table pages.
CI/CD Options
Exclude tests:
Regression jobs to run: