From 04204ec9b98923d90ad4e6f489bac25c3f46aa16 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Tue, 8 Sep 2026 15:02:42 -0400 Subject: [PATCH 01/12] Add isolated BFV packed Hamming research experiment --- CHANGELOG.md | 1 + docs/research/bfv-packed-hamming.md | 358 +++++++++++++++++++++ experiments/bfv/README.md | 119 +++++++ experiments/bfv/benchmark.py | 253 +++++++++++++++ experiments/bfv/noise_probe.py | 82 +++++ experiments/bfv/packed_hamming.py | 340 +++++++++++++++++++ experiments/bfv/requirements.txt | 2 + experiments/bfv/results/environment.txt | 8 + experiments/bfv/results/measurements.jsonl | 3 + experiments/bfv/results/noise_probe.jsonl | 2 + experiments/bfv/test_packed_hamming.py | 178 ++++++++++ 11 files changed, 1346 insertions(+) create mode 100644 docs/research/bfv-packed-hamming.md create mode 100644 experiments/bfv/README.md create mode 100644 experiments/bfv/benchmark.py create mode 100644 experiments/bfv/noise_probe.py create mode 100644 experiments/bfv/packed_hamming.py create mode 100644 experiments/bfv/requirements.txt create mode 100644 experiments/bfv/results/environment.txt create mode 100644 experiments/bfv/results/measurements.jsonl create mode 100644 experiments/bfv/results/noise_probe.jsonl create mode 100644 experiments/bfv/test_packed_hamming.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e35be6..fdfef85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ All notable changes to this project will be documented in this file. ### Added +- Offline BFV packed Hamming experiment under `experiments/bfv`, with an isolated public-key server evaluator, correctness tests, a comparison against an exported Paillier Git revision, and a research report. The production Paillier path and dependencies are unchanged. - `xtrace_sdk.x_vec.crypto.device.resolve_device` — shared device-resolution helper used by both Paillier clients and intended for future homomorphic clients with optional GPU backends. - `test_cross_device_interop` — parametrised test covering CPU↔GPU portability for both `PaillierClient` and `PaillierLookupClient`. Verifies that contexts saved on one device load correctly on the other (hash equality), and that ciphertexts produced on either backend round-trip correctly through the other for both encryption and server-side homomorphic add. diff --git a/docs/research/bfv-packed-hamming.md b/docs/research/bfv-packed-hamming.md new file mode 100644 index 0000000..eecbb6a --- /dev/null +++ b/docs/research/bfv-packed-hamming.md @@ -0,0 +1,358 @@ +# BFV for XTrace: repository analysis and first experiment + +Research date: 2026-09-08. Experimental branch: `research/bfv-packed-hamming`, +based on `staging-GPU-paillier-client` at `c8fc6838aec81e2c3a05fbd5c0c9c0e8635beb26`. +The local `main` reference used for comparison is +`0f8522e449d00622e10f4111ac8eb89d52ca577b`. No remote refresh was performed. + +**BFV can substantially reduce the distance-response payload.** The implemented +experiment computes distances under BFV and packs them before returning them. +For 8,192 binary vectors of dimension 512, the measured response fell from +4.26 MB to 127 KB including IDs and framing, a 33.4× reduction. Query plus response +fell by 7.6×. The cost was an approximately 52× larger encrypted index and a +39.6 MB public/evaluation-key bundle. These are local measurements, not a production +service or GPU performance claim. + +Packing still returns all candidate distances. Returning only the top three +requires a separate encrypted comparison and selection protocol. No such protocol +is implemented here. Existing Paillier ciphertexts are not automatically usable +as BFV ciphertexts. + +## What the repository implements + +The repository is the Python **client SDK**, with public arithmetic helpers and +CUDA client extensions. It does not contain the deployed search server. The +experiment therefore supplies a local server evaluator rather than changing the +live API. + +| Area | Responsibility and relevance | +| --- | --- | +| `src/xtrace_sdk/x_vec/inference/embedding.py` | Produces embeddings through Sentence Transformers, Ollama, or OpenAI. `float_2_bin` thresholds each coordinate with `> 0`. Search operates on those binary vectors, not the original floating-point vectors. Provider selection determines where embedding plaintext is processed. | +| `x_vec/data_loaders/loader.py` | AES-encrypts document text, binarizes embeddings, encrypts the binary index through the homomorphic client, and uploads via the integration. Supports per-vector and batch encryption. Metadata is supplied separately. | +| `x_vec/crypto/encryption/paillier.py` | Standard additive Paillier over plaintexts modulo `n`, with ciphertext multiplication modulo `n²` implementing plaintext addition. `key_len` is the bit length of **each prime**. | +| `x_vec/crypto/encryption/paillier_lookup.py` | Custom variant with an alternative decryption exponent `a`, tables for powers of `g` in 8-bit message chunks, and a randomization table. Its parameter/security analysis is separate from this BFV experiment. | +| `x_vec/crypto/{paillier_client,paillier_lookup_client}.py` | Packs bits with zero separators, dispatches CPU/GPU encryption and decryption, serializes keys/configuration, and supplies an offline public-key Hamming-encoding helper. | +| `x_vec/crypto/*_gpu_ext/` | CUDA/GMP/pybind11 implementations using CGBN for big integers, batched modular operations, and decoding/popcount. The lookup implementation also caches tables and exposes decrypt/re-encrypt helpers; those helpers require secret material and cannot simply move to an untrusted server. | +| `x_vec/crypto/device.py` | Chooses the requested CPU/GPU implementation or probes for automatic selection. It changes execution, not the result format or candidate count. | +| `x_vec/crypto/encryption/aes.py`, `crypto/key_provider.py` | AES-256-GCM for contents and protected secret-key storage; scrypt passphrases or AWS KMS envelope keys. BFV does not need to replace content encryption. | +| `x_vec/utils/execution_context.py` | Bundles the homomorphic client and AES/key-provider state, serializes/restores it locally or remotely, and enforces the embedding/key-size constraints. Current supported contexts remain Paillier and Paillier-Lookup. | +| `integrations/xtrace.py` | Async HTTP API for chunks, metadata, contexts, knowledge bases and administration. Queries use base64 ciphertexts in JSON. Distance responses are read completely and decoded as MessagePack or JSON. | +| `x_vec/retrievers/retriever.py` | Encrypts queries, downloads candidate encodings, decrypts/decodes every distance, then calls `np.argsort(... )[:k]`. A second request fetches selected chunks for AES decryption. Optional multiprocessing changes client decoding, not the wire volume. | +| `cli/` | Initialization/environment and cached session state; file chunking; load/upsert/fetch/retrieve and knowledge-base commands; interactive shell. These routes ultimately use the same SDK clients and integration. | +| `x_vec/inference/llm.py` | Optional RAG inference using retrieved plaintext context. Provider choice defines its separate plaintext boundary. | +| Other crypto / `x_mem` | Goldwasser–Micali is a separate XOR-capable implementation; Merkle commitments are a partial utility and do not provide search-result verification. Signatures and `x_mem` are placeholders. | +| `tests/` | Offline encryption, embedding conversion, execution-context/key persistence and CLI tests; separate live API tests; CPU/GPU interoperability tests. | +| `docs/`, `examples/` | Sphinx API/manual pages and a quick-start notebook documenting the same client workflow. | +| `pyproject.toml`, `hatch_build.py`, `build_gpu_binaries.sh`, `.github/workflows/` | Hatch/VCS packaging, CUDA builds and binary wheel tagging, lint/type checks, offline tests, gated GPU/integration tests and release/docs workflows. | + +Paths abbreviated to `x_vec/...` in the table are relative to `src/xtrace_sdk/`. +The existing untracked `benchmarks/paillier_client_matrix.py` is preserved. + +The branch comparison shows the GPU work primarily changes client dispatch, +CUDA implementations, configuration, packaging and tests. The primitive encryption +modules, retriever and integration are identical between the recorded `main` and +starting GPU-branch commits. Their response protocol has not been changed by the +GPU work. + +## The current arithmetic and I/O bottleneck + +For a binary vector `x`, the client effectively encodes a base-four integer: + +```text +P(x) = sum_j x[j] * 4^(d - 1 - j) +``` + +Each base-four digit is represented by the binary pair `0x[j]`. Adding two such +encodings produces digits in `{0, 1, 2}` without carries between coordinates. +The low bit of each digit is `x[j] XOR q[j]`. Counting those low bits after +decryption gives the Hamming distance. + +The server can obtain `Enc(P(x) + P(q))` through Paillier ciphertext multiplication. +It cannot use Paillier's additive operation alone to evaluate the nonlinear bit +extraction and popcount that the SDK currently performs after decryption. +Consequently, the returned encrypted object encodes coordinate-wise sums, not a +single scalar distance already computed on the server. + +`XTraceIntegration.compute_hamming_distances` awaits `res.read()` and parses the +whole response. `Retriever.nn_search_for_ids` then decodes all candidate results +and ranks them. There is no `k` argument in the distance API request. Reducing +`k` from ten to three does not reduce this response. + +The payload consists of one encrypted **result per candidate** (possibly multiple +chunks for larger vectors), not all AES-encrypted documents. Metadata/range +filters can restrict the candidate set first. Only the selected documents are +fetched afterwards. Thus the relevant scaling variable is the number of candidates +after filtering, which can be much smaller than the whole database. + +At the benchmark defaults, `key_len=1024` means approximately 2,048-bit `n`, so +each ciphertext modulo `n²` occupies about 512 binary bytes. GPU execution and +lookup tables retain that ciphertext representation. Streaming the existing +response could overlap download and decryption and reduce memory, but would not +remove these bytes. + +## What BFV adds, and what “put Paillier inside BFV” could mean + +BFV supports exact modular-integer addition and multiplication. With batch +encoding, one ciphertext holds many independent integer slots. SEAL's BFV +batching has `N` slots arranged as two rows of `N/2`; a prime plaintext modulus +congruent to `1 mod 2N` enables that encoding. +[SEAL batch-encoding example](https://github.com/microsoft/SEAL/blob/main/native/examples/2_encoders.cpp) + +For binary vectors, the distance is the low-depth arithmetic expression + +```text +h_i = sum_j (x[i,j] - q[j])² + = sum_j (x[i,j] + q[j] - 2*x[i,j]*q[j]). +``` + +The first form needs one encrypted subtraction and square per packed input, +followed by encrypted sums. Both database and query remain encrypted. If the +plaintext modulus exceeds `d` and the noise budget remains positive, decoding +gives the exact integer distance. This experiment uses `t=65537` and `d<=4096`. + +There are three distinct interpretations of the mentor's suggestion: + +| Interpretation | Assessment | +| --- | --- | +| Encrypt the bytes/limbs of existing Paillier ciphertexts inside BFV | This retains the Paillier representation and adds an encryption layer. It does not itself extract the small distance or make randomized ciphertext bytes compressible. | +| Evaluate Paillier decryption and distance decoding homomorphically under BFV | Possible in principle with appropriately encrypted secret-key material and a sufficiently capable circuit. It requires large-integer modular arithmetic and bit extraction; it is not ordinary BFV key switching. No practical implementation of this conversion was established in this investigation. | +| Compute the distances directly under BFV, then pack results from many input ciphertexts | Implemented here. This is a plausible interpretation of the demo, but cannot be identified as the actual demo without its code. It requires BFV-encrypted index data. | + +These distinctions follow from the two schemes' plaintext representations and +available operations; they are design conclusions, not a claim that all possible +hybrid protocols have been ruled out. An interactive conversion protocol is also +a separate avenue and would need its own trust, leakage and round-trip analysis. +The textbook's BFV key-switching discussion concerns changing the secret key of +an RLWE ciphertext, not a turnkey Paillier-to-BFV conversion. + +## The implemented packing layout + +Let `M` be the number of candidates, `d` the embedding dimension, `N=8192` the +slot count, and `D` the next power of two at least `d`. Define +`B=N/(2D)` lanes in each BFV row and `C=2B` records per input ciphertext. +At `d=512`, `D=512`, `B=8`, and `C=16`. + +Each input tile arranges a BFV row as follows; the second row holds another +eight candidates: + +```text +row 0: [dimension 0 of candidates 0..7] + [dimension 1 of candidates 0..7] + ... + [dimension 511 of candidates 0..7] + +row 1: the same arrangement for candidates 8..15 +``` + +The query repeats each bit across the eight lanes, then repeats that row. +The same one-ciphertext query is reused for every tile. This avoids sending +one full BFV ciphertext per query dimension. + +1. Subtract query from encrypted tile and square slotwise. Relinearize to two + ciphertext polynomials. +2. Rotate rows by `B, 2B, 4B, ...` and add. This sums across dimensions while + keeping different candidate lanes separate. It takes `log2(D)` rotations. +3. Apply a public plaintext mask, keeping only the first `B` distances in each + row and zeroing invalid candidates in the final tile. +4. Merge results from different ciphertexts with rotations and additions. A + binary carry tree uses one rotation per merge and logarithmic intermediate + ciphertext memory. It handles incomplete groups as well as full groups. +5. After up to `D` tiles, the output holds up to `N` distances. Drop unused + coefficient-modulus primes before serialization. + +This reduction uses approximately `ceil(M/C)` input ciphertexts and +`ceil(M/N)` response ciphertexts. For a full 8,192-record, 512-dimensional group, +there are 512 input ciphertexts, 512 squares, 4,608 dimension-sum rotations, +511 packing rotations, and one returned ciphertext. The whole index packet is +still materialized in memory; the logarithmic bound applies to intermediate +arithmetic, not index storage or MessagePack parsing. + +Rows rotate independently in BFV. The code keeps both rows separate and decodes +their known layout back to original record order. It does not assume a flat +8,192-slot circular rotation. This matches SEAL's +[rotation model](https://github.com/microsoft/SEAL/blob/main/native/examples/6_rotation.cpp). + +The server uses fixed, SEAL-validated `TC128` parameters. Final modulus switching +reduces ciphertext size while consuming remaining computation capacity. It happens +only after packing, when no further encrypted arithmetic is needed. +[SEAL's levels example](https://github.com/microsoft/SEAL/blob/main/native/examples/3_levels.cpp) + +`noise_probe.py` demonstrates why simply halving the ring is unsuitable for this +particular circuit and parameter set. In the 4,096-slot trial, coefficient-prime +sizes were `[36,36,37]`: remaining noise was about 21 bits after the square, 13 +after summation, and **zero after masking**. The 8,192-slot version retained ample +budget at that stage. Exact numbers vary with fresh encryption randomness. +This is a failure of that tested configuration, not a proof that every possible +4,096-slot Hamming design fails. The main prototype excludes that configuration. + +## Measurements and interpretation + +The raw [measurement records](../../experiments/bfv/results/measurements.jsonl) +contain seed, dependency versions, source hashes, exact byte counts, phase timings +and the baseline commit. These runs use 512-dimensional synthetic binary vectors, +sequential integer IDs, `k=3`, standard Paillier with 1,024-bit primes, and one +query per newly built index. Encryption randomness is fresh; only the synthetic +plaintext data is seeded. Every decoded distance and selected ID was checked. + +Sizes below use decimal KB/MB. **Response sizes include IDs and MessagePack framing.** +For a full BFV block, the ciphertext itself is about 103 KB; the remainder is +primarily candidate IDs. Real ID widths and transport framing affect the ratio. + +| Candidates | Paillier response | BFV response | Response reduction | Query + response reduction | +| ---: | ---: | ---: | ---: | ---: | +| 256 | 132.7 KB | 103.6 KB | 1.28× | 0.25×: BFV transfers about 4× more | +| 2,048 | 1.065 MB | 109.0 KB | 9.77× | 1.97× | +| 8,192 | 4.259 MB | 127.4 KB | 33.43× | 7.61× | + +The BFV query is approximately 432.6 KB in all these runs; the modeled current +Paillier JSON/base64 query is 798 bytes. Ignoring setup, the first BFV result block +therefore needs roughly a thousand candidates before **bidirectional** byte +savings begin. The much lower download-only crossover is not enough to establish +total I/O savings. + +For 8,192 candidates: + +| Cost | Paillier | BFV | +| --- | ---: | ---: | +| Binary index model | 4.26 MB | 221.44 MB | +| Public/evaluation setup | About 2.5 KB | 39.59 MB | +| Local server evaluation + response serialization | 0.105 s | 20.093 s | +| Local client response decoding + stable top-k | 56.958 s | 0.012 s | + +The BFV index is about 52× larger. Each input ciphertext holds only 16 vectors; +packing the **results** more tightly does not remove the storage needed for the +input coordinates. At this candidate count, about 3.70 MB are saved per query. +Byte-only amortization takes approximately 11 queries for the extra key bundle, +or 70 queries including the extra index. At 2,048 candidates those counts are +approximately 76 and 180. These calculations use binary Paillier index storage; +the actual JSON/base64 ingest protocol adds further overhead. + +These timing figures are illustrative local CPU measurements, not controlled +service benchmarks. They include the prototype's file-based serialization and +input parsing; they exclude actual network transfer, HTTP/TLS, authentication, +embeddings, AES content fetch, and multi-user contention. The large run overlapped +part of the existing test execution, so use isolated repeated runs for timing +decisions. Byte counts and correctness checks are unaffected. The baseline uses +the recorded `main` CPU cryptography and a local reference server operation, +with the same stable client top-k routine on both paths; it is not a replay of +the live service or its exact sorting implementation. + +**Do not infer a speedup over the optimized GPU/lookup branch from this table.** +Its decryption cost may be much smaller than this CPU baseline. BFV adds server +work; at a fast network link that extra work can outweigh the reduced traffic. +A useful next timing comparison is: + +```text +latency = query encryption + server evaluation + client decoding/selection + + query_bytes / upload_bytes_per_second + + response_bytes / download_bytes_per_second + + common round-trip overhead +``` + +Apply that model with the intended GPU server/client hardware, actual candidate +counts, index update rate, and asymmetric link bandwidth. Amortize setup and +index costs separately. The experimental code does not automatically choose a +backend from these estimates. + +## Boundaries and next steps + +The implementation and commands are in the +[experiment README](../../experiments/bfv/README.md). Production source and +dependencies remain unchanged. The generated assets contain measurements only; +no document data, API credentials, ciphertext indexes or secret keys are committed. + +For a usable BFV option, the next steps are: + +1. **Reconcile this layout with Liwen's demo.** Determine whether it used native + BFV distances, a hybrid conversion, or encrypted selection. Its parameters, + trust model, input format and measured bytes matter more than the scheme name. +2. **Benchmark the real workload and hardware.** Compare standard Paillier and + Paillier-Lookup with their effective parameter settings, include GPU decode, + and measure update/storage costs and amortized key transfer. The current + measurements compare configurations, not equal-security cryptosystems. +3. **Introduce a separate BFV index and endpoint.** The existing `EncryptedVector` + and `HammingClientBase` interfaces model per-record integer ciphertext lists. + Packed BFV tiles and multi-distance responses need an explicit, versioned + interface rather than changing the meaning of those existing methods. +4. **Reindex on the client.** Use retained binary vectors, regenerated local + embeddings, or client-side decryption of existing Paillier index encodings. + Keep the AES content and record IDs, and retain a Paillier index for fallback. + An existing Paillier-only knowledge base cannot answer a BFV query as-is. +5. **Implement tile-aware storage, updates and filtering.** Selecting arbitrary + IDs cannot simply relabel packed slots. Metadata/range filters need matching + lane masks and result compaction or prepartitioned indexes. A single-record + update may require rewriting a tile or a separately designed update protocol. + +If the goal is strictly **only three encrypted results on the wire**, add a +separate encrypted top-k experiment after packed distance evaluation. Exact +comparisons are not a native cheap BFV operation: a sorting/selection network +needs comparison circuits plus conditional moves, increasing depth and server +cost. IDs may need multiple limbs because real 64-bit IDs do not fit in one slot +modulo 65,537. Tie handling and whether the server learns selected IDs must be +specified. A final ordinary fetch by client-selected ID reveals an access pattern, +as it does in the current SDK. + +OpenFHE provides comparison/argmin examples through CKKS↔FHEW scheme switching. +That is a concrete direction to investigate, not a drop-in Paillier→BFV or +BFV→top-k conversion. Integer ranking correctness and conversion boundaries would +need testing. [OpenFHE scheme-switching example](https://github.com/openfheorg/openfhe-development/blob/main/src/pke/examples/scheme-switching.cpp) + +The threat model for this prototype is a single key owner and honest evaluation. +The server sees IDs, count and dimension, and receives encrypted vectors/query +plus public evaluation keys. Key-ID/schema checks catch accidental mismatches; +they do not authenticate messages, bind responses to queries, prevent replay or +prove correct computation. Noise and decrypted-result diagnostics remain local. +These protocol limits follow SEAL's documented distinctions around authenticity, +circuit privacy and decryption feedback. +[SEAL security guidance](https://github.com/microsoft/SEAL/blob/main/SECURITY.md) + +There is also a concrete dependency issue before any deployment: Microsoft now +recommends SEAL 4.4.0 or later for fixes involving untrusted inputs and native +memory safety. The convenient TenSEAL 0.3.16 experiment wheel must not be treated +as evidence of an updated native backend. Use a verified, maintained backend +before exposing the evaluator or deserializer to service traffic. +[SEAL update notice](https://github.com/microsoft/SEAL) + +## Reading guide for the provided textbook + +The supplied PDF is Ronny Ko's *The Beginner's Textbook for Fully Homomorphic +Encryption*, arXiv:2503.05136v26, dated 2026-06-02. Its relevant **printed** pages +and sections are: + +| Section | Printed page | Why it matters here | +| --- | ---: | --- | +| D-2.2, Batch Encoding | 137 | Explains SIMD integer slots and encoding. | +| D-2.3, Encryption and Decryption | 139 | Explains the plaintext/noise distinction and correctness conditions. | +| D-2.4–D-2.7, Arithmetic | 140–149 | Covers addition, plaintext masking, ciphertext multiplication and relinearization. | +| D-2.8, Key Switching | 151 | Clarifies that key switching is within the RLWE construction. | +| D-2.9, Slot Rotation | 151–163 | Supplies the operation used for dimension sums and result packing. | +| D-2.11, Bootstrapping | 164 onward | Useful for deeper circuits; unnecessary for this one-square distance circuit. | + +The book's [online BFV chapter](https://fhetextbook.github.io/BFVScheme.html) +is also available. For implementation details, use the SEAL batching, rotation +and modulus-level examples linked at the relevant points above; the low-level +Python bindings are exposed by +[TenSEAL's `sealapi`](https://github.com/OpenMined/TenSEAL/blob/main/tenseal/sealapi/__init__.py). + +## Validation recorded for this branch + +The BFV suite passed all 28 tests, including a separate-process server, full and +partial output groups, both batching rows, dimension padding, exact matches, +complements, ties, randomized encryption, malformed messages, and modulus +compaction. The three benchmark sizes also checked every distance and top-k. + +The final combined run passed **82 tests**: 28 BFV tests and 54 existing CPU/CLI +tests. It excluded the live API modules and the GPU test module: + +```bash +.venv/bin/python -m pytest experiments/bfv tests/cli tests/x_vec \ + --ignore=tests/x_vec/test_hamming_search.py \ + --ignore=tests/x_vec/test_meta_search.py \ + --ignore=tests/x_vec/test_GPU_paillier_clients.py -q +``` + +Ruff and mypy passed for the production source and the typed experiment code. +The initial existing offline regression run had 56 passes and seven GPU failures: +the compiled extensions import, but encryption raises `cudaErrorNoDevice`, and +`nvidia-smi` cannot communicate with the driver. This predates the experiment; +no production source was modified. Live API tests were not run. diff --git a/experiments/bfv/README.md b/experiments/bfv/README.md new file mode 100644 index 0000000..a56285d --- /dev/null +++ b/experiments/bfv/README.md @@ -0,0 +1,119 @@ +# BFV packed Hamming experiment + +This prototype moves exact Hamming-distance arithmetic and result packing to a +server that has no secret key. It returns up to **8,192 encrypted distances per +ciphertext**. The client decrypts those distances and selects top-k. + +It is an offline research experiment using synthetic binary vectors. It does +not implement encrypted server-side top-k or connect to the XTrace API. The +Paillier implementations, `ExecutionContext`, `Retriever`, production dependencies, +and server wire format are unchanged. This directory is outside the SDK wheel. + +The [research report](../../docs/research/bfv-packed-hamming.md) explains the +existing repository, the BFV design, measured tradeoffs, and integration work. + +## Run + +From the repository root, using the existing SDK virtual environment: + +```bash +uv pip install --python .venv/bin/python -r experiments/bfv/requirements.txt +.venv/bin/python -m pytest experiments/bfv -q +.venv/bin/python experiments/bfv/benchmark.py --num-vectors 8192 --paillier-ref main +``` + +The benchmark reads a **local** Git revision, exports its `src/` into a temporary +directory, and starts a subprocess that imports that exact Paillier implementation. +It does not fetch, check out, or modify `main`. Use `--paillier-ref c8fc683` to +compare the starting GPU branch's **CPU** path. It does not benchmark a GPU. + +```bash +# Save every measured phase and byte count; this appends a JSONL record. +.venv/bin/python experiments/bfv/benchmark.py \ + --num-vectors 2048 --repeats 3 --paillier-ref main \ + --output /tmp/xtrace-bfv-measurements.jsonl + +# Reproduce the smaller ring's noise-budget failure, with security still TC128. +.venv/bin/python experiments/bfv/noise_probe.py --degree 4096 +.venv/bin/python experiments/bfv/noise_probe.py --degree 8192 +``` + +The benchmark checks **every** distance and top-k against a plaintext oracle. It +records encryption, server evaluation, client decryption/selection, setup keys, +index storage, query upload, response download, and byte-only amortization. The +JSONL includes the baseline commit and hashes of the experiment source. + +## Minimal client/server example + +Run this Python code from the repository root: + +```python +from experiments.bfv.packed_hamming import BfvClient, BfvServer + +client = BfvClient(dimension=4) +server = BfvServer(client.server_bundle()) # public/evaluation-key bytes only + +index = client.encrypt_index( + [[1, 0, 1, 0], [0, 1, 0, 1], [1, 0, 0, 0]], + ids=[101, 102, 103], +) +query = client.encrypt_query([1, 0, 1, 0]) +response = server.search(index, query) +assert client.top_k(response, k=2) == [(101, 0), (103, 1)] +``` + +`index`, `query`, `response`, and `server_bundle()` are MessagePack `bytes`. +The tests also run the server in a separate process supplied only these public +or encrypted inputs. Secret keys are never serialized by this prototype. +Keys live in the client instance; creating another client creates a different +key and requires a corresponding index. + +`decrypt_distances()` returns `(ID, distance)` in original index order. `top_k()` +orders by distance, breaking ties with the smaller ID. Empty indexes return empty +results; `k=0` returns none and oversized `k` returns all candidates. Dimensions +1 through 4,096 are accepted and padded to a power of two. IDs are unique unsigned +64-bit integers. Embedding floats must be binarized before calling these APIs. + +## Parameters and serialization + +The protocol fixes BFV degree `N=8192`, plaintext modulus `t=65537`, and SEAL's +`BFVDefault(N, TC128)` coefficient moduli. The tested binding yields key-level +prime bit sizes `[43, 43, 44, 44, 44]`; input ciphertexts use four of those primes. +The server drops to one prime after completing the arithmetic and packing. +`server.search(..., compact=False)` retains all four for comparison. + +These are exact integer computations, with one ciphertext multiplication per +input tile. No bootstrapping or approximate arithmetic is used. The client +checks remaining noise budget and the decoded distance range; those checks do +not authenticate server output. `noise_budgets()` is a local diagnostic only. + +TenSEAL 0.3.16 exposes the required SEAL batching, rotations, and modulus switching +through `sealapi`. Its low-level save/load binding takes filenames, so `_save` +and `_load` use temporary files for **ciphertexts and public/evaluation keys**. +Measured serialization time includes that file I/O. No plaintexts or secret keys +pass through these helpers. A service implementation should use memory-based +serialization in a maintained binding or C++. + +**Backend limitation:** Microsoft currently recommends SEAL **4.4.0 or later** +for security fixes involving untrusted inputs and native memory safety. The pinned +TenSEAL wheel used for this experiment is not evidence of a patched backend. Use +this experiment with locally generated inputs; update and verify the native +backend before accepting service traffic. [Microsoft's SEAL notice](https://github.com/microsoft/SEAL) + +## What remains for a service + +Existing Paillier ciphertexts need a client-side migration/reindexing step before +this evaluator can use them. Keep both indexes to offer an operational fallback. +The current SDK cannot send a BFV query to an existing Paillier knowledge base. + +This prototype assumes one key owner and an honest evaluator. IDs, dimension, +and candidate count are visible. Messages have schema and identity checks but +no authentication, query nonce, replay protection, or proof of correct evaluation. +The client must retain decrypted values, errors, and noise diagnostics locally. +SEAL documents these distinctions between encryption, authenticity, and decryption +feedback in its [security guidance](https://github.com/microsoft/SEAL/blob/main/SECURITY.md). + +Server-side encrypted comparison/selection, index persistence, updates, +metadata/range filtering, key lifecycle management, and an actual transport are +future work. The [research report](../../docs/research/bfv-packed-hamming.md) +sets out the proposed boundaries for those changes. diff --git a/experiments/bfv/benchmark.py b/experiments/bfv/benchmark.py new file mode 100644 index 0000000..7893eb7 --- /dev/null +++ b/experiments/bfv/benchmark.py @@ -0,0 +1,253 @@ +"""Measure actual BFV payloads against an isolated local Git revision of Paillier. + +Run from the repository root, for example: + .venv/bin/python experiments/bfv/benchmark.py --num-vectors 8192 --paillier-ref main +No API calls, checkout changes, real documents, credentials, or secret-key exports. +""" + +from __future__ import annotations + +import argparse +import base64 +import hashlib +import heapq +import importlib.metadata +import io +import json +import math +import platform +import random +import statistics +import subprocess +import sys +import tarfile +import tempfile +import time +from collections.abc import Callable +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +import msgpack + + +REPO = Path(__file__).resolve().parents[2] + + +def timed(call: Callable[[], Any]) -> tuple[Any, float]: + start = time.perf_counter() + value = call() + return value, time.perf_counter() - start + + +def dataset(count: int, dimension: int, seed: int) -> tuple[list[list[int]], list[int], list[int]]: + rng = random.Random(seed) + query = [rng.randrange(2) for _ in range(dimension)] + vectors = [[rng.randrange(2) for _ in range(dimension)] for _ in range(count)] + if vectors: + vectors[0] = query[:] + distances = [sum(x != y for x, y in zip(v, query, strict=True)) for v in vectors] + return vectors, query, distances + + +def top_k(distances: list[int], k: int) -> list[tuple[int, int]]: + return heapq.nsmallest(k, enumerate(distances), key=lambda pair: (pair[1], pair[0])) + + +def run_bfv(args: argparse.Namespace) -> dict[str, Any]: + from packed_hamming import BfvClient, BfvServer + + vectors, query, expected = dataset(args.num_vectors, args.dimension, args.seed) + client, keygen_s = timed(lambda: BfvClient(args.dimension)) + public, key_serialize_s = timed(client.server_bundle) + server, server_setup_s = timed(lambda: BfvServer(public)) + index, index_encrypt_s = timed(lambda: client.encrypt_index(vectors)) + runs = [] + for _ in range(args.repeats): + query_payload, query_encrypt_s = timed(lambda: client.encrypt_query(query)) + def evaluate(query_payload: bytes = query_payload) -> bytes: + return server.search(index, query_payload) + + response, server_eval_s = timed(evaluate) + + def decode(response: bytes = response) -> tuple[list[int], list[tuple[int, int]]]: + values = [distance for _, distance in client.decrypt_distances(response)] + return values, top_k(values, args.k) + + (distances, selected), client_decode_select_s = timed(decode) + if distances != expected or selected != top_k(expected, args.k): + raise AssertionError("BFV distances/top-k differ from the plaintext oracle") + packet = msgpack.unpackb(response, raw=False) + runs.append(dict( + query_bytes=len(query_payload), response_bytes=len(response), + response_ciphertext_bytes=sum(map(len, packet["ciphertexts"])), + response_ciphertexts=len(packet["ciphertexts"]), query_encrypt_s=query_encrypt_s, + server_eval_s=server_eval_s, client_decode_select_s=client_decode_select_s, + minimum_noise_budget_bits=min(client.noise_budgets(response), default=None), + )) + return dict( + backend="BFV / TenSEAL 0.3.16 sealapi / CPU", poly_modulus_degree=8192, + plain_modulus=65537, coeff_modulus_bits=[43, 43, 44, 44, 44], + security_setting="SEAL TC128", final_coeff_modulus_count=1, + padded_dimension=client.layout.padded_dimension, + vectors_per_input_ciphertext=client.layout.vectors_per_ciphertext, + key_bundle_bytes=len(public), keygen_s=keygen_s, key_serialize_s=key_serialize_s, + server_setup_s=server_setup_s, index_bytes=len(index), index_encrypt_s=index_encrypt_s, + index_ciphertexts=len(msgpack.unpackb(index, raw=False)["ciphertexts"]), + all_distances_verified=True, top_k=selected, runs=runs, + ) + + +def run_paillier_worker(args: argparse.Namespace) -> dict[str, Any]: + # This subprocess has not imported xtrace_sdk. Prepend the exported revision + # so the installed editable SDK cannot silently substitute the current branch. + sys.path.insert(0, str(Path(args.paillier_source) / "src")) + import gmpy2 + import xtrace_sdk.x_vec.crypto.paillier_client as module + from xtrace_sdk.x_vec.crypto.encryption.paillier import Paillier + + if not Path(module.__file__).resolve().is_relative_to(Path(args.paillier_source).resolve()): + raise RuntimeError("Paillier baseline imported from the wrong revision") + vectors, query, expected = dataset(args.num_vectors, args.dimension, args.seed) + client, keygen_s = timed(lambda: module.PaillierCPU(embed_len=args.dimension, key_len=args.key_len)) + keys = client.keys + + def binary(value: Any) -> bytes: + return int(value).to_bytes((int(value).bit_length() + 7) // 8, "little") + + def encrypt_index() -> bytes: + return msgpack.packb([[[binary(c) for c in client.encrypt(v)], i] + for i, v in enumerate(vectors)], use_bin_type=True) + + index, index_encrypt_s = timed(encrypt_index) + runs = [] + for _ in range(args.repeats): + def encrypt_query() -> bytes: + return json.dumps(dict( + kb_id="benchmark", context_id="0" * 64, + query=[base64.b64encode(binary(c)).decode("ascii") for c in client.encrypt(query)], + ), separators=(",", ":")).encode() + + query_payload, query_encrypt_s = timed(encrypt_query) + + def evaluate(query_payload: bytes = query_payload) -> bytes: + encrypted_query = [gmpy2.mpz(int.from_bytes(base64.b64decode(c), "little")) + for c in json.loads(query_payload)["query"]] + results = [] + for ciphers, record_id in msgpack.unpackb(index, raw=False): + values = [gmpy2.mpz(int.from_bytes(c, "little")) for c in ciphers] + result = [binary(Paillier.add(x, q, keys["pk"])) + for x, q in zip(values, encrypted_query, strict=True)] + results.append([result, record_id]) + # The current integration expects server rows [ciphertext_chunks, ID]. + return msgpack.packb(results, use_bin_type=True) + + response, server_eval_s = timed(evaluate) + + def decode(response: bytes = response) -> tuple[list[int], list[tuple[int, int]]]: + values = [client.decode_hamming_client(ciphers) + for ciphers, _ in msgpack.unpackb(response, raw=False)] + return values, top_k(values, args.k) + + (distances, selected), client_decode_select_s = timed(decode) + if distances != expected or selected != top_k(expected, args.k): + raise AssertionError("Paillier distances/top-k differ from the plaintext oracle") + rows = msgpack.unpackb(response, raw=False) + runs.append(dict( + query_bytes=len(query_payload), response_bytes=len(response), + response_ciphertext_bytes=sum(len(c) for row, _ in rows for c in row), + response_ciphertexts=sum(len(row) for row, _ in rows), + query_encrypt_s=query_encrypt_s, server_eval_s=server_eval_s, + client_decode_select_s=client_decode_select_s, + )) + return dict( + backend="PaillierCPU from exported Git revision", key_len_prime_bits=args.key_len, + actual_modulus_bits=int(keys["pk"]["n"]).bit_length(), + key_bundle_bytes=len(client.stringify_pk().encode()), keygen_s=keygen_s, + index_bytes=len(index), index_encrypt_s=index_encrypt_s, + index_format="binary MessagePack model; production JSON/base64 ingest is larger", + all_distances_verified=True, top_k=selected, runs=runs, + ) + + +def baseline(args: argparse.Namespace, commit: str) -> dict[str, Any]: + archive = subprocess.check_output(["git", "archive", "--format=tar", commit, "src"], cwd=REPO) + with tempfile.TemporaryDirectory(prefix="xtrace-paillier-reference-") as directory: + with tarfile.open(fileobj=io.BytesIO(archive)) as tar: + tar.extractall(directory, filter="data") + command = [sys.executable, str(Path(__file__).resolve()), "--paillier-source", directory, + "--num-vectors", str(args.num_vectors), "--dimension", str(args.dimension), + "--seed", str(args.seed), "--key-len", str(args.key_len), + "--repeats", str(args.repeats), "--k", str(args.k)] + result = subprocess.run(command, capture_output=True, text=True, check=True) + return json.loads(result.stdout) + + +def median_runs(result: dict[str, Any]) -> dict[str, float]: + return {name: statistics.median(run[name] for run in result["runs"]) + for name in ["query_bytes", "response_bytes", "query_encrypt_s", "server_eval_s", "client_decode_select_s"]} + + +def compare(bfv: dict[str, Any], paillier: dict[str, Any]) -> dict[str, Any]: + b, p = median_runs(bfv), median_runs(paillier) + b_roundtrip, p_roundtrip = b["query_bytes"] + b["response_bytes"], p["query_bytes"] + p["response_bytes"] + saved = p_roundtrip - b_roundtrip + extra_keys = max(0, bfv["key_bundle_bytes"] - paillier["key_bundle_bytes"]) + extra_index = max(0, bfv["index_bytes"] - paillier["index_bytes"]) + return dict( + response_reduction_factor=p["response_bytes"] / b["response_bytes"], + query_plus_response_reduction_factor=p_roundtrip / b_roundtrip, + bytes_saved_per_query=saved, + queries_to_amortize_extra_keys=math.ceil(extra_keys / saved) if saved > 0 else None, + queries_to_amortize_extra_keys_and_index=math.ceil((extra_keys + extra_index) / saved) if saved > 0 else None, + bfv_medians=b, paillier_medians=p, + note="Byte-only amortization; excludes HTTP/TLS and content fetch. Timings are local CPU simulation, not production service measurements. Same stable client top-k used for both.", + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--num-vectors", type=int, default=2048) + parser.add_argument("--dimension", type=int, default=512) + parser.add_argument("--key-len", type=int, default=1024, help="Paillier prime bit length, not modulus bit length") + parser.add_argument("--seed", type=int, default=1337) + parser.add_argument("--k", type=int, default=3) + parser.add_argument("--repeats", type=int, default=1) + parser.add_argument("--paillier-ref", default="main", help="Local Git revision; exported without checkout changes") + parser.add_argument("--paillier-source", help=argparse.SUPPRESS) + parser.add_argument("--output", type=Path, help="Append this run's full measurements as one JSONL record") + args = parser.parse_args() + if args.num_vectors < 1 or args.repeats < 1 or args.k < 0: + parser.error("num-vectors and repeats must be positive; k must be nonnegative") + if not 1 <= args.dimension <= 4096 or args.dimension >= args.key_len or args.key_len < 1024: + parser.error("Require 1 <= dimension <= 4096, dimension < key-len, and key-len >= 1024") + if args.paillier_source: + print(json.dumps(run_paillier_worker(args))) + return + commit = subprocess.check_output( + ["git", "rev-parse", "--verify", "--end-of-options", args.paillier_ref + "^{commit}"], + cwd=REPO, text=True, + ).strip() + print(f"BFV: {args.num_vectors} vectors, dimension {args.dimension}", file=sys.stderr, flush=True) + bfv = run_bfv(args) + print(f"Paillier baseline: {commit}", file=sys.stderr, flush=True) + paillier = baseline(args, commit) + result: dict[str, Any] = dict( + recorded_at_utc=datetime.now(UTC).isoformat(), platform=platform.platform(), + python=platform.python_version(), + packages={name: importlib.metadata.version(name) for name in ["tenseal", "gmpy2", "msgpack"]}, + implementation_sha256=hashlib.sha256(Path(__file__).with_name("packed_hamming.py").read_bytes()).hexdigest(), + benchmark_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + baseline_ref=args.paillier_ref, baseline_commit=commit, + num_vectors=args.num_vectors, dimension=args.dimension, seed=args.seed, repeats=args.repeats, k=args.k, + ) + result.update(bfv=bfv, paillier=paillier, comparison=compare(bfv, paillier)) + if args.output: + args.output.parent.mkdir(parents=True, exist_ok=True) + with args.output.open("a") as stream: + stream.write(json.dumps(result) + "\n") + print(json.dumps(result, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/experiments/bfv/noise_probe.py b/experiments/bfv/noise_probe.py new file mode 100644 index 0000000..a30886f --- /dev/null +++ b/experiments/bfv/noise_probe.py @@ -0,0 +1,82 @@ +"""Check one tile's noise budget without weakening SEAL's TC128 security setting. + +This diagnostic deliberately permits the 4096-slot configuration excluded by +the main prototype. It stops on exhaustion, rather than accepting wrong distances. +All keys/data stay local. Run separately from timing benchmarks. +""" + +import argparse +import json +import random +from typing import Any + +from tenseal import sealapi as seal + + +def probe(degree: int, dimension: int, seed: int) -> dict[str, Any]: + if dimension < 1 or dimension > degree // 2 or dimension & (dimension - 1): + raise ValueError("dimension must be a power of two in [1, degree / 2]") + row_size, lanes = degree // 2, degree // (2 * dimension) + params = seal.EncryptionParameters(seal.SCHEME_TYPE.BFV) + params.set_poly_modulus_degree(degree) + params.set_plain_modulus(65537) + params.set_coeff_modulus(seal.CoeffModulus.BFVDefault(degree, seal.SEC_LEVEL_TYPE.TC128)) + context = seal.SEALContext(params, True, seal.SEC_LEVEL_TYPE.TC128) + if not context.parameters_set(): + raise ValueError(context.parameters_error_message()) + keygen = seal.KeyGenerator(context) + pk, relin, galois = seal.PublicKey(), seal.RelinKeys(), seal.GaloisKeys() + keygen.create_public_key(pk) + keygen.create_relin_keys(relin) + steps = [lanes << i for i in range(dimension.bit_length() - 1)] + if steps: + keygen.create_galois_keys([pow(3, step, 2 * degree) for step in steps], galois) + encryptor, encoder = seal.Encryptor(context, pk), seal.BatchEncoder(context) + evaluator, decryptor = seal.Evaluator(context), seal.Decryptor(context, keygen.secret_key()) + rng = random.Random(seed) + query = [rng.randrange(2) for _ in range(dimension)] + vectors = [[rng.randrange(2) for _ in range(dimension)] for _ in range(2 * lanes)] + + def encrypt(values: list[int]) -> Any: + plain, encrypted = seal.Plaintext(), seal.Ciphertext() + encoder.encode(values, plain) + encryptor.encrypt(plain, encrypted) + return encrypted + + q = encrypt([query[j] for _row in range(2) for j in range(dimension) for _lane in range(lanes)]) + x = encrypt([vectors[row * lanes + lane][j] for row in range(2) for j in range(dimension) for lane in range(lanes)]) + budgets = {"fresh": decryptor.invariant_noise_budget(x)} + result = seal.Ciphertext() + evaluator.sub(x, q, result) + evaluator.square_inplace(result) + evaluator.relinearize_inplace(result, relin) + budgets["square"] = decryptor.invariant_noise_budget(result) + for step in steps: + rotated = seal.Ciphertext() + evaluator.rotate_rows(result, step, galois, rotated) + evaluator.add_inplace(result, rotated) + budgets["sum"] = decryptor.invariant_noise_budget(result) + mask = seal.Plaintext() + encoder.encode([int(i % row_size < lanes) for i in range(degree)], mask) + evaluator.multiply_plain_inplace(result, mask) + budgets["mask"] = decryptor.invariant_noise_budget(result) + correct = None + if budgets["mask"] > 0: + decoded = seal.Plaintext() + decryptor.decrypt(result, decoded) + slots = encoder.decode_uint64(decoded) + expected = [sum(x != q for x, q in zip(v, query, strict=True)) for v in vectors] + correct = slots[:lanes] + slots[row_size:row_size + lanes] == expected + return dict(degree=degree, dimension=dimension, seed=seed, security="TC128", + coeff_modulus_bits=[modulus.bit_count() for modulus in params.coeff_modulus()], + noise_budget_bits=budgets, distances_correct=correct, + status="noise_exhausted" if budgets["mask"] <= 0 else "ok" if correct else "incorrect") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--degree", type=int, choices=[4096, 8192], default=4096) + parser.add_argument("--dimension", type=int, default=512) + parser.add_argument("--seed", type=int, default=1337) + args = parser.parse_args() + print(json.dumps(probe(args.degree, args.dimension, args.seed), indent=2)) diff --git a/experiments/bfv/packed_hamming.py b/experiments/bfv/packed_hamming.py new file mode 100644 index 0000000..12309a4 --- /dev/null +++ b/experiments/bfv/packed_hamming.py @@ -0,0 +1,340 @@ +"""Offline BFV research prototype; independent of the production SDK/API. + +Both inputs are encrypted. The server receives only serialized evaluation keys, +an encrypted index, and an encrypted query. It returns packed exact distances; +top-k selection still happens on the client. See README.md for the protocol, +layout, costs, and limits. This is not an authenticated production transport. +""" + +from __future__ import annotations + +import hashlib +import heapq +import tempfile +from collections.abc import Sequence +from dataclasses import dataclass +from numbers import Integral +from pathlib import Path +from typing import Any + +import msgpack + +try: + from tenseal import sealapi as seal +except ImportError as exc: + raise ImportError("Install experiments/bfv/requirements.txt to run the BFV experiment") from exc + + +PROTOCOL = "xtrace-bfv-tiled-v1" +SLOTS = 8192 +PLAIN_MODULUS = 65537 + + +@dataclass(frozen=True) +class Layout: + """Fixed SEAL TC128 parameters, with zero padding for arbitrary dimensions.""" + + dimension: int = 512 + + def __post_init__(self) -> None: + if type(self.dimension) is not int or not 1 <= self.dimension <= SLOTS // 2: + raise ValueError(f"dimension must be an integer in [1, {SLOTS // 2}]") + + @property + def padded_dimension(self) -> int: + return 1 << (self.dimension - 1).bit_length() + + @property + def lanes(self) -> int: + return SLOTS // 2 // self.padded_dimension + + @property + def vectors_per_ciphertext(self) -> int: + return 2 * self.lanes + + def result_slot(self, position: int) -> int: + """Map a candidate's position within a response block to its BFV slot.""" + tile, lane = divmod(position, self.vectors_per_ciphertext) + row, column = divmod(lane, self.lanes) + return row * (SLOTS // 2) + tile * self.lanes + column + + +def _context() -> Any: + params = seal.EncryptionParameters(seal.SCHEME_TYPE.BFV) + params.set_poly_modulus_degree(SLOTS) + params.set_plain_modulus(PLAIN_MODULUS) + params.set_coeff_modulus(seal.CoeffModulus.BFVDefault(SLOTS, seal.SEC_LEVEL_TYPE.TC128)) + context = seal.SEALContext(params, True, seal.SEC_LEVEL_TYPE.TC128) + if not context.parameters_set() or not context.first_context_data().qualifiers().using_batching: + raise ValueError(context.parameters_error_message()) + return context + + +def _save(obj: Any) -> bytes: + # TenSEAL 0.3.16's low-level SEAL binding only exposes file-based save/load. + # Only ciphertexts and public/evaluation keys pass through here, never secrets. + with tempfile.TemporaryDirectory(prefix="xtrace-bfv-") as directory: + path = Path(directory) / "object.seal" + obj.save(str(path)) + return path.read_bytes() + + +def _load(cls: Any, context: Any, payload: object) -> Any: + if not isinstance(payload, bytes) or not payload: + raise ValueError("Expected nonempty serialized SEAL bytes") + with tempfile.TemporaryDirectory(prefix="xtrace-bfv-") as directory: + path = Path(directory) / "object.seal" + path.write_bytes(payload) + obj = cls() + obj.load(context, str(path)) + return obj + + +def _pack(kind: str, layout: Layout, key_id: str, **fields: Any) -> bytes: + return msgpack.packb( + dict(protocol=PROTOCOL, kind=kind, dimension=layout.dimension, key_id=key_id, **fields), + use_bin_type=True, + ) + + +def _unpack(payload: bytes, kind: str) -> dict[str, Any]: + if not isinstance(payload, bytes): + raise ValueError("Protocol messages must be bytes") + try: + packet = msgpack.unpackb(payload, raw=False) + except (ValueError, TypeError, msgpack.ExtraData) as exc: + raise ValueError("Invalid MessagePack packet") from exc + if not isinstance(packet, dict) or packet.get("protocol") != PROTOCOL or packet.get("kind") != kind: + raise ValueError(f"Expected a {PROTOCOL} {kind} packet") + dimension = packet.get("dimension") + if type(dimension) is not int: + raise ValueError("Missing or invalid dimension") + Layout(dimension) + if not isinstance(packet.get("key_id"), str) or len(packet["key_id"]) != 64: + raise ValueError("Invalid key identifier") + return packet + + +def _check_identity(packet: dict[str, Any], layout: Layout, key_id: str) -> None: + if packet["dimension"] != layout.dimension or packet["key_id"] != key_id: + raise ValueError("Packet belongs to a different key or dimension") + + +def _ids(values: Any) -> list[int]: + if not isinstance(values, (list, tuple)): + raise ValueError("IDs must be a list or tuple") + if any(type(value) is not int or not 0 <= value < 2**64 for value in values): + raise ValueError("IDs must be unsigned 64-bit integers") + if len(set(values)) != len(values): + raise ValueError("IDs must be unique") + return list(values) + + +def _ciphertexts(packet: dict[str, Any], expected: int) -> list[bytes]: + values = packet.get("ciphertexts") + if not isinstance(values, list) or len(values) != expected: + raise ValueError(f"Expected {expected} ciphertexts") + if any(not isinstance(value, bytes) or not value for value in values): + raise ValueError("Ciphertexts must be nonempty bytes") + return values + + +def _vector(values: Sequence[int], dimension: int) -> list[int]: + if len(values) != dimension: + raise ValueError(f"Expected a {dimension}-dimensional binary vector") + if any(not isinstance(value, Integral) or value not in (0, 1) for value in values): + raise ValueError("Vector entries must be binary integers") + return [int(value) for value in values] + + +class BfvClient: + """Owns the BFV secret key. Construct a separate server from server_bundle().""" + + def __init__(self, dimension: int = 512) -> None: + self.layout = Layout(dimension) + self.context = _context() + generator = seal.KeyGenerator(self.context) + self._secret_key = generator.secret_key() + self._public_key = seal.PublicKey() + self._relin_keys = seal.RelinKeys() + generator.create_public_key(self._public_key) + generator.create_relin_keys(self._relin_keys) + steps = [self.layout.lanes << i for i in range(self.layout.padded_dimension.bit_length() - 1)] + # Pass explicit Galois elements: the binding has ambiguous list[int] + # overloads for rotation steps versus elements. SEAL's generator is 3. + elements = sorted({pow(3, step % (SLOTS // 2), 2 * SLOTS) for step in steps + [-v for v in steps]}) + self._galois_keys = seal.GaloisKeys() if elements else None + if elements: + generator.create_galois_keys(elements, self._galois_keys) + self._public_bytes = _save(self._public_key) + self.key_id = hashlib.sha256(self._public_bytes).hexdigest() + self._encoder = seal.BatchEncoder(self.context) + self._encryptor = seal.Encryptor(self.context, self._public_key) + self._decryptor = seal.Decryptor(self.context, self._secret_key) + + def server_bundle(self) -> bytes: + """Export public and evaluation keys only. Cache this once per key/layout.""" + return _pack( + "keys", self.layout, self.key_id, public_key=self._public_bytes, + relin_keys=_save(self._relin_keys), + galois_keys=_save(self._galois_keys) if self._galois_keys is not None else b"", + ) + + def _encrypt(self, slots: list[int]) -> bytes: + plain, encrypted = seal.Plaintext(), seal.Ciphertext() + self._encoder.encode(slots, plain) + self._encryptor.encrypt(plain, encrypted) + return _save(encrypted) + + def encrypt_query(self, vector: Sequence[int]) -> bytes: + bits = _vector(vector, self.layout.dimension) + padded = bits + [0] * (self.layout.padded_dimension - len(bits)) + slots = [bit for _row in range(2) for bit in padded for _lane in range(self.layout.lanes)] + return _pack("query", self.layout, self.key_id, ciphertexts=[self._encrypt(slots)]) + + def encrypt_index(self, vectors: Sequence[Sequence[int]], ids: Sequence[int] | None = None) -> bytes: + record_ids = _ids(list(range(len(vectors))) if ids is None else list(ids)) + if len(record_ids) != len(vectors): + raise ValueError("IDs and vectors must have the same length") + ciphertexts = [] + width = self.layout.vectors_per_ciphertext + for start in range(0, len(vectors), width): + slots = [0] * SLOTS + for lane, vector in enumerate(vectors[start:start + width]): + bits = _vector(vector, self.layout.dimension) + row, column = divmod(lane, self.layout.lanes) + for dim, bit in enumerate(bits): + slots[row * (SLOTS // 2) + dim * self.layout.lanes + column] = bit + ciphertexts.append(self._encrypt(slots)) + return _pack("index", self.layout, self.key_id, ids=record_ids, ciphertexts=ciphertexts) + + def decrypt_distances(self, response: bytes) -> list[tuple[int, int]]: + packet = _unpack(response, "distances") + _check_identity(packet, self.layout, self.key_id) + ids = _ids(packet.get("ids")) + ciphertexts = _ciphertexts(packet, (len(ids) + SLOTS - 1) // SLOTS) + results = [] + for block, payload in enumerate(ciphertexts): + encrypted = _load(seal.Ciphertext, self.context, payload) + if self._decryptor.invariant_noise_budget(encrypted) <= 0: + raise ValueError("BFV noise budget exhausted; distances cannot be trusted") + plain = seal.Plaintext() + self._decryptor.decrypt(encrypted, plain) + slots = self._encoder.decode_uint64(plain) + for position, record_id in enumerate(ids[block * SLOTS:(block + 1) * SLOTS]): + distance = slots[self.layout.result_slot(position)] + if not 0 <= distance <= self.layout.dimension: + raise ValueError("Decrypted distance is outside the valid Hamming range") + results.append((record_id, distance)) + return results + + def top_k(self, response: bytes, k: int = 3) -> list[tuple[int, int]]: + """Return (ID, distance), ordered by distance and then ID to break ties.""" + if type(k) is not int or k < 0: + raise ValueError("k must be a nonnegative integer") + return heapq.nsmallest(k, self.decrypt_distances(response), key=lambda item: (item[1], item[0])) + + def noise_budgets(self, response: bytes) -> list[int]: + """Local experiment diagnostic. Never send decryption/noise feedback to a server.""" + packet = _unpack(response, "distances") + _check_identity(packet, self.layout, self.key_id) + ids = _ids(packet.get("ids")) + return [ + self._decryptor.invariant_noise_budget(_load(seal.Ciphertext, self.context, payload)) + for payload in _ciphertexts(packet, (len(ids) + SLOTS - 1) // SLOTS) + ] + + +class BfvServer: + """Public evaluator reconstructed solely from bytes, with no decryptor or secret key.""" + + def __init__(self, public_bundle: bytes) -> None: + packet = _unpack(public_bundle, "keys") + self.layout = Layout(packet["dimension"]) + self.key_id = packet["key_id"] + public_bytes = packet.get("public_key") + if not isinstance(public_bytes, bytes) or hashlib.sha256(public_bytes).hexdigest() != self.key_id: + raise ValueError("Public key fingerprint mismatch") + self.context = _context() + _load(seal.PublicKey, self.context, public_bytes) + self._relin_keys = _load(seal.RelinKeys, self.context, packet.get("relin_keys")) + self._galois_keys = ( + _load(seal.GaloisKeys, self.context, packet.get("galois_keys")) + if self.layout.padded_dimension > 1 else None + ) + self._evaluator = seal.Evaluator(self.context) + self._encoder = seal.BatchEncoder(self.context) + + def _input(self, payload: bytes) -> Any: + encrypted = _load(seal.Ciphertext, self.context, payload) + if encrypted.size() != 2 or encrypted.parms_id() != self.context.first_parms_id(): + raise ValueError("Expected a fresh, size-2 BFV input ciphertext") + return encrypted + + def _rotate(self, encrypted: Any, step: int) -> Any: + result = seal.Ciphertext() + self._evaluator.rotate_rows(encrypted, step, self._galois_keys, result) + return result + + def _tile_distances(self, encrypted: Any, query: Any, count: int) -> Any: + ev = self._evaluator + result = seal.Ciphertext() + # For bits, (x - q)^2 = x XOR q. Exactly one ciphertext multiplication. + ev.sub(encrypted, query, result) + ev.square_inplace(result) + ev.relinearize_inplace(result, self._relin_keys) + step = self.layout.lanes + while step < SLOTS // 2: + ev.add_inplace(result, self._rotate(result, step)) + step *= 2 + # Keep one distance per valid candidate and erase all duplicated sums, + # including dummy candidates in an incomplete final tile. + mask = [0] * SLOTS + for lane in range(count): + row, column = divmod(lane, self.layout.lanes) + mask[row * (SLOTS // 2) + column] = 1 + plain_mask = seal.Plaintext() + self._encoder.encode(mask, plain_mask) + ev.multiply_plain_inplace(result, plain_mask) + return result + + def _merge(self, left: Any, right: Any, left_tiles: int) -> Any: + self._evaluator.add_inplace(left, self._rotate(right, -left_tiles * self.layout.lanes)) + return left + + def search(self, encrypted_index: bytes, encrypted_query: bytes, *, compact: bool = True) -> bytes: + """Compute every distance, packing up to 8192 into each returned ciphertext. + + compact=False retains the computation modulus for a serialization/noise + comparison. Both variants have identical plaintext results. + """ + index = _unpack(encrypted_index, "index") + query_packet = _unpack(encrypted_query, "query") + _check_identity(index, self.layout, self.key_id) + _check_identity(query_packet, self.layout, self.key_id) + ids = _ids(index.get("ids")) + width = self.layout.vectors_per_ciphertext + tiles = _ciphertexts(index, (len(ids) + width - 1) // width) + query = self._input(_ciphertexts(query_packet, 1)[0]) + output = [] + for group_start in range(0, len(tiles), self.layout.padded_dimension): + # Binary carry tree: logarithmic working ciphertext memory; each + # merge uses one power-of-two rotation, including incomplete groups. + pending: list[tuple[Any, int]] = [] + for tile_index in range(group_start, min(len(tiles), group_start + self.layout.padded_dimension)): + count = min(width, len(ids) - tile_index * width) + current = self._tile_distances(self._input(tiles[tile_index]), query, count) + span = 1 + while pending and pending[-1][1] == span: + left, _ = pending.pop() + current = self._merge(left, current, span) + span *= 2 + pending.append((current, span)) + current, _ = pending.pop() + while pending: + left, span = pending.pop() + current = self._merge(left, current, span) + if compact: + self._evaluator.mod_switch_to_inplace(current, self.context.last_parms_id()) + output.append(_save(current)) + return _pack("distances", self.layout, self.key_id, ids=ids, ciphertexts=output) diff --git a/experiments/bfv/requirements.txt b/experiments/bfv/requirements.txt new file mode 100644 index 0000000..2033481 --- /dev/null +++ b/experiments/bfv/requirements.txt @@ -0,0 +1,2 @@ +# Install alongside the existing SDK environment. Does not change production dependencies. +tenseal==0.3.16 diff --git a/experiments/bfv/results/environment.txt b/experiments/bfv/results/environment.txt new file mode 100644 index 0000000..02c07f7 --- /dev/null +++ b/experiments/bfv/results/environment.txt @@ -0,0 +1,8 @@ +Recorded: 2026-09-08 +CPU model: AMD Ryzen 7 5800X 8-Core Processor +Logical CPUs: 16 +Python: 3.12.3 (main, Jun 19 2026, 12:46:00) [GCC 13.3.0] +Platform: Linux-7.0.0-30-generic-x86_64-with-glibc2.39 +SEAL serialization header version: 4.1 +GPU: nvidia-smi could not communicate with the driver; CUDA encryption raised cudaErrorNoDevice. +Timing: single illustrative local CPU runs; the 8192-vector run overlapped part of regression testing. diff --git a/experiments/bfv/results/measurements.jsonl b/experiments/bfv/results/measurements.jsonl new file mode 100644 index 0000000..2554265 --- /dev/null +++ b/experiments/bfv/results/measurements.jsonl @@ -0,0 +1,3 @@ +{"recorded_at_utc": "2026-09-08T18:57:47.896135+00:00", "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", "python": "3.12.3", "packages": {"tenseal": "0.3.16", "gmpy2": "2.3.0", "msgpack": "1.1.2"}, "implementation_sha256": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01", "benchmark_sha256": "cc266c2fce17905071f2d6e5a926f2e9eccfa426c136d67b4783d5d3bf09259e", "baseline_ref": "main", "baseline_commit": "0f8522e449d00622e10f4111ac8eb89d52ca577b", "num_vectors": 256, "dimension": 512, "seed": 1337, "repeats": 1, "k": 3, "bfv": {"backend": "BFV / TenSEAL 0.3.16 sealapi / CPU", "poly_modulus_degree": 8192, "plain_modulus": 65537, "coeff_modulus_bits": [43, 43, 44, 44, 44], "security_setting": "SEAL TC128", "final_coeff_modulus_count": 1, "padded_dimension": 512, "vectors_per_input_ciphertext": 16, "key_bundle_bytes": 39586409, "keygen_s": 0.15007656998932362, "key_serialize_s": 0.22080954513512552, "server_setup_s": 0.14368783519603312, "index_bytes": 6919721, "index_encrypt_s": 0.12106319214217365, "index_ciphertexts": 16, "all_distances_verified": true, "top_k": [[0, 0], [3, 229], [29, 229]], "runs": [{"query_bytes": 432612, "response_bytes": 103649, "response_ciphertext_bytes": 103109, "response_ciphertexts": 1, "query_encrypt_s": 0.003877946175634861, "server_eval_s": 0.6047730781137943, "client_decode_select_s": 0.0012841750867664814, "minimum_noise_budget_bits": 19}]}, "paillier": {"backend": "PaillierCPU from exported Git revision", "key_len_prime_bits": 1024, "actual_modulus_bits": 2048, "key_bundle_bytes": 2502, "keygen_s": 0.3994619259610772, "index_bytes": 132735, "index_encrypt_s": 2.618243511999026, "index_format": "binary MessagePack model; production JSON/base64 ingest is larger", "all_distances_verified": true, "top_k": [[0, 0], [3, 229], [29, 229]], "runs": [{"query_bytes": 798, "response_bytes": 132733, "response_ciphertext_bytes": 131066, "response_ciphertexts": 256, "query_encrypt_s": 0.010259301168844104, "server_eval_s": 0.0018750668969005346, "client_decode_select_s": 1.752753498032689}]}, "comparison": {"response_reduction_factor": 1.280600874103947, "query_plus_response_reduction_factor": 0.24900375003962621, "bytes_saved_per_query": -402730, "queries_to_amortize_extra_keys": null, "queries_to_amortize_extra_keys_and_index": null, "bfv_medians": {"query_bytes": 432612, "response_bytes": 103649, "query_encrypt_s": 0.003877946175634861, "server_eval_s": 0.6047730781137943, "client_decode_select_s": 0.0012841750867664814}, "paillier_medians": {"query_bytes": 798, "response_bytes": 132733, "query_encrypt_s": 0.010259301168844104, "server_eval_s": 0.0018750668969005346, "client_decode_select_s": 1.752753498032689}, "note": "Byte-only amortization; excludes HTTP/TLS and content fetch. Timings are local CPU simulation, not production service measurements. Same stable client top-k used for both."}} +{"recorded_at_utc": "2026-09-08T18:51:35.770931+00:00", "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", "python": "3.12.3", "packages": {"tenseal": "0.3.16", "gmpy2": "2.3.0", "msgpack": "1.1.2"}, "implementation_sha256": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01", "benchmark_sha256": "cc266c2fce17905071f2d6e5a926f2e9eccfa426c136d67b4783d5d3bf09259e", "baseline_ref": "main", "baseline_commit": "0f8522e449d00622e10f4111ac8eb89d52ca577b", "num_vectors": 2048, "dimension": 512, "seed": 1337, "repeats": 1, "k": 3, "bfv": {"backend": "BFV / TenSEAL 0.3.16 sealapi / CPU", "poly_modulus_degree": 8192, "plain_modulus": 65537, "coeff_modulus_bits": [43, 43, 44, 44, 44], "security_setting": "SEAL TC128", "final_coeff_modulus_count": 1, "padded_dimension": 512, "vectors_per_input_ciphertext": 16, "key_bundle_bytes": 39587405, "keygen_s": 0.15069749485701323, "key_serialize_s": 0.23010723409242928, "server_setup_s": 0.1461251568980515, "index_bytes": 55360252, "index_encrypt_s": 1.0557792391628027, "index_ciphertexts": 128, "all_distances_verified": true, "top_k": [[0, 0], [1897, 215], [1048, 221]], "runs": [{"query_bytes": 432595, "response_bytes": 108996, "response_ciphertext_bytes": 103080, "response_ciphertexts": 1, "query_encrypt_s": 0.0042644229251891375, "server_eval_s": 4.98025533487089, "client_decode_select_s": 0.0035327188670635223, "minimum_noise_budget_bits": 19}]}, "paillier": {"backend": "PaillierCPU from exported Git revision", "key_len_prime_bits": 1024, "actual_modulus_bits": 2048, "key_bundle_bytes": 2502, "keygen_s": 0.06504316814243793, "index_bytes": 1064564, "index_encrypt_s": 21.3114642330911, "index_format": "binary MessagePack model; production JSON/base64 ingest is larger", "all_distances_verified": true, "top_k": [[0, 0], [1897, 215], [1048, 221]], "runs": [{"query_bytes": 798, "response_bytes": 1064565, "response_ciphertext_bytes": 1048562, "response_ciphertexts": 2048, "query_encrypt_s": 0.010317141888663173, "server_eval_s": 0.014922830043360591, "client_decode_select_s": 14.258555201115087}]}, "comparison": {"response_reduction_factor": 9.767009798524716, "query_plus_response_reduction_factor": 1.967098788569234, "bytes_saved_per_query": 523772, "queries_to_amortize_extra_keys": 76, "queries_to_amortize_extra_keys_and_index": 180, "bfv_medians": {"query_bytes": 432595, "response_bytes": 108996, "query_encrypt_s": 0.0042644229251891375, "server_eval_s": 4.98025533487089, "client_decode_select_s": 0.0035327188670635223}, "paillier_medians": {"query_bytes": 798, "response_bytes": 1064565, "query_encrypt_s": 0.010317141888663173, "server_eval_s": 0.014922830043360591, "client_decode_select_s": 14.258555201115087}, "note": "Byte-only amortization; excludes HTTP/TLS and content fetch. Timings are local CPU simulation, not production service measurements. Same stable client top-k used for both."}} +{"recorded_at_utc": "2026-09-08T18:50:00.610543+00:00", "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", "python": "3.12.3", "packages": {"tenseal": "0.3.16", "gmpy2": "2.3.0", "msgpack": "1.1.2"}, "implementation_sha256": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01", "benchmark_sha256": "cc266c2fce17905071f2d6e5a926f2e9eccfa426c136d67b4783d5d3bf09259e", "baseline_ref": "main", "baseline_commit": "0f8522e449d00622e10f4111ac8eb89d52ca577b", "num_vectors": 8192, "dimension": 512, "seed": 1337, "repeats": 1, "k": 3, "bfv": {"backend": "BFV / TenSEAL 0.3.16 sealapi / CPU", "poly_modulus_degree": 8192, "plain_modulus": 65537, "coeff_modulus_bits": [43, 43, 44, 44, 44], "security_setting": "SEAL TC128", "final_coeff_modulus_count": 1, "padded_dimension": 512, "vectors_per_input_ciphertext": 16, "key_bundle_bytes": 39585298, "keygen_s": 0.1499418148305267, "key_serialize_s": 0.2340137348510325, "server_setup_s": 0.1582595359068364, "index_bytes": 221444236, "index_encrypt_s": 4.394628831883892, "index_ciphertexts": 512, "all_distances_verified": true, "top_k": [[0, 0], [7316, 213], [1897, 215]], "runs": [{"query_bytes": 432633, "response_bytes": 127406, "response_ciphertext_bytes": 103058, "response_ciphertexts": 1, "query_encrypt_s": 0.004074418917298317, "server_eval_s": 20.092529614223167, "client_decode_select_s": 0.011931942077353597, "minimum_noise_budget_bits": 19}]}, "paillier": {"backend": "PaillierCPU from exported Git revision", "key_len_prime_bits": 1024, "actual_modulus_bits": 2047, "key_bundle_bytes": 2502, "keygen_s": 0.43998923199251294, "index_bytes": 4259287, "index_encrypt_s": 84.98682740307413, "index_format": "binary MessagePack model; production JSON/base64 ingest is larger", "all_distances_verified": true, "top_k": [[0, 0], [7316, 213], [1897, 215]], "runs": [{"query_bytes": 798, "response_bytes": 4259284, "response_ciphertext_bytes": 4194129, "response_ciphertexts": 8192, "query_encrypt_s": 0.010357261169701815, "server_eval_s": 0.10501185082830489, "client_decode_select_s": 56.9584474961739}]}, "comparison": {"response_reduction_factor": 33.43079603786321, "query_plus_response_reduction_factor": 7.60675952924707, "bytes_saved_per_query": 3700043, "queries_to_amortize_extra_keys": 11, "queries_to_amortize_extra_keys_and_index": 70, "bfv_medians": {"query_bytes": 432633, "response_bytes": 127406, "query_encrypt_s": 0.004074418917298317, "server_eval_s": 20.092529614223167, "client_decode_select_s": 0.011931942077353597}, "paillier_medians": {"query_bytes": 798, "response_bytes": 4259284, "query_encrypt_s": 0.010357261169701815, "server_eval_s": 0.10501185082830489, "client_decode_select_s": 56.9584474961739}, "note": "Byte-only amortization; excludes HTTP/TLS and content fetch. Timings are local CPU simulation, not production service measurements. Same stable client top-k used for both."}} diff --git a/experiments/bfv/results/noise_probe.jsonl b/experiments/bfv/results/noise_probe.jsonl new file mode 100644 index 0000000..b3a6b7b --- /dev/null +++ b/experiments/bfv/results/noise_probe.jsonl @@ -0,0 +1,2 @@ +{"degree": 4096, "dimension": 512, "seed": 1337, "security": "TC128", "coeff_modulus_bits": [36, 36, 37], "noise_budget_bits": {"fresh": 49, "square": 21, "sum": 13, "mask": 0}, "distances_correct": null, "status": "noise_exhausted"} +{"degree": 8192, "dimension": 512, "seed": 1337, "security": "TC128", "coeff_modulus_bits": [43, 43, 44, 44, 44], "noise_budget_bits": {"fresh": 150, "square": 121, "sum": 113, "mask": 96}, "distances_correct": true, "status": "ok"} diff --git a/experiments/bfv/test_packed_hamming.py b/experiments/bfv/test_packed_hamming.py new file mode 100644 index 0000000..cb426eb --- /dev/null +++ b/experiments/bfv/test_packed_hamming.py @@ -0,0 +1,178 @@ +"""Real cryptographic tests; no XTrace account, network, or GPU required.""" + +import random +import subprocess +import sys +from pathlib import Path + +import msgpack +import pytest + +pytest.importorskip("tenseal", reason="Install experiments/bfv/requirements.txt") + +from packed_hamming import BfvClient, BfvServer, Layout, SLOTS + + +def vectors_and_query(count: int, dimension: int) -> tuple[list[list[int]], list[int]]: + rng = random.Random(404 + dimension + count) + query = [rng.randrange(2) for _ in range(dimension)] + vectors = [[rng.randrange(2) for _ in range(dimension)] for _ in range(count)] + if count: + vectors[0] = query[:] + if count > 1: + vectors[1] = [1 - bit for bit in query] + return vectors, query + + +def oracle(vectors: list[list[int]], query: list[int], ids: list[int]) -> list[tuple[int, int]]: + return [(record_id, sum(x != y for x, y in zip(vector, query, strict=True))) + for record_id, vector in zip(ids, vectors, strict=True)] + + +def rewrite(payload: bytes, **changes: object) -> bytes: + packet = msgpack.unpackb(payload, raw=False) + packet.update(changes) + return msgpack.packb(packet, use_bin_type=True) + + +@pytest.fixture(scope="module") +def default_pair() -> tuple[BfvClient, BfvServer]: + client = BfvClient() + return client, BfvServer(client.server_bundle()) + + +@pytest.fixture(scope="module") +def default_case(default_pair: tuple[BfvClient, BfvServer]) -> tuple[bytes, list[tuple[int, int]]]: + client, server = default_pair + vectors, query = vectors_and_query(45, 512) + vectors[-1] = query[:] + ids = list(reversed(range(1000, 1045))) + response = server.search(client.encrypt_index(vectors, ids), client.encrypt_query(query)) + return response, oracle(vectors, query, ids) + + +def test_default_exact_distances_and_stable_top_k(default_pair, default_case) -> None: + client, _ = default_pair + response, expected = default_case + assert client.decrypt_distances(response) == expected + assert client.top_k(response) == sorted(expected, key=lambda pair: (pair[1], pair[0]))[:3] + assert client.top_k(response, 0) == [] + assert len(client.top_k(response, 100)) == 45 + assert min(client.noise_budgets(response)) >= 10 + + +@pytest.mark.parametrize("dimension", [1, 3, 513, 4096]) +def test_dimensions_with_padding_and_extremes(dimension: int) -> None: + client = BfvClient(dimension) + server = BfvServer(client.server_bundle()) + vectors, query = vectors_and_query(9, dimension) + response = server.search(client.encrypt_index(vectors), client.encrypt_query(query)) + assert client.decrypt_distances(response) == oracle(vectors, query, list(range(9))) + + +@pytest.fixture(scope="module") +def small_dimension_pair() -> tuple[BfvClient, BfvServer]: + client = BfvClient(4) + return client, BfvServer(client.server_bundle()) + + +@pytest.mark.parametrize("count", [0, 1, 1025, 2049, SLOTS - 1, SLOTS, SLOTS + 1]) +def test_row_tile_and_response_boundaries(small_dimension_pair, count: int) -> None: + client, server = small_dimension_pair + vectors, query = vectors_and_query(count, 4) + response = server.search(client.encrypt_index(vectors), client.encrypt_query(query)) + assert client.decrypt_distances(response) == oracle(vectors, query, list(range(count))) + assert len(msgpack.unpackb(response, raw=False)["ciphertexts"]) == (count + SLOTS - 1) // SLOTS + + +@pytest.mark.parametrize("bit", [0, 1]) +def test_uniform_vectors_and_zero_distance(default_pair, bit: int) -> None: + client, server = default_pair + query = [bit] * 512 + vectors = [query[:], [1 - bit] * 512] * 10 + response = server.search(client.encrypt_index(vectors), client.encrypt_query(query)) + assert client.decrypt_distances(response) == oracle(vectors, query, list(range(20))) + + +def test_modulus_compaction_preserves_results(default_pair) -> None: + client, server = default_pair + vectors, query = vectors_and_query(65, 512) + index, encrypted_query = client.encrypt_index(vectors), client.encrypt_query(query) + full = server.search(index, encrypted_query, compact=False) + compact = server.search(index, encrypted_query) + assert client.decrypt_distances(full) == client.decrypt_distances(compact) + assert len(compact) < len(full) / 3 + assert min(client.noise_budgets(full)) > min(client.noise_budgets(compact)) > 0 + + +def test_serialized_server_in_separate_process(default_pair, tmp_path: Path) -> None: + client, _ = default_pair + vectors, query = vectors_and_query(19, 512) + public = client.server_bundle() + packet = msgpack.unpackb(public, raw=False) + assert set(packet) == {"protocol", "kind", "dimension", "key_id", "public_key", "relin_keys", "galois_keys"} + for name, payload in [("keys", public), ("index", client.encrypt_index(vectors)), + ("query", client.encrypt_query(query))]: + (tmp_path / name).write_bytes(payload) + subprocess.run( + [sys.executable, "-c", + "from pathlib import Path; from packed_hamming import BfvServer; " + "import sys; p=Path(sys.argv[1]); " + "s=BfvServer((p/'keys').read_bytes()); " + "(p/'response').write_bytes(s.search((p/'index').read_bytes(), (p/'query').read_bytes()))", + str(tmp_path)], + cwd=Path(__file__).parent, check=True, capture_output=True, timeout=60, + ) + assert client.decrypt_distances((tmp_path / "response").read_bytes()) == oracle(vectors, query, list(range(19))) + + +def test_encryption_is_randomized(default_pair) -> None: + client, _ = default_pair + vector = [0, 1] * 256 + assert client.encrypt_query(vector) != client.encrypt_query(vector) + assert client.encrypt_index([vector]) != client.encrypt_index([vector]) + + +def test_mismatched_keys_and_malformed_messages(default_pair, default_case) -> None: + client, server = default_pair + response, _ = default_case + query = client.encrypt_query([0] * 512) + index = client.encrypt_index([[0] * 512]) + with pytest.raises(ValueError, match="different key"): + server.search(index, rewrite(query, key_id="0" * 64)) + with pytest.raises(ValueError, match="different key"): + client.decrypt_distances(rewrite(response, dimension=511)) + with pytest.raises(ValueError, match="ciphertexts"): + server.search(rewrite(index, ciphertexts=[]), query) + with pytest.raises(ValueError, match="unique"): + client.decrypt_distances(rewrite(response, ids=[1, 1])) + with pytest.raises(ValueError, match="Expected"): + client.decrypt_distances(query) + with pytest.raises(ValueError): + client.decrypt_distances(b"not a MessagePack response") + + +@pytest.mark.parametrize("vector", [[0], [2] * 512, [-1] * 512, [0.5] * 512]) +def test_invalid_vectors(default_pair, vector: list[int]) -> None: + client, _ = default_pair + with pytest.raises(ValueError): + client.encrypt_query(vector) + with pytest.raises(ValueError): + client.encrypt_index([vector]) + + +def test_invalid_ids_and_k(default_pair, default_case) -> None: + client, _ = default_pair + response, _ = default_case + for ids in [[1, 1], [-1, 2], [1, 2**64], [True, 2], [1]]: + with pytest.raises(ValueError): + client.encrypt_index([[0] * 512, [1] * 512], ids) + for k in [-1, 1.5, True]: + with pytest.raises(ValueError, match="k must"): + client.top_k(response, k) + + +@pytest.mark.parametrize("dimension", [0, -1, 4097, 1.5, True]) +def test_invalid_dimensions(dimension: int) -> None: + with pytest.raises(ValueError): + Layout(dimension) From cf454e5eb6e053822501645bd44c6f23aa832a19 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Thu, 10 Sep 2026 16:38:19 -0400 Subject: [PATCH 02/12] Add native GMP BFV scheme and packed Hamming client --- CHANGELOG.md | 1 + benchmarks/bfv_client_matrix.py | 315 ++++++++ benchmarks/results/native_bfv_1024.json | 177 +++++ docs/api_rst/xtrace_sdk.crypto.encryption.rst | 8 + docs/api_rst/xtrace_sdk.crypto.rst | 8 + docs/index.rst | 7 + docs/research/native-bfv.md | 273 +++++++ src/xtrace_sdk/x_vec/crypto/__init__.py | 5 +- src/xtrace_sdk/x_vec/crypto/bfv_client.py | 334 +++++++++ .../x_vec/crypto/encryption/__init__.py | 3 +- src/xtrace_sdk/x_vec/crypto/encryption/bfv.py | 703 ++++++++++++++++++ .../crypto/encryption/homomorphic_base.py | 4 +- src/xtrace_sdk/x_vec/utils/xtrace_types.py | 47 ++ tests/x_vec/test_bfv_client.py | 211 ++++++ tests/x_vec/test_bfv_encryption.py | 239 ++++++ 15 files changed, 2331 insertions(+), 4 deletions(-) create mode 100644 benchmarks/bfv_client_matrix.py create mode 100644 benchmarks/results/native_bfv_1024.json create mode 100644 docs/research/native-bfv.md create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_client.py create mode 100644 src/xtrace_sdk/x_vec/crypto/encryption/bfv.py create mode 100644 tests/x_vec/test_bfv_client.py create mode 100644 tests/x_vec/test_bfv_encryption.py diff --git a/CHANGELOG.md b/CHANGELOG.md index fdfef85..1552783 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ All notable changes to this project will be documented in this file. ### Added +- Native GMP BFV primitives in `crypto/encryption/bfv.py` and a `BFVClient` in `crypto/bfv_client.py`, with CRT batching, ciphertext multiplication/relinearization, public-key Hamming evaluation, packed distance responses, and terminal modulus compaction. Includes CPU benchmarks and tests against plaintext algebra and an optional SEAL oracle. This is an experimental local implementation; the existing Paillier clients and SEAL research example are preserved. - Offline BFV packed Hamming experiment under `experiments/bfv`, with an isolated public-key server evaluator, correctness tests, a comparison against an exported Paillier Git revision, and a research report. The production Paillier path and dependencies are unchanged. - `xtrace_sdk.x_vec.crypto.device.resolve_device` — shared device-resolution helper used by both Paillier clients and intended for future homomorphic clients with optional GPU backends. - `test_cross_device_interop` — parametrised test covering CPU↔GPU portability for both `PaillierClient` and `PaillierLookupClient`. Verifies that contexts saved on one device load correctly on the other (hash equality), and that ciphertexts produced on either backend round-trip correctly through the other for both encryption and server-side homomorphic add. diff --git a/benchmarks/bfv_client_matrix.py b/benchmarks/bfv_client_matrix.py new file mode 100644 index 0000000..5acc275 --- /dev/null +++ b/benchmarks/bfv_client_matrix.py @@ -0,0 +1,315 @@ +#!/usr/bin/env python3 +"""CPU BFV/Paillier Hamming comparison with separate setup, compute and wire sizes. + +Run from the checkout: .venv/bin/python benchmarks/bfv_client_matrix.py --help +Only synthetic plaintexts are seeded. Encryption always uses scheme randomness. +No network, CUDA, SEAL or XTrace service is required. +""" + +from __future__ import annotations + +import argparse +import gc +import hashlib +import json +import platform +import random +import statistics +import subprocess +import sys +import time +from collections.abc import Callable, Sequence +from datetime import UTC, datetime +from pathlib import Path +from typing import Any, TypeVar + +import gmpy2 +import msgpack + +REPO_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO_ROOT / "src")) + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient # noqa: E402 +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV # noqa: E402 +from xtrace_sdk.x_vec.crypto.encryption.paillier import Paillier # noqa: E402 +from xtrace_sdk.x_vec.crypto.paillier_client import PaillierClient # noqa: E402 +from xtrace_sdk.x_vec.crypto.paillier_lookup_client import PaillierLookupClient # noqa: E402 + +T = TypeVar("T") +VARIANTS = ("bfv-packed", "bfv-individual", "paillier-cpu", "paillier-lookup-cpu") + + +def time_call(fn: Callable[[], T]) -> tuple[float, T]: + gc.collect() + start = time.perf_counter() + result = fn() + return time.perf_counter() - start, result + + +def packet(ciphers: Sequence[Sequence[int]], count: int) -> bytes: + """Same MessagePack framing for both schemes; big integers become LE bytes. + + Candidate IDs are implicit positions, so no IDs, content, HTTP or TLS bytes + are included. Native BFV's per-ciphertext header remains in the measurement. + """ + return msgpack.packb( + { + "vector_count": count, + "ciphertexts": [ + [int(v).to_bytes(max(1, (int(v).bit_length() + 7) // 8), "little") for v in row] + for row in ciphers + ], + }, + use_bin_type=True, + ) + + +def unpack_packet(payload: bytes) -> list[list[int]]: + data = msgpack.unpackb(payload, raw=False) + return [[int.from_bytes(v, "little") for v in row] for row in data["ciphertexts"]] + + +def make_data( + count: int, dimension: int, seed: int +) -> tuple[list[list[int]], list[int], list[int]]: + rng = random.Random(seed) + query = [rng.getrandbits(1) for _ in range(dimension)] + vectors = [[rng.getrandbits(1) for _ in range(dimension)] for _ in range(count)] + vectors[0] = query[:] + if count > 1: + vectors[1] = [1 - bit for bit in query] + if count > 2: + vectors[-1] = query[:] + expected = [sum(a != b for a, b in zip(v, query, strict=True)) for v in vectors] + return vectors, query, expected + + +def benchmark( + variant: str, + args: argparse.Namespace, + vectors: list[list[int]], + query: list[int], + expected: list[int], +) -> dict[str, Any]: + n = len(vectors) + timings: dict[str, float] = {} + print(f"{variant}: key generation and public server setup", file=sys.stderr, flush=True) + native = variant.startswith("bfv-") + client: Any + if native: + timings["key_generation_s"], client = time_call( + lambda: BFVClient( + embed_len=args.embed_len, + poly_modulus_degree=args.poly_modulus_degree, + plain_modulus=args.plain_modulus, + coeff_modulus_bits=args.coeff_modulus_bits, + decomposition_bits=args.decomposition_bits, + response_modulus_bits=args.response_modulus_bits, + device="cpu", + ) + ) + elif variant == "paillier-cpu": + timings["key_generation_s"], client = time_call( + lambda: PaillierClient(args.embed_len, args.key_len, device="cpu") + ) + else: + timings["key_generation_s"], client = time_call( + lambda: PaillierLookupClient(args.embed_len, args.key_len, args.alpha_len, device="cpu") + ) + timings["public_key_export_s"], public_json = time_call(client.stringify_pk) + public_key_bytes = len(public_json.encode("utf-8")) + config = json.loads(client.stringify_config()) + if native: + + def load_server() -> BFVClient: + result = BFVClient(skip_key_gen=True) + result.load_config(config) + result.load_stringified_keys(public_json) + assert result.keys is None + return result + + timings["public_key_import_s"], server = time_call(load_server) + else: + # Paillier server multiplication needs n^2 only; retain a public dict, + # not a Paillier client containing the decryption key. + timings["public_key_import_s"], server_pk = time_call(lambda: json.loads(public_json)) + modulus = gmpy2.mpz(server_pk["n_squared"]) + public_json = "" + + packed = variant == "bfv-packed" + print(f"{variant}: encrypting {n} vectors and query", file=sys.stderr, flush=True) + timings["index_encrypt_s"], index = time_call( + lambda: client.encrypt_vec_packed(vectors) if packed else client.encrypt_vec_batch(vectors) + ) + timings["query_encrypt_s"], encrypted_query = time_call(lambda: client.encrypt_vec_one(query)) + timings["index_serialize_s"], index_wire = time_call(lambda: packet(index, n)) + timings["query_serialize_s"], query_wire = time_call(lambda: packet([encrypted_query], 1)) + # Feed actual deserialized packets to the evaluator, and do the same for the + # response. Byte totals are measured lengths, not object-size estimates. + timings["index_deserialize_s"], server_index = time_call(lambda: unpack_packet(index_wire)) + timings["query_deserialize_s"], server_query = time_call(lambda: unpack_packet(query_wire)[0]) + + def evaluate() -> list[list[int]]: + if packed: + return server.encode_hamming_server_packed( + server_query, server_index, n, compact=not args.no_compact + ) + if native: + return [ + server.encode_hamming_server(server_query, ct, compact=not args.no_compact) + for ct in server_index + ] + return [ + [int(gmpy2.mpz(a) * b % modulus) for a, b in zip(server_query, row, strict=True)] + for row in server_index + ] + + print(f"{variant}: public server evaluation", file=sys.stderr, flush=True) + timings["server_compute_s"], response = time_call(evaluate) + timings["response_serialize_s"], response_wire = time_call(lambda: packet(response, n)) + timings["response_deserialize_s"], client_response = time_call( + lambda: unpack_packet(response_wire) + ) + timings["client_decode_s"], distances = time_call( + lambda: ( + client.decode_hamming_client_packed(client_response, n) + if packed + else client.decode_hamming_client_batch(client_response) + ) + ) + if distances != expected: + raise AssertionError( + f"{variant}: decrypted distances differ from the plaintext Hamming oracle" + ) + top3 = sorted(range(n), key=lambda i: (distances[i], i))[:3] + assert top3 == sorted(range(n), key=lambda i: (expected[i], i))[:3] + budgets = [] + if native: + budgets = [ + BFV.noise_budget(BFV.ciphertext_from_ints(ct, client.public_key), client.keys) + for ct in client_response + ] + sizes = { + "encrypted_index_bytes": len(index_wire), + "public_keys_bytes": public_key_bytes, + "query_bytes": len(query_wire), + "response_bytes": len(response_wire), + "query_plus_response_bytes": len(query_wire) + len(response_wire), + } + print( + f"{variant}: verified {n} distances; response {len(response_wire):,} bytes", + file=sys.stderr, + flush=True, + ) + return { + "variant": variant, + "vectors": n, + "embed_len": args.embed_len, + "config": config, + "compact": native and not args.no_compact, + "timings": timings, + "sizes": sizes, + "index_ciphertexts": len(index), + "response_ciphertexts": len(response), + "minimum_response_noise_budget_bits": min(budgets) if budgets else None, + "top3": [{"index": i, "distance": distances[i]} for i in top3], + "correct": True, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--num-vectors", type=int, default=256) + parser.add_argument("--embed-len", type=int, default=512) + parser.add_argument( + "--key-len", + type=int, + default=1024, + help="Paillier prime bit length, as in the existing clients", + ) + parser.add_argument("--alpha-len", type=int, default=50) + parser.add_argument("--poly-modulus-degree", type=int, default=8192) + parser.add_argument("--plain-modulus", type=int, default=65537) + parser.add_argument("--coeff-modulus-bits", type=int, default=180) + parser.add_argument("--decomposition-bits", type=int, default=30) + parser.add_argument("--response-modulus-bits", type=int, default=50) + parser.add_argument("--no-compact", action="store_true") + parser.add_argument("--seed", type=int, default=1337) + parser.add_argument( + "--repeats", + type=int, + default=1, + help="Independent runs; each includes fresh key generation", + ) + parser.add_argument( + "--variants", + default="bfv-packed,paillier-cpu", + help="Comma-separated: " + ", ".join(VARIANTS), + ) + parser.add_argument("--json-out", type=Path) + args = parser.parse_args() + variants = [v.strip() for v in args.variants.split(",")] + if args.num_vectors < 1 or args.embed_len < 1 or args.repeats < 1: + parser.error("num-vectors, embed-len and repeats must be positive") + if any(v not in VARIANTS for v in variants): + parser.error("unknown variant") + vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) + revision = subprocess.run( + ["git", "rev-parse", "HEAD"], cwd=REPO_ROOT, capture_output=True, text=True, check=True + ).stdout.strip() + dirty = bool( + subprocess.run( + ["git", "status", "--porcelain"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip() + ) + output: dict[str, Any] = { + "environment": { + "utc": datetime.now(UTC).isoformat(), + "python": platform.python_version(), + "platform": platform.platform(), + "gmpy2": gmpy2.version(), + "gmp": gmpy2.mp_version(), + "revision": revision, + "working_tree_dirty": dirty, + }, + "command": sys.argv, + "measurement_notes": "CPU synthetic experiment; no equivalent-security claim. Keys are public JSON including BFV evaluation keys. Index/query/response use common MessagePack framing with LE integer bytes and implicit IDs. No network/HTTP/TLS, secret-key, plaintext-content or AES-content bytes. Timings are separate from serialization; no concurrent test/benchmark processes intended.", + "seed": args.seed, + "runs": [], + "source_sha256": { + str(path.relative_to(REPO_ROOT)): hashlib.sha256(path.read_bytes()).hexdigest() + for path in [ + Path(__file__).resolve(), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto").glob("*client.py"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption").glob("*.py"), + REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", + ] + }, + } + for repeat in range(args.repeats): + for variant in variants: + row = benchmark(variant, args, vectors, query, expected) + row["repeat"] = repeat + output["runs"].append(row) + gc.collect() + output["median_timings"] = { + v: { + metric: statistics.median( + row["timings"][metric] for row in output["runs"] if row["variant"] == v + ) + for metric in output["runs"][0]["timings"] + } + for v in variants + } + print(json.dumps(output, indent=2)) + if args.json_out: + args.json_out.parent.mkdir(parents=True, exist_ok=True) + args.json_out.write_text(json.dumps(output, indent=2) + "\n") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/results/native_bfv_1024.json b/benchmarks/results/native_bfv_1024.json new file mode 100644 index 0000000..d8e98eb --- /dev/null +++ b/benchmarks/results/native_bfv_1024.json @@ -0,0 +1,177 @@ +{ + "environment": { + "utc": "2026-09-10T20:27:09.027921+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "04204ec9b98923d90ad4e6f489bac25c3f46aa16", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_client_matrix.py", + "--num-vectors", + "1024", + "--json-out", + "benchmarks/results/native_bfv_1024.json" + ], + "measurement_notes": "CPU synthetic experiment; no equivalent-security claim. Keys are public JSON including BFV evaluation keys. Index/query/response use common MessagePack framing with LE integer bytes and implicit IDs. No network/HTTP/TLS, secret-key, plaintext-content or AES-content bytes. Timings are separate from serialization; no concurrent test/benchmark processes intended.", + "seed": 1337, + "runs": [ + { + "variant": "bfv-packed", + "vectors": 1024, + "embed_len": 512, + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "response_modulus_bits": 50, + "device": "cpu" + }, + "compact": true, + "timings": { + "key_generation_s": 4.32266186689958, + "public_key_export_s": 0.7880964139476418, + "public_key_import_s": 0.853526117047295, + "index_encrypt_s": 6.748511495068669, + "query_encrypt_s": 0.1036944561637938, + "index_serialize_s": 0.038064548978582025, + "query_serialize_s": 0.0002170950174331665, + "index_deserialize_s": 0.022240569116547704, + "query_deserialize_s": 0.00021161511540412903, + "server_compute_s": 108.29552126303315, + "response_serialize_s": 0.0003346470184624195, + "response_deserialize_s": 6.959214806556702e-05, + "client_decode_s": 0.0412755839060992 + }, + "sizes": { + "encrypted_index_bytes": 23598496, + "public_keys_bytes": 85602812, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "index_ciphertexts": 64, + "response_ciphertexts": 1, + "minimum_response_noise_budget_bits": 26, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "correct": true, + "repeat": 0 + }, + { + "variant": "paillier-cpu", + "vectors": 1024, + "embed_len": 512, + "config": { + "embed_len": 512, + "key_len": 1024 + }, + "compact": false, + "timings": { + "key_generation_s": 0.5261988120619208, + "public_key_export_s": 4.5150984078645706e-05, + "public_key_import_s": 1.7459969967603683e-05, + "index_encrypt_s": 10.52772010397166, + "query_encrypt_s": 0.010291318176314235, + "index_serialize_s": 0.001604192890226841, + "query_serialize_s": 1.933099702000618e-05, + "index_deserialize_s": 0.0005795308388769627, + "query_deserialize_s": 1.3759825378656387e-05, + "server_compute_s": 0.005445469869300723, + "response_serialize_s": 0.0007037140894681215, + "response_deserialize_s": 0.0005315609741955996, + "client_decode_s": 7.0446601149160415 + }, + "sizes": { + "encrypted_index_bytes": 528396, + "public_keys_bytes": 2502, + "query_bytes": 544, + "response_bytes": 528403, + "query_plus_response_bytes": 528947 + }, + "index_ciphertexts": 1024, + "response_ciphertexts": 1024, + "minimum_response_noise_budget_bits": null, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "correct": true, + "repeat": 0 + } + ], + "source_sha256": { + "benchmarks/bfv_client_matrix.py": "c5d1f2eafe6941219c27c2501f2f753579bcbe8cce871a9f23a9591367424658", + "src/xtrace_sdk/x_vec/crypto/paillier_client.py": "40582d7acb86ee7dd2060e2d795c76829f4879db3b1eca9518be7a9599de4788", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "1f55ce8061c18644f98e99d683351bc849ef78fb917ac511c9a08eddff214d73", + "src/xtrace_sdk/x_vec/crypto/goldwasser_micali_client.py": "0982ad7532d07c8e26a701359ce8f4f7d4c1a3beb714050e741f3b534d59935f", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_client.py": "4faa0dd71d03b3e9bc9e49c4e60e8be7b173ff514b52cf1d94a3b1a2426bf699", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "6f03ac75f42218170da2bd785605ee394940a202afdd5471545a8ae3fdc8c3d8", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier_lookup.py": "eed0be7e0e2c1f6e8e8c0138262ec071fef73648e334a948ef1f82df6e9b9712", + "src/xtrace_sdk/x_vec/crypto/encryption/__init__.py": "261a50ed1758cee38c880cc625187f8acb051c6d5c96ba7a71ee956bdbea2cc5", + "src/xtrace_sdk/x_vec/crypto/encryption/goldwasser_micali.py": "4b32296c6210663c098718fa49bd810c427b1f8102d026d12a7d734f58e8e3d1", + "src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py": "915da9e500d0a142651243bb13de7631dc96103fea2ec9e5b9288bee498010e7", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier.py": "525a1944caef38d5d216ae84fcac982627d2a2fddea50b747e47ef05e2e91c9f", + "src/xtrace_sdk/x_vec/crypto/encryption/aes.py": "2c833351db3b249c381e315307d858791ee4e5b633d3465127cad763b4d095d5", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "7a10c977ee648eb1975286b313f009b3508183efe9e942cd710ae6ad5ae279a3" + }, + "median_timings": { + "bfv-packed": { + "key_generation_s": 4.32266186689958, + "public_key_export_s": 0.7880964139476418, + "public_key_import_s": 0.853526117047295, + "index_encrypt_s": 6.748511495068669, + "query_encrypt_s": 0.1036944561637938, + "index_serialize_s": 0.038064548978582025, + "query_serialize_s": 0.0002170950174331665, + "index_deserialize_s": 0.022240569116547704, + "query_deserialize_s": 0.00021161511540412903, + "server_compute_s": 108.29552126303315, + "response_serialize_s": 0.0003346470184624195, + "response_deserialize_s": 6.959214806556702e-05, + "client_decode_s": 0.0412755839060992 + }, + "paillier-cpu": { + "key_generation_s": 0.5261988120619208, + "public_key_export_s": 4.5150984078645706e-05, + "public_key_import_s": 1.7459969967603683e-05, + "index_encrypt_s": 10.52772010397166, + "query_encrypt_s": 0.010291318176314235, + "index_serialize_s": 0.001604192890226841, + "query_serialize_s": 1.933099702000618e-05, + "index_deserialize_s": 0.0005795308388769627, + "query_deserialize_s": 1.3759825378656387e-05, + "server_compute_s": 0.005445469869300723, + "response_serialize_s": 0.0007037140894681215, + "response_deserialize_s": 0.0005315609741955996, + "client_decode_s": 7.0446601149160415 + } + } +} diff --git a/docs/api_rst/xtrace_sdk.crypto.encryption.rst b/docs/api_rst/xtrace_sdk.crypto.encryption.rst index 9876089..8c33cfd 100644 --- a/docs/api_rst/xtrace_sdk.crypto.encryption.rst +++ b/docs/api_rst/xtrace_sdk.crypto.encryption.rst @@ -4,6 +4,14 @@ xtrace\_sdk.crypto.encryption package Submodules ---------- +xtrace\_sdk.crypto.encryption.bfv module +--------------------------------------- + +.. automodule:: xtrace_sdk.x_vec.crypto.encryption.bfv + :members: + :show-inheritance: + :undoc-members: + xtrace\_sdk.crypto.encryption.aes module ---------------------------------------- diff --git a/docs/api_rst/xtrace_sdk.crypto.rst b/docs/api_rst/xtrace_sdk.crypto.rst index c00b411..645ea45 100644 --- a/docs/api_rst/xtrace_sdk.crypto.rst +++ b/docs/api_rst/xtrace_sdk.crypto.rst @@ -14,6 +14,14 @@ Subpackages Submodules ---------- +xtrace\_sdk.crypto.bfv\_client module +------------------------------------ + +.. automodule:: xtrace_sdk.x_vec.crypto.bfv_client + :members: + :show-inheritance: + :undoc-members: + xtrace\_sdk.crypto.goldwasser\_micali\_client module ---------------------------------------------------- diff --git a/docs/index.rst b/docs/index.rst index c407d3c..be92059 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -77,6 +77,13 @@ Choose your path: manual_rst/changelog manual_rst/contribute +.. toctree:: + :maxdepth: 1 + :caption: Research + + research/native-bfv + research/bfv-packed-hamming + .. toctree:: :maxdepth: 1 :caption: API Reference diff --git a/docs/research/native-bfv.md b/docs/research/native-bfv.md new file mode 100644 index 0000000..2e44003 --- /dev/null +++ b/docs/research/native-bfv.md @@ -0,0 +1,273 @@ +# Native BFV implementation + +The SDK now has its own BFV arithmetic and Hamming client. It uses Python and the +existing `gmpy2` dependency; it does not import SEAL or TenSEAL. The earlier +`experiments/bfv/` implementation remains an independent SEAL reference. + +This is a **leveled BFV research implementation** for understanding the scheme and +developing faster kernels. It has no bootstrapping, is not constant time, and has +not undergone a security audit. The parameter settings below are arithmetic +defaults, not a claim of a particular security level. + +## Code map + +All paths are relative to the repository root. + +| File | Responsibility | +| --- | --- | +| `src/xtrace_sdk/x_vec/crypto/encryption/bfv.py` | Polynomial arithmetic, key generation, encryption/decryption, batch encoding, homomorphic operations, evaluation keys, modulus switching, serialization | +| `src/xtrace_sdk/x_vec/crypto/bfv_client.py` | `HammingClientBase` interface, binary-vector layout, public-only server evaluation, packed responses, client decoding | +| `src/xtrace_sdk/x_vec/utils/xtrace_types.py` | BFV parameter, polynomial, ciphertext and key types alongside the existing scheme types | +| `tests/x_vec/test_bfv_encryption.py` | Independent polynomial oracle and cryptographic arithmetic tests | +| `tests/x_vec/test_bfv_client.py` | Client, persistence, process separation, packing boundaries, default settings and optional SEAL comparison | +| `benchmarks/bfv_client_matrix.py` | Native BFV and existing CPU Paillier/Lookup timing and wire-size comparison | + +The only shared cryptographic interface correction is to the return annotations +in `HomomorphicBase`: encryption returns a ciphertext, and decryption returns a +plaintext. Existing Paillier code uses the same type for both, so its behavior is +unchanged. No Paillier implementation or SEAL experiment was replaced. + +## Local use + +```python +import json + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + +client = BFVClient(embed_len=512, device="cpu") +query = [0, 1] * 256 +vectors = [query[:], [1 - bit for bit in query], [1 - query[0], *query[1:]]] + +# The server gets configuration and public/evaluation keys only. +server = BFVClient(skip_key_gen=True) +server.load_config(json.loads(client.stringify_config())) +server.load_stringified_keys(client.stringify_pk()) +assert server.keys is None + +encrypted_index = client.encrypt_vec_packed(vectors) +encrypted_query = client.encrypt_vec_one(query) +response = server.encode_hamming_server_packed( + encrypted_query, encrypted_index, vector_count=len(vectors) +) +distances = client.decode_hamming_client_packed(response, len(vectors)) +assert distances == [0, 512, 1] +top3 = sorted(range(len(vectors)), key=lambda i: (distances[i], i))[:3] +assert top3 == [0, 2, 1] + +# Keep this on the private client when saving/restoring its state. +saved_sk = client.stringify_sk() +restored = BFVClient(skip_key_gen=True) +restored.load_config(json.loads(client.stringify_config())) +restored.load_stringified_keys(client.stringify_pk(), saved_sk) +assert restored.decode_hamming_client_packed(response, len(vectors)) == distances +``` + +For the existing per-vector interface, use `encrypt_vec_batch(vectors)`, call +`server.encode_hamming_server(encrypted_query, ciphertext)` for each result, and +pass those results to `decode_hamming_client_batch`. This returns one encrypted +distance per vector. `encrypt_vec_batch` deliberately keeps that contract; +packing is exposed through separate methods. + +`device="auto"` currently resolves to CPU. `device="gpu"` raises a clear error. +BFV is available directly as a local client; `ExecutionContext`, `DataLoader`, +and the current XTrace HTTP endpoints do not implement its index/wire protocol. +Adding a production server path is separate work. Distances are computed at the +server; selecting the top three still happens after client decryption. + +## How the scheme works + +The polynomial ring is `Z_q[X]/(X^N + 1)`. Plaintexts have N coefficients modulo +`t`; ciphertexts normally have two polynomials modulo `q`. `BFV.batch_encode` +maps N integer slots into a polynomial using the Chinese Remainder Theorem. Our +finite-field NTT performs that transform, with no floating-point arithmetic. +The slots form two independently rotating rows of N/2 entries. This follows the +[textbook's BFV batching construction](https://fhetextbook.github.io/BatchEncoding.html). + +In the implementation's sign convention: + +```text +Delta = floor(q/t) +secret key: s, with coefficients in {-1, 0, 1} +public key: (b, a), where a is uniform and b = -a*s + e mod q +encrypt m: (b*u + e0 + Delta*m, a*u + e1) mod q +decrypt: round(t * (c0 + c1*s mod q) / q) mod t +``` + +`u` is another random ternary polynomial. Each error coefficient is sampled as +the difference between two independent sums of `eta` random bits. Key generation +and encryption use Python's OS-backed `secrets` module. Benchmark seeds affect +synthetic input vectors only. See the textbook's +[encryption and decryption section](https://fhetextbook.github.io/EncryptionandDecryption1.html) +for the underlying scale-and-noise construction. + +Multiplication first computes the three tensor-product polynomials as **exact +integers**, rounds each coefficient after multiplication by `t/q`, and then +reduces modulo `q`. Reducing the tensor product modulo `q` before scaling would +lose information required to recover the message. Relinearization replaces the +third component's dependence on `s²` with two components under `s`, using public +gadget keys. Scaling before relinearization is a valid ordering; see +[BFV multiplication](https://fhetextbook.github.io/CiphertexttoCiphertextMultiplication.html). + +The public gadget key for a source polynomial `f` has pairs satisfying +`b_i + a_i*s = 2^(i*decomposition_bits)*f + e_i mod q`. It is used with `f=s²` +for relinearization and `f=s(X^g)` for a rotation. The evaluator decomposes the +corresponding ciphertext component into radix digits and combines these pairs. +It does not receive `s`. A key fingerprint catches accidental key mixups; it +does not authenticate ciphertexts or serialized key material. + +### GMP polynomial multiplication + +`_poly_product` is the main optimization boundary. It packs nonnegative +coefficients into one GMP integer with a radix wider than any possible +convolution coefficient. A GMP integer multiplication then computes the ordinary +convolution. Unpacking and subtracting the upper half implements `X^N = -1`. +The result remains exact, including negative coefficients, until the caller +scales or reduces it. Squaring reuses the packed operand and one cross product. + +This is **Kronecker substitution**, not SEAL's RNS implementation. Ciphertext +arithmetic uses a single large modulus; only the plaintext batch transform uses +an NTT. The small-ring schoolbook oracle tests products through 512-bit +coefficients, including worst-case all-maximum inputs that exercise radix carries. + +Natural next optimizations are cached transform plans, RNS/NTT polynomial +products, faster gadget decomposition/key switching, then CUDA kernels behind +the same primitive operations. Any replacement must preserve the unreduced +product needed by BFV's scale-and-round step. + +## Packed Hamming layout + +Let `D` be the embedding length rounded up to a power of two, and let +`B = N/(2D)`. Each ciphertext stores `2B` database vectors. Within each row, +dimension `j` occupies the B slots beginning at `j*B`; those slots belong to +different candidate vectors. Query encryption repeats its bit across those +lanes. Missing dimensions and unused candidate lanes are padded with zeros. + +The server squares the slotwise difference: for binary inputs, `(a-b)^2` is the +XOR bit. Rotating and adding by `B, 2B, 4B, ...` sums the D dimensions separately +for each candidate. A plaintext mask retains one copy of each distance and +clears unused lanes. A binary merge tree rotates distance tiles into disjoint +positions, filling up to N result slots per response ciphertext. + +For the defaults, `N=8192`, `D=512`, `B=8`: an encrypted index ciphertext holds +16 vectors, while a response ciphertext holds up to 8192 distances. Input order +is restored during decoding. IDs stay with the caller; it must supply the true +`vector_count` and preserve the index order. Dimension, key and layout must be +shared consistently between client and evaluator. + +The finished response is rounded to a smaller modulus to reduce transmitted +bytes. This terminal switch changes its coefficient width from 180 to 50 bits. +The implementation does not provide evaluation keys for multiplication or +rotation at the reduced modulus. Pass `compact=False` to server evaluation to +retain the original modulus for further work or noise comparisons. + +## Parameters and limits + +| Setting | Default | Meaning | +| --- | ---: | --- | +| `poly_modulus_degree` | 8192 | N, the ring degree and number of batch slots | +| `plain_modulus` | 65537 | t, a prime congruent to 1 modulo 2N | +| `coeff_modulus_bits` | 180 | q is the largest probable prime below 2^180 | +| `decomposition_bits` | 30 | Gadget radix is 2^30; six digit pairs per evaluation key | +| `error_eta` | 21 | Centered-binomial error variance is eta/2 | +| `response_modulus_bits` | 50 | Terminal response modulus width | + +`embed_len` must be positive, at most N/2, and smaller than t, so valid Hamming +distances do not wrap modulo t. The parameter checks enforce algebraic +constraints, not cryptographic security or sufficient noise for every circuit. +Small ring settings used in tests are explicitly insecure. The defaults and +SEAL's settings are different; performance comparisons do not imply equivalent +security. + +`BFV.noise_budget` is a private diagnostic requiring the secret key. It must not +be exposed as a server oracle. A positive budget cannot establish the integrity +of a result or rule out earlier noise overflow. Tests and benchmarks compare +every decoded distance against independent plaintext computation. The client +also rejects distances outside `[0, embed_len]`, but in-range errors can still +occur if an unsupported circuit exhausts its noise. + +## Tests and benchmarks + +```bash +.venv/bin/python -m pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_client.py -q +.venv/bin/python benchmarks/bfv_client_matrix.py --num-vectors 1024 \ + --json-out benchmarks/results/native_bfv_1024.json + +# Compare both BFV layouts and both existing CPU Paillier clients on a small batch. +.venv/bin/python benchmarks/bfv_client_matrix.py --num-vectors 16 \ + --variants bfv-packed,bfv-individual,paillier-cpu,paillier-lookup-cpu +``` + +The optional SEAL sanity test runs when `tenseal` from +`experiments/bfv/requirements.txt` is installed; otherwise that test is skipped. +The core and all other new tests work without it. Tests cover two multiplication +levels, rotations, public-only evaluation in another process, invalid inputs, +key persistence, multiple response groups, partial tiles and default 512-bit +vectors. No account, network connection or GPU is needed. + +The benchmark reports key generation/export/import, index and query encryption, +server evaluation, client decryption/decoding, and serialization separately. +`--repeats` generates independent runs; raw results and median timings are saved. +The four size categories are: + +| Measurement | What is included | When it is paid | +| --- | --- | --- | +| Encrypted index | Encrypted database vectors with count/framing | Upload/storage, then updates | +| Public keys | Public encryption and evaluation keys serialized as JSON; no secret key | Server setup or key rotation | +| Query | One encrypted query with framing | Each search, client to server | +| Response | Encrypted distances with framing | Each search, server to client | + +Index, query and response use the same MessagePack envelope across schemes; +large integers are encoded as little-endian byte strings. These are measured +serialized lengths, not Python object sizes. IDs are implicit positions. AES +content, HTTP/TLS overhead, network latency and top-k content retrieval are +excluded. The public-key JSON representation is intentionally readable and +considerably larger than a compact binary format would be. Comparisons use the +current branch's explicit CPU Paillier clients, not GPU timings. + +The ratio `Paillier(query + response) / BFV(query + response)` measures recurring +traffic savings. Index and key costs are separate. A smaller BFV response alone +does not guarantee less total traffic for a small database. + +### Initial measurement: 1,024 vectors of 512 bits + +The [saved run](../../benchmarks/results/native_bfv_1024.json) is in +`benchmarks/results/native_bfv_1024.json` (2026-09-10, +Python 3.12.3, gmpy2 2.3.0, GMP 6.3.0). Both variants used the same synthetic +inputs. Each decrypted all 1,024 distances correctly and selected positions +`[0, 1023, 81]` as the top three, with distances `[0, 0, 220]`. This is one CPU +run, not a statistically established performance claim. The JSON includes +source-file SHA-256 hashes because the implementation was uncommitted during +measurement; no other test or benchmark was running concurrently. All SDK source +hashes match this implementation. The benchmark harness received formatting and +lint fixes outside the measured operations afterward. + +| Bytes | Native BFV, packed | Existing Paillier CPU, key_len=1024 | +| --- | ---: | ---: | +| Encrypted index | 23,598,496 | 528,396 | +| Public/evaluation keys, JSON | 85,602,812 | 2,502 | +| Query | 368,754 | 544 | +| Response | 102,496 | 528,403 | +| Query + response | 471,250 | 528,947 | + +The response is **5.16 times smaller**, while query plus response is **1.12 times +smaller**, about **10.9% fewer bytes**. The BFV index contains 64 ciphertexts and +the response contains one; Paillier uses 1,024 ciphertexts for each. These native +results do not inherit the earlier SEAL experiment's traffic ratios. + +| Seconds, excluding wire serialization | Native BFV, packed | Paillier CPU | +| --- | ---: | ---: | +| Key generation | 4.323 | 0.526 | +| Encrypt index | 6.749 | 10.528 | +| Encrypt query | 0.104 | 0.010 | +| Server compute | 108.296 | 0.00545 | +| Client decrypt and decode | 0.0413 | 7.045 | + +BFV's result retained a 26-bit diagnostic noise budget after compaction. +Server compute and larger index/key storage are the present costs of moving +distance decoding off the client. The result is a working optimization baseline, +not an end-to-end latency improvement. + +Validation for this change: 121 offline tests passed across `experiments/bfv`, +`tests/cli`, and `tests/x_vec`, with the live Hamming/metadata service tests and +CUDA tests excluded. The new native modules account for 39 of those tests, +including the optional SEAL oracle in this environment. diff --git a/src/xtrace_sdk/x_vec/crypto/__init__.py b/src/xtrace_sdk/x_vec/crypto/__init__.py index 174baa9..3a386fc 100644 --- a/src/xtrace_sdk/x_vec/crypto/__init__.py +++ b/src/xtrace_sdk/x_vec/crypto/__init__.py @@ -9,4 +9,7 @@ In addition, it provides ``hamming_client_base`` as a base class for crytpographic clients that compute hamming distance between two ciphers. -""" \ No newline at end of file + + ``bfv_client`` provides a native GMP BFV implementation of that interface, + with additional packed-index and packed-response operations (experimental). +""" diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_client.py new file mode 100644 index 0000000..186acab --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_client.py @@ -0,0 +1,334 @@ +"""Native CPU BFV client for individual and packed encrypted Hamming distances. + +This implements the local HammingClientBase interface. The BFV wire format and +packed index need corresponding server support before use with XTrace's HTTP API. +""" + +import json +from collections.abc import Sequence +from dataclasses import asdict +from numbers import Integral +from typing import Any + +from xtrace_sdk.x_vec.crypto.device import DeviceMode +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV +from xtrace_sdk.x_vec.crypto.hamming_client_base import HammingClientBase +from xtrace_sdk.x_vec.utils.xtrace_types import ( + BFVCiphertext, + BFVKeyPair, + BFVParameters, + BFVPolynomial, + BFVPublicKey, + EncryptedVector, +) + + +class BFVClient(HammingClientBase): + """Self-implemented BFV Hamming client, using GMP on the CPU. + + ``encrypt_vec_batch`` keeps the existing one-ciphertext-per-vector contract. + The explicit ``*_packed`` methods amortize each ciphertext over several + database vectors and pack up to N distances into each response ciphertext. + All homomorphic evaluation uses public keys only; top-k stays on the client. + + :param embed_len: Binary embedding dimension, at most N/2 and less than t. + :param poly_modulus_degree: Ring degree N, a power of two. + :param coeff_modulus_bits: Bit length of the initial ciphertext modulus q. + :param response_modulus_bits: Smaller modulus used after the Hamming circuit. + :param skip_key_gen: Construct an empty client for loading saved keys. + :param device: ``cpu`` or ``auto``; a CUDA implementation is not yet available. + """ + + def __init__( + self, + embed_len: int = 512, + poly_modulus_degree: int = 8192, + plain_modulus: int = 65537, + coeff_modulus_bits: int = 180, + decomposition_bits: int = 30, + error_eta: int = 21, + response_modulus_bits: int = 50, + skip_key_gen: bool = False, + device: DeviceMode = "auto", + ) -> None: + if device not in ("cpu", "auto", "gpu"): + raise ValueError("device must be 'auto', 'cpu', or 'gpu'") + if device == "gpu": + raise NotImplementedError("Native BFV currently supports only the CPU backend") + self.device = "cpu" + self.params = BFVParameters( + poly_modulus_degree, plain_modulus, coeff_modulus_bits, decomposition_bits, error_eta + ) + self.embed_len = embed_len + self.response_modulus_bits = response_modulus_bits + self._configure_layout() + self.keys: BFVKeyPair | None = None + self.public_key: BFVPublicKey | None = None + if not skip_key_gen: + self.keys = BFV.key_gen(**asdict(self.params), rotation_steps=self._rotation_steps()) + self.public_key = self.keys["pk"] + + def _configure_layout(self) -> None: + BFV.validate_parameters(self.params) + n, t = self.params.poly_modulus_degree, self.params.plain_modulus + if ( + type(self.embed_len) is not int + or not 1 <= self.embed_len <= n // 2 + or self.embed_len >= t + ): + raise ValueError("embed_len must be positive, <= N/2, and < plain_modulus") + if ( + type(self.response_modulus_bits) is not int + or not t.bit_length() + 8 + <= self.response_modulus_bits + <= self.params.coeff_modulus_bits + ): + raise ValueError( + "response_modulus_bits must be between plain_modulus bits + 8 and coeff_modulus_bits" + ) + self.padded_embed_len = 1 << (self.embed_len - 1).bit_length() + self.lanes_per_row = n // (2 * self.padded_embed_len) + self.vectors_per_ciphertext = 2 * self.lanes_per_row + self.distances_per_ciphertext = n + self._mask_cache: dict[int, BFVPolynomial] = {} + + def _rotation_steps(self) -> list[int]: + return [ + sign * self.lanes_per_row * (1 << i) + for i in range(self.padded_embed_len.bit_length() - 1) + for sign in (1, -1) + ] + + def _pk(self) -> BFVPublicKey: + if self.public_key is None: + raise RuntimeError("Keys not initialized") + return self.public_key + + def _keys(self) -> BFVKeyPair: + if self.keys is None: + raise RuntimeError("Secret key not initialized; public-only clients cannot decrypt") + return self.keys + + @staticmethod + def has_gpu() -> bool: + return False + + def stringify_pk(self) -> str: + """Public encryption/evaluation keys; safe to separate from the secret key.""" + return BFV.serialize_public_key(self._pk()) + + def stringify_sk(self) -> str: + return BFV.serialize_secret_key(self._keys()["sk"]) + + def stringify_config(self) -> str: + return json.dumps( + { + "embed_len": self.embed_len, + **asdict(self.params), + "response_modulus_bits": self.response_modulus_bits, + "device": "cpu", + } + ) + + def load_config(self, config: dict[str, Any]) -> None: + """Load layout/arithmetic settings before loading keys, as in the Paillier clients.""" + candidate = BFVClient( + **{k: v for k, v in config.items() if k != "device"}, skip_key_gen=True + ) + if self.public_key is not None and candidate.stringify_config() != self.stringify_config(): + raise ValueError( + "Load a changed configuration into an empty client before loading its keys" + ) + self.params = candidate.params + self.embed_len = candidate.embed_len + self.response_modulus_bits = candidate.response_modulus_bits + self._configure_layout() + + def load_stringified_keys(self, pk: str, sk: str | None = None) -> None: + """Load private client keys, or just public keys for server-side evaluation. + + :param pk: ``stringify_pk()`` JSON, including the evaluation keys. + :param sk: ``stringify_sk()`` JSON. Omit on the server. + """ + public_key = BFV.deserialize_public_key(pk) + if public_key["params"] != self.params: + raise ValueError("Loaded BFV key parameters do not match the client configuration") + n = self.params.poly_modulus_degree + required = {pow(3, step % (n // 2), 2 * n) for step in self._rotation_steps()} - {1} + if not public_key["relin_key"] or not required.issubset(public_key["galois_keys"]): + raise ValueError("Public key lacks evaluation keys required by this Hamming layout") + keys: BFVKeyPair | None = None + if sk is not None: + keys = {"pk": public_key, "sk": BFV.deserialize_secret_key(sk, public_key)} + self.public_key = public_key + self.keys = keys + + def _validate_vector(self, embd: Sequence[int]) -> None: + if len(embd) != self.embed_len: + raise ValueError( + f"Embedding length {len(embd)} does not match expected {self.embed_len}" + ) + if any(not isinstance(bit, Integral) or bit not in (0, 1) for bit in embd): + raise ValueError("Embedding vector must contain binary integers") + + def _query_slots(self, embd: Sequence[int]) -> list[int]: + self._validate_vector(embd) + # Dimension-major layout: each dimension occupies B consecutive lanes. + # Repeat the query across every lane, identically in the two rows. + row = [int(bit) for bit in embd for _ in range(self.lanes_per_row)] + row.extend([0] * ((self.padded_embed_len - self.embed_len) * self.lanes_per_row)) + return row + row + + def encrypt_vec_one(self, embd: list[int]) -> EncryptedVector: + """Encrypt one vector, replicated across all lanes so it also serves as a query.""" + pk = self._pk() + return BFV.ciphertext_to_ints( + BFV.encrypt(BFV.batch_encode(self._query_slots(embd), self.params), pk), pk + ) + + def encrypt_vec_batch(self, embds: list[list[int]]) -> list[EncryptedVector]: + """Encrypt vectors independently, preserving HammingClientBase semantics.""" + return [self.encrypt_vec_one(embd) for embd in embds] + + def encrypt_vec_packed(self, embds: list[list[int]]) -> list[EncryptedVector]: + """Build a packed encrypted index with up to N/padded_dimension vectors per ciphertext.""" + pk = self._pk() + for embd in embds: + self._validate_vector(embd) + result = [] + n, b, capacity = ( + self.params.poly_modulus_degree, + self.lanes_per_row, + self.vectors_per_ciphertext, + ) + for start in range(0, len(embds), capacity): + slots = [0] * n + for lane, embd in enumerate(embds[start : start + capacity]): + row, column = divmod(lane, b) + for dimension, bit in enumerate(embd): + slots[row * (n // 2) + dimension * b + column] = int(bit) + result.append( + BFV.ciphertext_to_ints(BFV.encrypt(BFV.batch_encode(slots, self.params), pk), pk) + ) + return result + + def _distance_tile( + self, query: BFVCiphertext, tile: BFVCiphertext, count: int + ) -> BFVCiphertext: + pk = self._pk() + distance = BFV.xor(query, tile, pk) + for i in range(self.padded_embed_len.bit_length() - 1): + step = self.lanes_per_row * (1 << i) + distance = BFV.add(distance, BFV.rotate_rows(distance, step, pk), pk) + # The rotate/add tree repeats sums in each dimension block. Retain only + # the first block in each row and clear unused lanes in a partial tile. + if count not in self._mask_cache: + if len(self._mask_cache) >= 8: + del self._mask_cache[next(iter(self._mask_cache))] + mask = [0] * self.params.poly_modulus_degree + for lane in range(count): + row, col = divmod(lane, self.lanes_per_row) + mask[row * (len(mask) // 2) + col] = 1 + self._mask_cache[count] = BFV.batch_encode(mask, self.params) + return BFV.multiply_plain(distance, self._mask_cache[count], pk) + + def encode_hamming_server( + self, + ct1: Sequence[int | bytes], + ct2: Sequence[int | bytes], + *, + compact: bool = True, + ) -> EncryptedVector: + """Compute an encrypted distance using only public evaluation keys.""" + pk = self._pk() + result = self._distance_tile( + BFV.ciphertext_from_ints(ct1, pk), BFV.ciphertext_from_ints(ct2, pk), 1 + ) + if compact: + result = BFV.modulus_switch(result, self.response_modulus_bits, pk) + return BFV.ciphertext_to_ints(result, pk) + + def encode_hamming_server_packed( + self, + query: Sequence[int | bytes], + index: Sequence[Sequence[int | bytes]], + vector_count: int, + *, + compact: bool = True, + ) -> list[EncryptedVector]: + """Evaluate a packed index and return ceil(vector_count/N) ciphertexts. + + A binary carry tree packs distance tiles using right rotations by powers + of two. Only a logarithmic number of intermediate tiles is kept per + response. The caller retains candidate IDs in the original index order. + """ + pk = self._pk() + capacity = self.vectors_per_ciphertext + self._validate_count(vector_count, len(index), capacity) + query_ct = BFV.ciphertext_from_ints(query, pk) + result = [] + tiles_per_response = self.padded_embed_len + for start in range(0, len(index), tiles_per_response): + stack: list[tuple[BFVCiphertext, int]] = [] + for offset, wire in enumerate(index[start : start + tiles_per_response]): + count = min(capacity, vector_count - (start + offset) * capacity) + tile = self._distance_tile(query_ct, BFV.ciphertext_from_ints(wire, pk), count) + size = 1 + while stack and stack[-1][1] == size: + left, _ = stack.pop() + tile = BFV.add(left, BFV.rotate_rows(tile, -self.lanes_per_row * size, pk), pk) + size *= 2 + stack.append((tile, size)) + # Partial groups have a decreasing list of powers of two. Combining + # from the right uses the same already-generated rotation keys. + combined, _ = stack.pop() + while stack: + left, size = stack.pop() + combined = BFV.add( + left, BFV.rotate_rows(combined, -self.lanes_per_row * size, pk), pk + ) + if compact: + combined = BFV.modulus_switch(combined, self.response_modulus_bits, pk) + result.append(BFV.ciphertext_to_ints(combined, pk)) + return result + + @staticmethod + def _validate_count(count: int, ciphertexts: int, capacity: int) -> None: + if type(count) is not int or count < 0 or ciphertexts != (count + capacity - 1) // capacity: + raise ValueError("vector_count does not match the packed ciphertext count") + + def _decode_slots(self, cipher: Sequence[int | bytes]) -> list[int]: + return BFV.batch_decode( + BFV.decrypt(BFV.ciphertext_from_ints(cipher, self._pk()), self._keys()), self.params + ) + + def _check_distances(self, distances: list[int]) -> list[int]: + if any(not 0 <= value <= self.embed_len for value in distances): + raise ValueError( + "Decoded distance is out of range; check input, parameters and noise budget" + ) + return distances + + def decode_hamming_client_one(self, cipher: Sequence[int | bytes]) -> int: + """Decrypt the first distance slot of an individual response.""" + return self._check_distances([self._decode_slots(cipher)[0]])[0] + + def decode_hamming_client_batch(self, ciphers: Sequence[Sequence[int | bytes]]) -> list[int]: + return [self.decode_hamming_client_one(cipher) for cipher in ciphers] + + def decode_hamming_client_packed( + self, + ciphers: Sequence[Sequence[int | bytes]], + vector_count: int, + ) -> list[int]: + """Decrypt packed distances in input order; padded slots are omitted.""" + n, b = self.params.poly_modulus_degree, self.lanes_per_row + self._validate_count(vector_count, len(ciphers), n) + result = [] + for group, cipher in enumerate(ciphers): + slots = self._decode_slots(cipher) + for position in range(min(n, vector_count - group * n)): + tile, lane = divmod(position, self.vectors_per_ciphertext) + row, column = divmod(lane, b) + result.append(slots[row * (n // 2) + tile * b + column]) + return self._check_distances(result) diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/__init__.py b/src/xtrace_sdk/x_vec/crypto/encryption/__init__.py index bde4d9c..7622d34 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/__init__.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/__init__.py @@ -3,8 +3,9 @@ Includes: - ``aes``: AES encryption for chunk content +- ``bfv``: native GMP BFV with batching and multiplication (experimental) - ``paillier``: Paillier homomorphic encryption - ``paillier_lookup``: Paillier variant with precomputed lookup tables - ``goldwasser_micali``: Goldwasser-Micali homomorphic encryption (experimental) - ``homomorphic_base``: base class for homomorphic encryption schemes -""" \ No newline at end of file +""" diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py new file mode 100644 index 0000000..f37e72d --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py @@ -0,0 +1,703 @@ +"""Native, leveled BFV over Z_q[X]/(X^N + 1), using GMP integer arithmetic. + +The implementation deliberately keeps a single large ciphertext modulus. GMP +Kronecker substitution handles polynomial products; a small-modulus NTT handles +plaintext batching. No SEAL/TenSEAL code is called. This is an experimental, +variable-time implementation for learning and optimization, not audited crypto. +""" + +import hashlib +import json +import secrets +from collections.abc import Sequence +from dataclasses import asdict +from functools import lru_cache +from numbers import Integral +from typing import Any + +import gmpy2 +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.encryption.homomorphic_base import HomomorphicBase +from xtrace_sdk.x_vec.utils.xtrace_types import ( + BFVCiphertext, + BFVKeyPair, + BFVParameters, + BFVPolynomial, + BFVPublicKey, + BFVSecretKey, + BFVSwitchKey, + EncryptedVector, +) + + +def _round_div(value: mpz, divisor: mpz) -> mpz: + """Nearest integer, with ties toward +infinity; works for negative inputs.""" + return (2 * value + divisor) // (2 * divisor) + + +def _poly_product(lhs: BFVPolynomial, rhs: BFVPolynomial) -> BFVPolynomial: + """Exact negacyclic integer product of two nonnegative coefficient arrays. + + Choose base 2^width larger than every ordinary convolution coefficient. + Packing then multiplying with GMP computes that convolution without carries + crossing coefficient boundaries. Fold X^N = -1 *after* unpacking. In BFV + multiplication these signed integers must survive until scale-and-round. + This function is the main seam for a future RNS/NTT or CUDA backend. + """ + n = len(lhs) + width = max(c.bit_length() for c in lhs) + max(c.bit_length() for c in rhs) + width += n.bit_length() + packed_lhs = gmpy2.pack(list(lhs), width) + packed_rhs = packed_lhs if lhs is rhs else gmpy2.pack(list(rhs), width) + product = packed_lhs * packed_rhs + coefficients = gmpy2.unpack(product, width) + coefficients.extend([mpz(0)] * (2 * n - len(coefficients))) + return tuple(coefficients[i] - coefficients[i + n] for i in range(n)) + + +def _ring_product(lhs: BFVPolynomial, rhs: BFVPolynomial, q: mpz) -> BFVPolynomial: + return tuple(c % q for c in _poly_product(lhs, rhs)) + + +def _automorphism(poly: BFVPolynomial, exponent: int, q: mpz) -> BFVPolynomial: + """Substitute X -> X^exponent, where exponent is odd, using X^N = -1.""" + n = len(poly) + result = [mpz(0)] * n + for i, value in enumerate(poly): + position = (i * exponent) % (2 * n) + result[position % n] = (value if position < n else -value) % q + return tuple(result) + + +def _ntt(values: Sequence[int], root: int, modulus: int) -> list[int]: + """Radix-2 finite-field transform; inverse uses root^-1 and an external 1/N.""" + result = list(values) + n = len(result) + j = 0 + for i in range(1, n): + bit = n >> 1 + while j & bit: + j ^= bit + bit >>= 1 + j ^= bit + if i < j: + result[i], result[j] = result[j], result[i] + length = 2 + while length <= n: + stride_root = pow(root, n // length, modulus) + for start in range(0, n, length): + twiddle = 1 + for offset in range(length // 2): + i = start + offset + u = result[i] + v = result[i + length // 2] * twiddle % modulus + result[i] = (u + v) % modulus + result[i + length // 2] = (u - v) % modulus + twiddle = twiddle * stride_root % modulus + length *= 2 + return result + + +@lru_cache(maxsize=8) +def _batch_plan(n: int, t: int) -> tuple[int, tuple[int, ...], tuple[int, ...]]: + # t = 1 mod 2N ensures a primitive 2N-th root psi exists in F_t. + candidate = 2 + while True: + psi = pow(candidate, (t - 1) // (2 * n), t) + if pow(psi, n, t) == t - 1: + break + candidate += 1 + twists = tuple(pow(psi, i, t) for i in range(n)) + # Two rows: psi^(3^i) and psi^(-3^i). Substitution X -> X^3 rotates + # each row one slot to the left; it does not rotate across the rows. + exponents = [pow(3, i, 2 * n) for i in range(n // 2)] + order = tuple((e - 1) // 2 for e in exponents) + order += tuple(((-e) % (2 * n) - 1) // 2 for e in exponents) + return psi * psi % t, twists, order + + +@lru_cache(maxsize=16) +def _coefficient_modulus(bits: int) -> mpz: + # No ciphertext NTT is used, so q need not be 1 modulo 2N. + # Avoid prev_prime(), which is unavailable on older supported GMP builds. + candidate = (mpz(1) << bits) - 1 + while not gmpy2.is_prime(candidate): + candidate -= 2 + return candidate + + +def _small_poly(n: int, eta: int, q: mpz) -> BFVPolynomial: + # Centered binomial: sum of eta independent bits minus another eta bits. + # Each sample uses OS-backed randomness; never use benchmark PRNGs here. + return tuple( + mpz(secrets.randbits(eta).bit_count() - secrets.randbits(eta).bit_count()) % q + for _ in range(n) + ) + + +def _ternary_poly(n: int, q: mpz) -> BFVPolynomial: + return tuple(mpz(secrets.randbelow(3) - 1) % q for _ in range(n)) + + +def _key_id(params: BFVParameters, b: BFVPolynomial, a: BFVPolynomial) -> str: + digest = hashlib.sha256(json.dumps(asdict(params), sort_keys=True).encode()) + width = (params.coeff_modulus_bits + 7) // 8 + for poly in (b, a): + for c in poly: + digest.update(int(c).to_bytes(width, "little")) + return digest.hexdigest() + + +class BFV(HomomorphicBase[BFVCiphertext, BFVPolynomial, BFVKeyPair, BFVPublicKey]): + """Self-implemented BFV with batching, relinearization and row rotations. + + Arithmetic is exact modulo the plaintext modulus while noise permits. + There is no bootstrapping, automatic parameter selection or security-level + claim. Tiny rings are accepted for algebra tests and are insecure. + """ + + def __init__(self, keys: BFVKeyPair) -> None: + self.keys = keys + + @staticmethod + def validate_parameters(params: BFVParameters) -> None: + """Check algebraic constraints, independently of a security assessment.""" + if any(not isinstance(v, int) or isinstance(v, bool) for v in asdict(params).values()): + raise ValueError("BFV parameters must be integers") + n, t = params.poly_modulus_degree, params.plain_modulus + if n < 8 or n > 32768 or n & (n - 1): + raise ValueError("poly_modulus_degree must be a power of two in [8, 32768]") + if t < 3 or t.bit_length() > 60 or not gmpy2.is_prime(t) or t % (2 * n) != 1: + raise ValueError("plain_modulus must be prime, <= 60 bits, and 1 modulo 2N") + if not t.bit_length() + 8 <= params.coeff_modulus_bits <= 512: + raise ValueError("coeff_modulus_bits must be between plain_modulus bits + 8 and 512") + if not 1 <= params.decomposition_bits <= params.coeff_modulus_bits: + raise ValueError("decomposition_bits must be in [1, coeff_modulus_bits]") + if not 1 <= params.error_eta <= 64: + raise ValueError("error_eta must be in [1, 64]") + + @staticmethod + def key_gen( + poly_modulus_degree: int = 8192, + plain_modulus: int = 65537, + coeff_modulus_bits: int = 180, + decomposition_bits: int = 30, + error_eta: int = 21, + rotation_steps: Sequence[int] = (), + relinearization: bool = True, + ) -> BFVKeyPair: + """Generate public/secret keys and requested public evaluation keys. + + :param rotation_steps: Left rotations within each of the two N/2-slot rows. + Negative steps rotate right. Only requested rotations get keys. + :param relinearization: Generate the key needed to reduce products to two polynomials. + :return: PK/SK pair; only the PK (including evaluation keys) goes to the server. + :rtype: BFVKeyPair + """ + params = BFVParameters( + poly_modulus_degree, plain_modulus, coeff_modulus_bits, decomposition_bits, error_eta + ) + BFV.validate_parameters(params) + n, q = poly_modulus_degree, _coefficient_modulus(coeff_modulus_bits) + s = _ternary_poly(n, q) + a = tuple(mpz(secrets.randbelow(int(q))) for _ in range(n)) + error = _small_poly(n, error_eta, q) + a_s = _ring_product(a, s, q) + b = tuple((e - v) % q for e, v in zip(error, a_s, strict=True)) + + def switch_key(source: BFVPolynomial) -> BFVSwitchKey: + result = [] + for shift in range(0, coeff_modulus_bits, decomposition_bits): + a_i = tuple(mpz(secrets.randbelow(int(q))) for _ in range(n)) + e_i = _small_poly(n, error_eta, q) + product = _ring_product(a_i, s, q) + b_i = tuple( + ((v << shift) - a_v + e) % q + for v, a_v, e in zip(source, product, e_i, strict=True) + ) + result.append((b_i, a_i)) + return tuple(result) + + relin_key = switch_key(_ring_product(s, s, q)) if relinearization else () + galois_keys = {} + for steps in rotation_steps: + if not isinstance(steps, int): + raise ValueError("rotation_steps must contain integers") + exponent = pow(3, steps % (n // 2), 2 * n) + if exponent != 1 and exponent not in galois_keys: + galois_keys[exponent] = switch_key(_automorphism(s, exponent, q)) + identity = _key_id(params, b, a) + return { + "pk": { + "params": params, + "q": q, + "key_id": identity, + "b": b, + "a": a, + "relin_key": relin_key, + "galois_keys": galois_keys, + }, + "sk": {"s": tuple(v if v <= q // 2 else v - q for v in s), "key_id": identity}, + } + + @staticmethod + def _plaintext(plaintext: Sequence[int], params: BFVParameters) -> BFVPolynomial: + if len(plaintext) != params.poly_modulus_degree: + raise ValueError("Plaintext must contain exactly N coefficients") + if any(not isinstance(c, Integral) or not 0 <= c < params.plain_modulus for c in plaintext): + raise ValueError("Plaintext coefficients must be integers in [0, t)") + return tuple(mpz(c) for c in plaintext) + + @staticmethod + def batch_encode(slots: Sequence[int], params: BFVParameters) -> BFVPolynomial: + """CRT encode up to N integers into a polynomial, zero-padding empty slots. + + :param slots: Values in [0, t), ordered as two rows of N/2 slots. + :return: N polynomial coefficients in [0, t). + :rtype: BFVPolynomial + """ + BFV.validate_parameters(params) + n, t = params.poly_modulus_degree, params.plain_modulus + if len(slots) > n: + raise ValueError("At most N batch slots can be encoded") + if any(not isinstance(c, Integral) or not 0 <= c < t for c in slots): + raise ValueError("Batch slots must be integers in [0, t)") + root, twists, order = _batch_plan(n, t) + evaluations = [0] * n + for position, value in zip(order, slots, strict=False): + evaluations[position] = int(value) + coefficients = _ntt(evaluations, pow(root, -1, t), t) + inv_n = pow(n, -1, t) + return tuple( + mpz(c * inv_n * pow(twist, -1, t) % t) + for c, twist in zip(coefficients, twists, strict=True) + ) + + @staticmethod + def batch_decode(plaintext: BFVPolynomial, params: BFVParameters) -> list[int]: + """Decode a polynomial into the two rows of exact integer slots.""" + BFV.validate_parameters(params) + plaintext = BFV._plaintext(plaintext, params) + t = params.plain_modulus + root, twists, order = _batch_plan(params.poly_modulus_degree, t) + evaluations = _ntt( + [int(c) * twist % t for c, twist in zip(plaintext, twists, strict=True)], root, t + ) + return [evaluations[i] for i in order] + + @staticmethod + def validate_ciphertext(ciphertext: BFVCiphertext, pk: BFVPublicKey) -> None: + """Reject mixed keys, incompatible shapes, and noncanonical coefficients.""" + n, q = pk["params"].poly_modulus_degree, ciphertext.modulus + if ciphertext.key_id != pk["key_id"]: + raise ValueError("Ciphertext belongs to a different BFV key") + if not pk["params"].plain_modulus < q <= pk["q"]: + raise ValueError("Invalid ciphertext modulus") + if len(ciphertext.components) not in (2, 3): + raise ValueError("BFV ciphertext must have two or three components") + for poly in ciphertext.components: + if len(poly) != n or any( + not isinstance(c, Integral) or not 0 <= int(c) < q for c in poly + ): + raise ValueError("Invalid ciphertext polynomial") + + @staticmethod + def encrypt(plaintext: BFVPolynomial, pk: BFVPublicKey) -> BFVCiphertext: + """Encrypt polynomial m: (b*u + e0 + floor(q/t)*m, a*u + e1). + + :param plaintext: Polynomial produced by ``batch_encode`` or N coefficients. + :param pk: Public key; the secret key is not used for encryption. + :return: A randomized two-component BFV ciphertext. + :rtype: BFVCiphertext + """ + params, q = pk["params"], pk["q"] + plaintext = BFV._plaintext(plaintext, params) + n = params.poly_modulus_degree + u = _ternary_poly(n, q) + e0, e1 = _small_poly(n, params.error_eta, q), _small_poly(n, params.error_eta, q) + bu, au = _ring_product(pk["b"], u, q), _ring_product(pk["a"], u, q) + delta = q // params.plain_modulus + c0 = tuple((v + e + delta * m) % q for v, e, m in zip(bu, e0, plaintext, strict=True)) + c1 = tuple((v + e) % q for v, e in zip(au, e1, strict=True)) + return BFVCiphertext((c0, c1), q, pk["key_id"]) + + @staticmethod + def _phase(ciphertext: BFVCiphertext, keys: BFVKeyPair) -> BFVPolynomial: + BFV.validate_ciphertext(ciphertext, keys["pk"]) + if keys["sk"]["key_id"] != keys["pk"]["key_id"]: + raise ValueError("Secret and public keys do not match") + q = ciphertext.modulus + s = tuple(c % q for c in keys["sk"]["s"]) + # Horner evaluation handles both relinearized and quadratic ciphertexts. + phase = ciphertext.components[-1] + for poly in reversed(ciphertext.components[:-1]): + phase = tuple( + (a + b) % q for a, b in zip(_ring_product(phase, s, q), poly, strict=True) + ) + return phase + + @staticmethod + def decrypt(ciphertext: BFVCiphertext, keys: BFVKeyPair) -> BFVPolynomial: + """Recover round(t * (c0 + c1*s + ...) / q) modulo t. + + Decryption does not authenticate ciphertexts or detect every noise failure. + """ + t = keys["pk"]["params"].plain_modulus + return tuple( + _round_div(t * c, ciphertext.modulus) % t for c in BFV._phase(ciphertext, keys) + ) + + @staticmethod + def _pair(lhs: BFVCiphertext, rhs: BFVCiphertext, pk: BFVPublicKey) -> None: + BFV.validate_ciphertext(lhs, pk) + BFV.validate_ciphertext(rhs, pk) + if lhs.modulus != rhs.modulus or len(lhs.components) != len(rhs.components): + raise ValueError("Ciphertexts must have the same modulus and component count") + + @staticmethod + def add( + ciphertext1: BFVCiphertext, ciphertext2: BFVCiphertext, pk: BFVPublicKey + ) -> BFVCiphertext: + """Homomorphic polynomial addition modulo t.""" + BFV._pair(ciphertext1, ciphertext2, pk) + q = ciphertext1.modulus + components = tuple( + tuple((a + b) % q for a, b in zip(p1, p2, strict=True)) + for p1, p2 in zip(ciphertext1.components, ciphertext2.components, strict=True) + ) + return BFVCiphertext(components, q, pk["key_id"]) + + @staticmethod + def subtract( + ciphertext1: BFVCiphertext, ciphertext2: BFVCiphertext, pk: BFVPublicKey + ) -> BFVCiphertext: + """Homomorphic polynomial subtraction modulo t.""" + BFV._pair(ciphertext1, ciphertext2, pk) + q = ciphertext1.modulus + components = tuple( + tuple((a - b) % q for a, b in zip(p1, p2, strict=True)) + for p1, p2 in zip(ciphertext1.components, ciphertext2.components, strict=True) + ) + return BFVCiphertext(components, q, pk["key_id"]) + + @staticmethod + def add_plain( + ciphertext: BFVCiphertext, plaintext: BFVPolynomial, pk: BFVPublicKey + ) -> BFVCiphertext: + """Add a plaintext polynomial at the ciphertext's current modulus.""" + BFV.validate_ciphertext(ciphertext, pk) + plaintext = BFV._plaintext(plaintext, pk["params"]) + q = ciphertext.modulus + delta = q // pk["params"].plain_modulus + c0 = tuple( + (c + delta * m) % q for c, m in zip(ciphertext.components[0], plaintext, strict=True) + ) + return BFVCiphertext((c0, *ciphertext.components[1:]), q, pk["key_id"]) + + @staticmethod + def multiply_plain( + ciphertext: BFVCiphertext, plaintext: BFVPolynomial, pk: BFVPublicKey + ) -> BFVCiphertext: + """Multiply by an unscaled plaintext polynomial (e.g. a batch slot mask).""" + BFV.validate_ciphertext(ciphertext, pk) + plaintext = BFV._plaintext(plaintext, pk["params"]) + return BFVCiphertext( + tuple(_ring_product(c, plaintext, ciphertext.modulus) for c in ciphertext.components), + ciphertext.modulus, + pk["key_id"], + ) + + @staticmethod + def _switch( + poly: BFVPolynomial, switch_key: BFVSwitchKey, pk: BFVPublicKey + ) -> tuple[BFVPolynomial, BFVPolynomial]: + q, params = pk["q"], pk["params"] + n, bits = params.poly_modulus_degree, params.decomposition_bits + mask = (mpz(1) << bits) - 1 + result = [[mpz(0)] * n for _ in range(2)] + for i, pair in enumerate(switch_key): + digits = tuple((c >> (i * bits)) & mask for c in poly) + for k in range(2): + product = _ring_product(digits, pair[k], q) + result[k] = [(a + b) % q for a, b in zip(result[k], product, strict=True)] + return tuple(result[0]), tuple(result[1]) + + @staticmethod + def relinearize(ciphertext: BFVCiphertext, pk: BFVPublicKey) -> BFVCiphertext: + """Replace the s^2 component with a gadget key encryption under s.""" + BFV.validate_ciphertext(ciphertext, pk) + if len(ciphertext.components) == 2: + return ciphertext + if ciphertext.modulus != pk["q"]: + raise ValueError("Relinearization requires the original ciphertext modulus") + if not pk["relin_key"]: + raise ValueError("Public key has no relinearization key") + c0, c1, c2 = ciphertext.components + k0, k1 = BFV._switch(c2, pk["relin_key"], pk) + return BFVCiphertext( + ( + tuple((a + b) % pk["q"] for a, b in zip(c0, k0, strict=True)), + tuple((a + b) % pk["q"] for a, b in zip(c1, k1, strict=True)), + ), + pk["q"], + pk["key_id"], + ) + + @staticmethod + def multiply( + ciphertext1: BFVCiphertext, + ciphertext2: BFVCiphertext, + pk: BFVPublicKey, + *, + relinearize: bool = True, + ) -> BFVCiphertext: + """BFV multiply, scale-and-round, then optionally relinearize. + + The tensor products are computed over Z[X]/(X^N+1), not R_q. Reducing + them modulo q before rounding t*product/q would destroy the message. + """ + BFV._pair(ciphertext1, ciphertext2, pk) + if ciphertext1.modulus != pk["q"] or len(ciphertext1.components) != 2: + raise ValueError("Multiplication requires two components at the original modulus") + a0, a1 = ciphertext1.components + b0, b1 = ciphertext2.components + d0, d2 = _poly_product(a0, b0), _poly_product(a1, b1) + cross1 = _poly_product(a0, b1) + # Hamming uses squares: the two cross products are then identical. + cross2 = cross1 if ciphertext1 is ciphertext2 else _poly_product(a1, b0) + d1 = tuple(a + b for a, b in zip(cross1, cross2, strict=True)) + q, t = pk["q"], pk["params"].plain_modulus + result = BFVCiphertext( + tuple(tuple(_round_div(t * c, q) % q for c in poly) for poly in (d0, d1, d2)), + q, + pk["key_id"], + ) + return BFV.relinearize(result, pk) if relinearize else result + + @staticmethod + def xor( + ciphertext1: BFVCiphertext, ciphertext2: BFVCiphertext, pk: BFVPublicKey + ) -> BFVCiphertext: + """Slotwise XOR for batched *binary* inputs: (a-b)^2. No bit validation is possible here.""" + difference = BFV.subtract(ciphertext1, ciphertext2, pk) + return BFV.multiply(difference, difference, pk) + + @staticmethod + def rotate_rows(ciphertext: BFVCiphertext, steps: int, pk: BFVPublicKey) -> BFVCiphertext: + """Rotate both N/2-slot rows left by steps; negative steps rotate right.""" + BFV.validate_ciphertext(ciphertext, pk) + if ciphertext.modulus != pk["q"] or len(ciphertext.components) != 2: + raise ValueError("Rotation requires two components at the original modulus") + n, q = pk["params"].poly_modulus_degree, pk["q"] + exponent = pow(3, steps % (n // 2), 2 * n) + if exponent == 1: + return ciphertext + if exponent not in pk["galois_keys"]: + raise ValueError(f"Public key has no rotation key for steps={steps}") + c0, c1 = (_automorphism(c, exponent, q) for c in ciphertext.components) + k0, k1 = BFV._switch(c1, pk["galois_keys"][exponent], pk) + return BFVCiphertext( + (tuple((a + b) % q for a, b in zip(c0, k0, strict=True)), k1), q, pk["key_id"] + ) + + @staticmethod + def modulus_switch( + ciphertext: BFVCiphertext, modulus_bits: int, pk: BFVPublicKey + ) -> BFVCiphertext: + """Round each component from q to q' to compact a completed result. + + Further key switching/multiplication at q' is not implemented. Reducing + too far can make decryption fail; callers must validate their circuit. + """ + BFV.validate_ciphertext(ciphertext, pk) + if ( + not isinstance(modulus_bits, int) + or not pk["params"].plain_modulus.bit_length() + 8 + <= modulus_bits + <= ciphertext.modulus.bit_length() + ): + raise ValueError("Invalid target modulus bit length") + target = _coefficient_modulus(modulus_bits) + if target > ciphertext.modulus: + raise ValueError("Modulus switching cannot increase the modulus") + components = tuple( + tuple(_round_div(c * target, ciphertext.modulus) % target for c in poly) + for poly in ciphertext.components + ) + return BFVCiphertext(components, target, pk["key_id"]) + + @staticmethod + def noise_budget(ciphertext: BFVCiphertext, keys: BFVKeyPair) -> int: + """Local diagnostic: floor(log2(q / (2*||center(t*phase)||_inf))). + + Like decryption this needs the secret key. Do not expose it as a server + oracle. A positive result does not authenticate a ciphertext or prove + that an earlier operation did not wrap its noise. + """ + q, t = ciphertext.modulus, keys["pk"]["params"].plain_modulus + values = [(t * c) % q for c in BFV._phase(ciphertext, keys)] + largest = max(min(c, q - c) for c in values) + return ( + q.bit_length() - 1 if largest == 0 else max(0, int(q // (2 * largest)).bit_length() - 1) + ) + + @staticmethod + def ciphertext_to_ints(ciphertext: BFVCiphertext, pk: BFVPublicKey) -> EncryptedVector: + """Native wire v1: magic, version, N, q, key id, size, packed polynomials. + + Preserves the clients' list[int] interface. This is a BFV-specific format, + not compatible with the existing Paillier HTTP endpoint or SEAL bytes. + """ + BFV.validate_ciphertext(ciphertext, pk) + return [ + 0x58424656, + 1, + pk["params"].poly_modulus_degree, + int(ciphertext.modulus), + int(ciphertext.key_id, 16), + len(ciphertext.components), + *( + int(gmpy2.pack(list(c), ciphertext.modulus.bit_length())) + for c in ciphertext.components + ), + ] + + @staticmethod + def ciphertext_from_ints(values: Sequence[int | bytes], pk: BFVPublicKey) -> BFVCiphertext: + """Load native wire v1; byte entries use the same little endian convention as Paillier.""" + if len(values) not in (8, 9): + raise ValueError("Invalid BFV ciphertext framing") + if any(not isinstance(v, (Integral, bytes)) for v in values): + raise ValueError("Ciphertext entries must be nonnegative integers or bytes") + data = [int.from_bytes(v, "little") if isinstance(v, bytes) else int(v) for v in values] + if any(v < 0 for v in data): + raise ValueError("Ciphertext entries must be nonnegative") + n, q = pk["params"].poly_modulus_degree, mpz(data[3]) + if ( + data[:3] != [0x58424656, 1, n] + or data[4] != int(pk["key_id"], 16) + or data[5] != len(data) - 6 + ): + raise ValueError("Incompatible BFV ciphertext header or key") + if not pk["params"].plain_modulus < q <= pk["q"]: + raise ValueError("Invalid ciphertext modulus") + components = [] + for packed in data[6:]: + if packed.bit_length() > n * q.bit_length(): + raise ValueError("Packed polynomial exceeds N coefficients") + coefficients = gmpy2.unpack(mpz(packed), q.bit_length()) + coefficients.extend([mpz(0)] * (n - len(coefficients))) + components.append(tuple(coefficients)) + result = BFVCiphertext(tuple(components), q, pk["key_id"]) + BFV.validate_ciphertext(result, pk) + return result + + @staticmethod + def serialize_public_key(pk: BFVPublicKey) -> str: + """Export public encryption, relinearization and rotation keys as JSON.""" + + def poly(values: BFVPolynomial) -> list[str]: + return [format(c, "x") for c in values] + + def switch(values: BFVSwitchKey) -> list[list[list[str]]]: + return [[poly(b), poly(a)] for b, a in values] + + return json.dumps( + { + "version": 1, + "params": asdict(pk["params"]), + "key_id": pk["key_id"], + "b": poly(pk["b"]), + "a": poly(pk["a"]), + "relin_key": switch(pk["relin_key"]), + "galois_keys": {str(g): switch(key) for g, key in pk["galois_keys"].items()}, + }, + separators=(",", ":"), + ) + + @staticmethod + def deserialize_public_key(value: str) -> BFVPublicKey: + """Load public JSON, checking parameters, polynomial shapes and key identity. + + Serialization is intended for trusted local storage/experiments. Its + fingerprint detects mixups, not malicious modification or resource abuse. + """ + try: + data = json.loads(value) + if data["version"] != 1: + raise ValueError("Unsupported public key version") + params = BFVParameters(**data["params"]) + BFV.validate_parameters(params) + n, q = params.poly_modulus_degree, _coefficient_modulus(params.coeff_modulus_bits) + digits = ( + params.coeff_modulus_bits + params.decomposition_bits - 1 + ) // params.decomposition_bits + + def poly(values: Any) -> BFVPolynomial: + if ( + not isinstance(values, list) + or len(values) != n + or any(not isinstance(v, str) for v in values) + ): + raise ValueError("Invalid key polynomial") + result = tuple(mpz(c, 16) for c in values) + if any(not 0 <= c < q for c in result): + raise ValueError("Noncanonical key coefficient") + return result + + def switch(values: Any) -> BFVSwitchKey: + if not isinstance(values, list) or len(values) != digits: + raise ValueError("Invalid gadget key length") + if any(not isinstance(pair, list) or len(pair) != 2 for pair in values): + raise ValueError("Invalid gadget key pair") + return tuple((poly(pair[0]), poly(pair[1])) for pair in values) + + b, a = poly(data["b"]), poly(data["a"]) + identity = _key_id(params, b, a) + if data["key_id"] != identity: + raise ValueError("Public key fingerprint does not match") + galois_keys = {int(g): switch(key) for g, key in data["galois_keys"].items()} + if any(not 1 < g < 2 * n or g % 2 == 0 for g in galois_keys): + raise ValueError("Invalid Galois exponent") + return { + "params": params, + "q": q, + "key_id": identity, + "b": b, + "a": a, + "relin_key": switch(data["relin_key"]) if data["relin_key"] else (), + "galois_keys": galois_keys, + } + except (KeyError, TypeError, AttributeError, OverflowError) as exc: + raise ValueError("Malformed BFV public key") from exc + + @staticmethod + def serialize_secret_key(sk: BFVSecretKey) -> str: + """Export the secret separately; never include this in server setup.""" + return json.dumps( + {"version": 1, "key_id": sk["key_id"], "s": [int(c) for c in sk["s"]]}, + separators=(",", ":"), + ) + + @staticmethod + def deserialize_secret_key(value: str, pk: BFVPublicKey) -> BFVSecretKey: + """Load and check a ternary secret against its public RLWE sample.""" + try: + data = json.loads(value) + if data["version"] != 1 or data["key_id"] != pk["key_id"]: + raise ValueError("Secret key version or identity mismatch") + values = data["s"] + if not isinstance(values, list) or len(values) != pk["params"].poly_modulus_degree: + raise ValueError("Invalid secret key polynomial") + if any(type(c) is not int or c not in (-1, 0, 1) for c in values): + raise ValueError("Secret key coefficients must be ternary integers") + s = tuple(mpz(c) for c in values) + q = pk["q"] + product = _ring_product(pk["a"], tuple(c % q for c in s), q) + error = [(a + b) % q for a, b in zip(product, pk["b"], strict=True)] + if any(min(e, q - e) > pk["params"].error_eta for e in error): + raise ValueError("Secret key does not match the public key") + return {"s": s, "key_id": pk["key_id"]} + except (KeyError, TypeError, AttributeError, OverflowError) as exc: + raise ValueError("Malformed BFV secret key") from exc diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py b/src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py index 021b4bd..21a75bd 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py @@ -29,13 +29,13 @@ def key_gen(*args: Any, **kwargs: Any) -> T_KP: @staticmethod @abstractmethod - def encrypt(pt: T_PT, key: T_KEY) -> T_PT: + def encrypt(pt: T_PT, key: T_KEY) -> T_CT: """Encrypts a plain text with key""" pass @staticmethod @abstractmethod - def decrypt(ct: T_CT, key: T_KP) -> T_CT: + def decrypt(ct: T_CT, key: T_KP) -> T_PT: """decrypts a cipher""" pass diff --git a/src/xtrace_sdk/x_vec/utils/xtrace_types.py b/src/xtrace_sdk/x_vec/utils/xtrace_types.py index 01e879d..fff6946 100644 --- a/src/xtrace_sdk/x_vec/utils/xtrace_types.py +++ b/src/xtrace_sdk/x_vec/utils/xtrace_types.py @@ -1,4 +1,5 @@ from collections.abc import Sequence +from dataclasses import dataclass from typing import TypeAlias, TypedDict import gmpy2 @@ -63,6 +64,52 @@ class GoldwasserMicaliKeyPair(TypedDict): sk: GoldwasserMicaliSecretKey +# ── BFV ─────────────────────────────────────────────────────────────────── + +BFVPolynomial: TypeAlias = tuple[gmpy2.mpz, ...] +BFVSwitchKey: TypeAlias = tuple[tuple[BFVPolynomial, BFVPolynomial], ...] + + +@dataclass(frozen=True) +class BFVParameters: + """Native BFV arithmetic parameters; these are not a security certification.""" + + poly_modulus_degree: int = 8192 + plain_modulus: int = 65537 + coeff_modulus_bits: int = 180 + decomposition_bits: int = 30 + error_eta: int = 21 + + +@dataclass(frozen=True) +class BFVCiphertext: + """Polynomials modulo ``modulus`` in Z[X]/(X^N + 1), tied to one public key.""" + + components: tuple[BFVPolynomial, ...] + modulus: gmpy2.mpz + key_id: str + + +class BFVPublicKey(TypedDict): + params: BFVParameters + q: gmpy2.mpz + key_id: str + b: BFVPolynomial + a: BFVPolynomial + relin_key: BFVSwitchKey + galois_keys: dict[int, BFVSwitchKey] + + +class BFVSecretKey(TypedDict): + s: BFVPolynomial + key_id: str + + +class BFVKeyPair(TypedDict): + pk: BFVPublicKey + sk: BFVSecretKey + + # ── Encrypted vector type ────────────────────────────────────────────────── EncryptedVector: TypeAlias = list[int] diff --git a/tests/x_vec/test_bfv_client.py b/tests/x_vec/test_bfv_client.py new file mode 100644 index 0000000..24b2e77 --- /dev/null +++ b/tests/x_vec/test_bfv_client.py @@ -0,0 +1,211 @@ +"""Native BFV Hamming interface, packing, persistence and public-only evaluation.""" + +import json +import random +import subprocess +import sys +from pathlib import Path + +import pytest + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV +from xtrace_sdk.x_vec.crypto.hamming_client_base import HammingClientBase + + +def small_client(dimension: int = 3) -> BFVClient: + # Deliberately insecure parameters: fast tests, not a deployment preset. + return BFVClient(dimension, 32, 193, 120, 20, response_modulus_bits=40) + + +def public_server(client: BFVClient) -> BFVClient: + server = BFVClient(skip_key_gen=True) + server.load_config(json.loads(client.stringify_config())) + server.load_stringified_keys(client.stringify_pk()) + assert server.keys is None + return server + + +def data(count: int, dimension: int) -> tuple[list[list[int]], list[int], list[int]]: + rng = random.Random(97 + dimension + count) + query = [rng.randrange(2) for _ in range(dimension)] + vectors = [[rng.randrange(2) for _ in range(dimension)] for _ in range(count)] + if count: + vectors[0] = query[:] + if count > 1: + vectors[1] = [1 - bit for bit in query] + if count > 2: + vectors[-1] = query[:] + return vectors, query, [sum(x != y for x, y in zip(query, v, strict=True)) for v in vectors] + + +def test_base_interface_and_individual_batch() -> None: + client = small_client() + server = public_server(client) + assert isinstance(client, HammingClientBase) + vectors, query, expected = data(13, 3) + index, encrypted_query = client.encrypt_vec_batch(vectors), client.encrypt_vec_one(query) + assert len(index) == len(vectors) + response = [server.encode_hamming_server(encrypted_query, ct) for ct in index] + assert client.decode_hamming_client_batch(response) == expected + assert client.encrypt_vec_batch([]) == [] + assert client.decode_hamming_client_batch([]) == [] + with pytest.raises(RuntimeError, match="Secret key"): + server.decode_hamming_client_one(response[0]) + with pytest.raises(RuntimeError, match="Secret key"): + server.stringify_sk() + restored = BFVClient(skip_key_gen=True) + restored.load_config(json.loads(client.stringify_config())) + restored.load_stringified_keys(client.stringify_pk(), client.stringify_sk()) + assert restored.decode_hamming_client_batch(response) == expected + # Public-only clients can also encrypt, without access to a secret key. + fresh_query = server.encrypt_vec_one(query) + assert ( + client.decode_hamming_client_one(server.encode_hamming_server(fresh_query, index[0])) == 0 + ) + + +@pytest.mark.parametrize("dimension", [1, 3, 9, 16]) +def test_packed_tile_row_and_response_boundaries(dimension: int) -> None: + client = small_client(dimension) + server = public_server(client) + c = client.vectors_per_ciphertext + for count in sorted({0, 1, c - 1, c, c + 1, 31, 32, 33, 67}): + vectors, query, expected = data(count, dimension) + index = client.encrypt_vec_packed(vectors) + assert len(index) == (count + c - 1) // c + result = server.encode_hamming_server_packed(client.encrypt_vec_one(query), index, count) + assert len(result) == (count + 31) // 32 + distances = client.decode_hamming_client_packed(result, count) + assert distances == expected + assert ( + sorted(range(count), key=lambda i: (distances[i], i))[:3] + == sorted(range(count), key=lambda i: (expected[i], i))[:3] + ) + + +def test_partial_tile_padding_is_zero_and_compaction_preserves_distances() -> None: + client = small_client(9) + vectors, query, expected = data(13, 9) + query_ct, index = client.encrypt_vec_one(query), client.encrypt_vec_packed(vectors) + response = client.encode_hamming_server_packed(query_ct, index, 13, compact=False) + pk, keys = client._pk(), client._keys() + ct = BFV.ciphertext_from_ints(response[0], pk) + slots = BFV.batch_decode(BFV.decrypt(ct, keys), client.params) + used = set() + for pos, value in enumerate(expected): + tile, lane = divmod(pos, client.vectors_per_ciphertext) + row, column = divmod(lane, client.lanes_per_row) + slot = row * 16 + tile * client.lanes_per_row + column + used.add(slot) + assert slots[slot] == value + assert all(slots[i] == 0 for i in range(32) if i not in used) + compact = BFV.ciphertext_to_ints(BFV.modulus_switch(ct, 40, pk), pk) + assert client.decode_hamming_client_packed([compact], 13) == expected + assert sum(v.bit_length() for v in compact) < sum(v.bit_length() for v in response[0]) / 2 + + +def test_public_only_evaluator_in_separate_process(tmp_path: Path) -> None: + client = small_client(9) + vectors, query, expected = data(35, 9) + public = json.loads(client.stringify_pk()) + assert set(public) == {"version", "params", "key_id", "b", "a", "relin_key", "galois_keys"} + packet = { + "config": json.loads(client.stringify_config()), + "pk": client.stringify_pk(), + "query": [format(v, "x") for v in client.encrypt_vec_one(query)], + "index": [[format(v, "x") for v in ct] for ct in client.encrypt_vec_packed(vectors)], + "count": len(vectors), + } + path = tmp_path / "public.json" + path.write_text(json.dumps(packet)) + script = """ +import json, sys +from pathlib import Path +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +p = json.loads(Path(sys.argv[1]).read_text()) +s = BFVClient(skip_key_gen=True) +s.load_config(p['config']) +s.load_stringified_keys(p['pk']) +assert s.keys is None +assert not any('tenseal' in name or 'sealapi' in name for name in sys.modules) +response = s.encode_hamming_server_packed( + [int(v, 16) for v in p['query']], + [[int(v, 16) for v in row] for row in p['index']], p['count']) +print(json.dumps([[format(v, 'x') for v in row] for row in response])) +""" + run = subprocess.run( + [sys.executable, "-c", script, str(path)], + capture_output=True, + text=True, + check=True, + timeout=30, + ) + response = [[int(v, 16) for v in row] for row in json.loads(run.stdout)] + assert client.decode_hamming_client_packed(response, len(vectors)) == expected + + +def test_invalid_vectors_keys_configuration_and_counts() -> None: + client = small_client() + query = client.encrypt_vec_one([0, 1, 0]) + index = client.encrypt_vec_packed([[0, 1, 0]]) + for vector in ([0], [0, 1, 2], [0, -1, 1], [0, 1, 0.0]): + with pytest.raises(ValueError): + client.encrypt_vec_one(vector) + with pytest.raises(ValueError): + client.encrypt_vec_packed([vector]) + for count in (-1, 0, 9, True): + with pytest.raises(ValueError, match="vector_count"): + client.encode_hamming_server_packed(query, index, count) + with pytest.raises(ValueError, match="vector_count"): + client.decode_hamming_client_packed([], 1) + with pytest.raises(ValueError, match="configuration"): + BFVClient(skip_key_gen=True).load_stringified_keys(client.stringify_pk()) + with pytest.raises(ValueError, match="empty client"): + client.load_config({**json.loads(client.stringify_config()), "embed_len": 2}) + with pytest.raises(RuntimeError, match="Keys not initialized"): + BFVClient(skip_key_gen=True).encrypt_vec_one([0] * 512) + with pytest.raises(NotImplementedError, match="CPU"): + BFVClient(device="gpu") + assert not client.has_gpu() + for dimension in (0, -1, 17): + with pytest.raises(ValueError, match="embed_len"): + small_client(dimension) + + +@pytest.fixture(scope="module") +def default_case() -> tuple[BFVClient, list[list[int]], list[int], list[int]]: + client = BFVClient() + vectors, query, expected = data(33, 512) + index = client.encrypt_vec_packed(vectors) + result = client.encode_hamming_server_packed(client.encrypt_vec_one(query), index, len(vectors)) + distances = client.decode_hamming_client_packed(result, len(vectors)) + assert distances == expected + assert BFV.noise_budget(BFV.ciphertext_from_ints(result[0], client._pk()), client._keys()) >= 10 + assert len(index) == 3 and len(result) == 1 + return client, vectors, query, distances + + +def test_default_parameters_512_bit_vectors(default_case: tuple) -> None: + client, vectors, query, distances = default_case + assert client.device == "cpu" + assert distances[0] == 0 and distances[1] == 512 and distances[-1] == 0 + individual = client.encrypt_vec_batch(vectors[:2]) + encrypted_query = client.encrypt_vec_one(query) + response = [client.encode_hamming_server(encrypted_query, ct) for ct in individual] + assert client.decode_hamming_client_batch(response) == [0, 512] + + +def test_optional_seal_reference(default_case: tuple) -> None: + # SEAL is only an optional independent oracle in tests. Native code does not + # use its arithmetic, keys, serialization or slot ordering internally. + ts = pytest.importorskip( + "tenseal", reason="Optional sanity check: install experiments/bfv/requirements.txt" + ) + _, vectors, query, native_distances = default_case + context = ts.context(ts.SCHEME_TYPE.BFV, poly_modulus_degree=8192, plain_modulus=65537) + context.generate_galois_keys() + encrypted_query = ts.bfv_vector(context, query) + for i in (0, 1, 17, 32): + difference = encrypted_query - ts.bfv_vector(context, vectors[i]) + assert (difference * difference).sum().decrypt() == [native_distances[i]] diff --git a/tests/x_vec/test_bfv_encryption.py b/tests/x_vec/test_bfv_encryption.py new file mode 100644 index 0000000..aca08fb --- /dev/null +++ b/tests/x_vec/test_bfv_encryption.py @@ -0,0 +1,239 @@ +"""BFV algebra tests use deliberately insecure small rings to exercise boundaries cheaply.""" + +import json +import random +from dataclasses import replace + +import pytest +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _poly_product, _round_div +from xtrace_sdk.x_vec.utils.xtrace_types import BFVCiphertext, BFVKeyPair, BFVParameters + + +@pytest.fixture(scope="module") +def keys() -> BFVKeyPair: + return BFV.key_gen(16, 97, 100, 15, rotation_steps=[1, -1, 2, 4]) + + +def schoolbook(lhs: list[int], rhs: list[int]) -> list[int]: + """Independent O(N^2) integer oracle, including the sign from X^N = -1.""" + result = [0] * len(lhs) + for i, a in enumerate(lhs): + for j, b in enumerate(rhs): + result[(i + j) % len(lhs)] += a * b * (1 if i + j < len(lhs) else -1) + return result + + +@pytest.mark.parametrize("bits", [1, 17, 180, 512]) +def test_gmp_polynomial_product_against_integer_oracle(bits: int) -> None: + rng = random.Random(bits) + for n in (8, 16, 32): + for a, b in [ + ([0] * n, [0] * n), + ([2**bits - 1] * n, [2**bits - 1] * n), + ([rng.getrandbits(bits) for _ in range(n)], [rng.getrandbits(bits) for _ in range(n)]), + ]: + assert _poly_product(tuple(map(mpz, a)), tuple(map(mpz, b))) == tuple(schoolbook(a, b)) + + +def test_signed_exact_rounding() -> None: + assert [_round_div(mpz(v), mpz(2)) for v in range(-5, 6)] == [ + -2, + -2, + -1, + -1, + 0, + 0, + 1, + 1, + 2, + 2, + 3, + ] + value = (mpz(1) << 400) + 123 + assert _round_div(value * 7 - 3, mpz(7)) == value + assert _round_div(-value * 7 + 3, mpz(7)) == -value + + +def test_batching_is_ring_crt_not_just_a_roundtrip(keys: BFVKeyPair) -> None: + params = keys["pk"]["params"] + a, b = list(range(16)), [96 - i for i in range(16)] + pa, pb = BFV.batch_encode(a, params), BFV.batch_encode(b, params) + product = tuple(mpz(c % 97) for c in schoolbook(list(map(int, pa)), list(map(int, pb)))) + assert BFV.batch_decode(product, params) == [x * y % 97 for x, y in zip(a, b, strict=True)] + assert BFV.batch_decode(BFV.batch_encode([96, 0, 1], params), params) == [96, 0, 1] + [0] * 13 + assert BFV.batch_decode(BFV.batch_encode([], params), params) == [0] * 16 + + +def test_encrypt_decrypt_and_randomization(keys: BFVKeyPair) -> None: + pk = keys["pk"] + for values in ([0] * 16, [96] * 16, list(range(16))): + plaintext = BFV.batch_encode(values, pk["params"]) + ct1, ct2 = BFV.encrypt(plaintext, pk), BFV.encrypt(plaintext, pk) + assert ct1 != ct2 + assert BFV.decrypt(ct1, keys) == plaintext + assert BFV.decrypt(ct2, keys) == plaintext + assert BFV.noise_budget(ct1, keys) > 0 + + +def test_encrypted_arithmetic_and_relinearization(keys: BFVKeyPair) -> None: + pk, rng = keys["pk"], random.Random(103) + params = pk["params"] + for _ in range(5): + a, b = [rng.randrange(97) for _ in range(16)], [rng.randrange(97) for _ in range(16)] + pa, pb = BFV.batch_encode(a, params), BFV.batch_encode(b, params) + ca, cb = BFV.encrypt(pa, pk), BFV.encrypt(pb, pk) + operations = [ + (BFV.add(ca, cb, pk), [(x + y) % 97 for x, y in zip(a, b, strict=True)]), + (BFV.subtract(ca, cb, pk), [(x - y) % 97 for x, y in zip(a, b, strict=True)]), + (BFV.add_plain(ca, pb, pk), [(x + y) % 97 for x, y in zip(a, b, strict=True)]), + (BFV.multiply_plain(ca, pb, pk), [x * y % 97 for x, y in zip(a, b, strict=True)]), + ] + quadratic = BFV.multiply(ca, cb, pk, relinearize=False) + linear = BFV.relinearize(quadratic, pk) + assert len(quadratic.components) == 3 + assert len(linear.components) == 2 + expected = [x * y % 97 for x, y in zip(a, b, strict=True)] + operations.extend( + [(quadratic, expected), (linear, expected), (BFV.multiply(ca, cb, pk), expected)] + ) + # A second multiplication exercises the scale and relinearization noise + # together; it would fail if the tensor product were reduced prematurely. + operations.append( + (BFV.multiply(linear, ca, pk), [x * y * x % 97 for x, y in zip(a, b, strict=True)]) + ) + for ct, expected in operations: + assert BFV.batch_decode(BFV.decrypt(ct, keys), params) == expected + + +@pytest.mark.parametrize("steps", [0, 1, -1, 2, 4, 9]) +def test_row_rotations(keys: BFVKeyPair, steps: int) -> None: + pk = keys["pk"] + a = list(range(16)) + ct = BFV.encrypt(BFV.batch_encode(a, pk["params"]), pk) + expected = [a[row * 8 + (column + steps) % 8] for row in range(2) for column in range(8)] + assert ( + BFV.batch_decode(BFV.decrypt(BFV.rotate_rows(ct, steps, pk), keys), pk["params"]) + == expected + ) + + +def test_binary_xor_and_terminal_modulus_switch(keys: BFVKeyPair) -> None: + pk, params = keys["pk"], keys["pk"]["params"] + a, b = [0, 0, 1, 1] * 4, [0, 1, 0, 1] * 4 + ca, cb = (BFV.encrypt(BFV.batch_encode(values, params), pk) for values in (a, b)) + full = BFV.xor(ca, cb, pk) + compact = BFV.modulus_switch(full, 40, pk) + assert BFV.batch_decode(BFV.decrypt(compact, keys), params) == [ + x ^ y for x, y in zip(a, b, strict=True) + ] + assert BFV.decrypt(full, keys) == BFV.decrypt(compact, keys) + assert compact.modulus.bit_length() == 40 + assert BFV.noise_budget(full, keys) > BFV.noise_budget(compact, keys) > 0 + with pytest.raises(ValueError, match="original modulus"): + BFV.multiply(compact, compact, pk) + with pytest.raises(ValueError, match="original modulus"): + BFV.rotate_rows(compact, 1, pk) + with pytest.raises(ValueError): + BFV.add(full, compact, pk) + with pytest.raises(ValueError): + BFV.modulus_switch(compact, 50, pk) + + +def test_key_and_ciphertext_serialization(keys: BFVKeyPair) -> None: + pk = BFV.deserialize_public_key(BFV.serialize_public_key(keys["pk"])) + sk = BFV.deserialize_secret_key(BFV.serialize_secret_key(keys["sk"]), pk) + assert {"pk": pk, "sk": sk} == keys + for values in ([0] * 16, [96] * 16): + ct = BFV.encrypt(BFV.batch_encode(values, pk["params"]), pk) + for candidate in ( + ct, + BFV.multiply(ct, ct, pk, relinearize=False), + BFV.modulus_switch(ct, 40, pk), + ): + wire = BFV.ciphertext_to_ints(candidate, pk) + assert BFV.ciphertext_from_ints(wire, pk) == candidate + as_bytes = [v.to_bytes((v.bit_length() + 7) // 8, "little") for v in wire] + assert BFV.ciphertext_from_ints(as_bytes, pk) == candidate + + +def test_reject_missing_evaluation_keys() -> None: + keys = BFV.key_gen(16, 97, 100, 15, relinearization=False) + pk = keys["pk"] + ct = BFV.encrypt(BFV.batch_encode([1], pk["params"]), pk) + assert ( + BFV.batch_decode( + BFV.decrypt(BFV.multiply(ct, ct, pk, relinearize=False), keys), pk["params"] + ) + == [1] + [0] * 15 + ) + with pytest.raises(ValueError, match="relinearization key"): + BFV.multiply(ct, ct, pk) + with pytest.raises(ValueError, match="rotation key"): + BFV.rotate_rows(ct, 1, pk) + + +def test_reject_mixed_keys_and_bad_wire(keys: BFVKeyPair) -> None: + pk = keys["pk"] + ct = BFV.encrypt(BFV.batch_encode([1], pk["params"]), pk) + other = BFV.key_gen(16, 97, 100, 15) + with pytest.raises(ValueError, match="different BFV key"): + BFV.decrypt(ct, other) + with pytest.raises(ValueError): + BFV.decrypt(replace(ct, components=(ct.components[0][:-1], ct.components[1])), keys) + wire = BFV.ciphertext_to_ints(ct, pk) + malformed = [ + wire[:-1], + [-1, *wire[1:]], + [*wire[:1], 2, *wire[2:]], + [*wire[:3], 97, *wire[4:]], + [*wire[:4], 0, *wire[5:]], + [*wire[:6], 1 << (16 * 100), wire[7]], + [*wire[:6], int(pk["q"]), wire[7]], + ] + for value in malformed: + with pytest.raises(ValueError): + BFV.ciphertext_from_ints(value, pk) + + +def test_reject_corrupted_keys(keys: BFVKeyPair) -> None: + data = json.loads(BFV.serialize_public_key(keys["pk"])) + data["b"][0] = "0" if data["b"][0] != "0" else "1" + with pytest.raises(ValueError, match="fingerprint"): + BFV.deserialize_public_key(json.dumps(data)) + sk = json.loads(BFV.serialize_secret_key(keys["sk"])) + sk["s"][0] = 0 if sk["s"][0] else 1 + with pytest.raises(ValueError, match="does not match"): + BFV.deserialize_secret_key(json.dumps(sk), keys["pk"]) + for value in ("{}", "null", "[]"): + with pytest.raises(ValueError): + BFV.deserialize_public_key(value) + + +@pytest.mark.parametrize( + "changes", + [ + {"poly_modulus_degree": 12}, + {"poly_modulus_degree": 4}, + {"plain_modulus": 33}, + {"plain_modulus": 101}, + {"coeff_modulus_bits": 12}, + {"decomposition_bits": 0}, + {"error_eta": 0}, + {"error_eta": 65}, + {"poly_modulus_degree": True}, + ], +) +def test_reject_invalid_parameters(changes: dict[str, int]) -> None: + with pytest.raises(ValueError): + BFV.validate_parameters(replace(BFVParameters(16, 97, 100, 15), **changes)) + + +def test_reject_invalid_plaintext(keys: BFVKeyPair) -> None: + pk = keys["pk"] + for slots in ([97], [-1], [0] * 17, [1.5]): + with pytest.raises(ValueError): + BFV.batch_encode(slots, pk["params"]) + with pytest.raises(ValueError, match="exactly N"): + BFV.encrypt((mpz(0),), pk) From ff8d5a29a58874e6498e1cdae6b3fcde21e19a5d Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Thu, 10 Sep 2026 18:29:28 -0400 Subject: [PATCH 03/12] Optimize native BFV server evaluation with cached GMP products --- CHANGELOG.md | 1 + benchmarks/bfv_client_matrix.py | 5 +- benchmarks/bfv_server.py | 220 ++++++++++++++++++ .../results/native_bfv_server_1024.json | 128 ++++++++++ benchmarks/results/native_bfv_server_32.json | 142 +++++++++++ .../results/native_bfv_server_profile_32.txt | 75 ++++++ docs/api_rst/xtrace_sdk.crypto.encryption.rst | 8 + docs/research/native-bfv.md | 146 +++++++++++- src/xtrace_sdk/x_vec/crypto/bfv_client.py | 39 +++- .../x_vec/crypto/encryption/bfv_evaluator.py | 214 +++++++++++++++++ tests/x_vec/test_bfv_client.py | 41 +++- tests/x_vec/test_bfv_evaluator.py | 145 ++++++++++++ 12 files changed, 1143 insertions(+), 21 deletions(-) create mode 100644 benchmarks/bfv_server.py create mode 100644 benchmarks/results/native_bfv_server_1024.json create mode 100644 benchmarks/results/native_bfv_server_32.json create mode 100644 benchmarks/results/native_bfv_server_profile_32.txt create mode 100644 src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py create mode 100644 tests/x_vec/test_bfv_evaluator.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1552783..0fd15f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ All notable changes to this project will be documented in this file. ### Changed +- Native `BFVClient` server evaluation now defaults to cached GMP key switching: gadget products are accumulated and folded before unpacking, with fewer coefficient checks in Python. `server_backend="reference"` retains the original arithmetic. Parameters, ciphertexts and serialized keys remain compatible. A paired server benchmark checks identical ciphertext outputs and separates first-query from warm-query timings. - **Device selection moved from the `DEVICE` environment variable to a `device=` constructor keyword** on `PaillierClient`, `PaillierLookupClient`, `ExecutionContext.create`, `ExecutionContext.load_from_disk`, and `ExecutionContext.load_from_remote`. The new default is `device="auto"`: clients probe for the GPU extension at construction time and fall back to CPU silently when no GPU is present. If a GPU is detected on the host (`/dev/nvidia0`, `nvidia-smi`, or `CUDA_VISIBLE_DEVICES`) but the extension fails to load, a warning is emitted instead of a silent CPU fallback so misconfigured GPU hosts don't quietly run on CPU. `device="gpu"` raises if the extension is unavailable; `device="cpu"` skips probing entirely. - The `DEVICE` environment variable is no longer read. Existing callers must migrate to the `device=` kwarg. diff --git a/benchmarks/bfv_client_matrix.py b/benchmarks/bfv_client_matrix.py index 5acc275..de6fe03 100644 --- a/benchmarks/bfv_client_matrix.py +++ b/benchmarks/bfv_client_matrix.py @@ -106,6 +106,7 @@ def benchmark( decomposition_bits=args.decomposition_bits, response_modulus_bits=args.response_modulus_bits, device="cpu", + server_backend=args.server_backend, ) ) elif variant == "paillier-cpu": @@ -122,7 +123,7 @@ def benchmark( if native: def load_server() -> BFVClient: - result = BFVClient(skip_key_gen=True) + result = BFVClient(skip_key_gen=True, server_backend=args.server_backend) result.load_config(config) result.load_stringified_keys(public_json) assert result.keys is None @@ -203,6 +204,7 @@ def evaluate() -> list[list[int]]: ) return { "variant": variant, + "server_backend": args.server_backend if native else None, "vectors": n, "embed_len": args.embed_len, "config": config, @@ -234,6 +236,7 @@ def main() -> None: parser.add_argument("--decomposition-bits", type=int, default=30) parser.add_argument("--response-modulus-bits", type=int, default=50) parser.add_argument("--no-compact", action="store_true") + parser.add_argument("--server-backend", choices=("optimized", "reference"), default="optimized") parser.add_argument("--seed", type=int, default=1337) parser.add_argument( "--repeats", diff --git a/benchmarks/bfv_server.py b/benchmarks/bfv_server.py new file mode 100644 index 0000000..151158c --- /dev/null +++ b/benchmarks/bfv_server.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python3 +"""Compare native BFV server backends on identical keys, queries, and index bytes. + +The first search includes lazy evaluation-key preparation. Subsequent searches +reuse the same public-only server. No SEAL, network, GPU or secret server key is +required. Run without concurrent tests/benchmarks for useful timing comparisons. +""" + +from __future__ import annotations + +import argparse +import cProfile +import hashlib +import json +import platform +import statistics +import subprocess +import sys +from datetime import UTC, datetime +from functools import partial +from pathlib import Path +from typing import Any + +import gmpy2 + +from bfv_client_matrix import REPO_ROOT, make_data, packet, time_call, unpack_packet +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVServerBackend + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--num-vectors", type=int, default=1024) + parser.add_argument("--embed-len", type=int, default=512) + parser.add_argument("--poly-modulus-degree", type=int, default=8192) + parser.add_argument("--plain-modulus", type=int, default=65537) + parser.add_argument("--coeff-modulus-bits", type=int, default=180) + parser.add_argument("--decomposition-bits", type=int, default=30) + parser.add_argument("--response-modulus-bits", type=int, default=50) + parser.add_argument("--seed", type=int, default=1337) + parser.add_argument( + "--repeats", + type=int, + default=3, + help="Searches per backend, including the first cold search", + ) + parser.add_argument("--json-out", type=Path) + parser.add_argument( + "--profile-dir", + type=Path, + help="Run one extra warm search per backend under cProfile, outside reported timings", + ) + args = parser.parse_args() + if args.num_vectors < 1 or args.repeats < 1 or args.embed_len < 1: + parser.error("num-vectors, embed-len and repeats must be positive") + vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) + print("Generating one key set and encrypting the shared input", file=sys.stderr, flush=True) + setup: dict[str, float] = {} + setup["key_generation_s"], client = time_call( + lambda: BFVClient( + embed_len=args.embed_len, + poly_modulus_degree=args.poly_modulus_degree, + plain_modulus=args.plain_modulus, + coeff_modulus_bits=args.coeff_modulus_bits, + decomposition_bits=args.decomposition_bits, + response_modulus_bits=args.response_modulus_bits, + device="cpu", + ) + ) + setup["index_encrypt_s"], index = time_call(lambda: client.encrypt_vec_packed(vectors)) + setup["query_encrypt_s"], encrypted_query = time_call(lambda: client.encrypt_vec_one(query)) + index_wire, query_wire = packet(index, len(vectors)), packet([encrypted_query], 1) + server_index, server_query = unpack_packet(index_wire), unpack_packet(query_wire)[0] + setup["public_key_export_s"], public_json = time_call(client.stringify_pk) + config = json.loads(client.stringify_config()) + backends: tuple[BFVServerBackend, ...] = ("reference", "optimized") + servers = {} + for backend in backends: + server = BFVClient(skip_key_gen=True, server_backend=backend, **config) + setup[f"{backend}_public_key_import_s"], _ = time_call( + partial(server.load_stringified_keys, public_json) + ) + assert server.keys is None + servers[backend] = server + key_bytes = len(public_json.encode()) + del public_json + + def evaluate(backend: BFVServerBackend) -> list[list[int]]: + return servers[backend].encode_hamming_server_packed( + server_query, server_index, len(vectors) + ) + + timings: dict[str, list[float]] = {backend: [] for backend in backends} + order = [] + reference_response: list[list[int]] | None = None + for repeat in range(args.repeats): + # Alternate ordering to reduce systematic first/last-run effects. + for backend in backends if repeat % 2 == 0 else backends[::-1]: + print(f"{backend}: search {repeat + 1}/{args.repeats}", file=sys.stderr, flush=True) + elapsed, response = time_call(partial(evaluate, backend)) + timings[backend].append(elapsed) + order.append({"backend": backend, "repeat": repeat, "server_compute_s": elapsed}) + if reference_response is None: + reference_response = response + assert response == reference_response, "Backend ciphertexts differ" + assert client.decode_hamming_client_packed(response, len(vectors)) == expected + print( + f"{backend}: {elapsed:.6f} s; ciphertexts and all distances verified", + file=sys.stderr, + flush=True, + ) + + assert reference_response is not None + response_wire = packet(reference_response, len(vectors)) + # Profiles are separate extra runs: cProfile substantially distorts Python + # loop timings and must not be used to calculate benchmark speedups. + if args.profile_dir: + args.profile_dir.mkdir(parents=True, exist_ok=True) + for backend in backends: + profiler = cProfile.Profile() + response = profiler.runcall(evaluate, backend) + profiler.dump_stats(str(args.profile_dir / f"{backend}.prof")) + assert response == reference_response + + summary = { + backend: { + "first_search_s": values[0], + "warm_searches_s": values[1:], + "warm_median_s": statistics.median(values[1:]) if len(values) > 1 else None, + } + for backend, values in timings.items() + } + source_paths = [ + Path(__file__).resolve(), + REPO_ROOT / "benchmarks/bfv_client_matrix.py", + REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_client.py", + REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py", + REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py", + REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", + ] + output: dict[str, Any] = { + "environment": { + "utc": datetime.now(UTC).isoformat(), + "python": platform.python_version(), + "platform": platform.platform(), + "gmpy2": gmpy2.version(), + "gmp": gmpy2.mp_version(), + "revision": subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip(), + "working_tree_dirty": bool( + subprocess.run( + ["git", "status", "--porcelain"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip() + ), + }, + "command": sys.argv, + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for both backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": config, + "seed": args.seed, + "vectors": len(vectors), + "setup_timings": setup, + "runs": order, + "summary": summary, + "first_search_speedup": timings["reference"][0] / timings["optimized"][0], + "warm_median_speedup": statistics.median(timings["reference"][1:]) + / statistics.median(timings["optimized"][1:]) + if args.repeats > 1 + else None, + "sizes": { + "encrypted_index_bytes": len(index_wire), + "public_keys_bytes": key_bytes, + "query_bytes": len(query_wire), + "response_bytes": len(response_wire), + "query_plus_response_bytes": len(query_wire) + len(response_wire), + }, + "optimized_cached_switch_keys": len(servers["optimized"]._evaluator()._switch_keys), + "optimized_packed_cache_payload_bytes": sum( + sys.getsizeof(value) + for key in servers["optimized"]._evaluator()._switch_keys.values() + for value in ( + key.low_mask, + key.bias_coefficient, + key.bias, + *(v for pair in key.pairs for v in pair), + ) + ), + "cache_size_note": "sys.getsizeof of packed GMP values, including folding constants; additional to the existing public key. Excludes small Python containers and transient arithmetic allocations; not peak RSS.", + "minimum_response_noise_budget_bits": min( + BFV.noise_budget(BFV.ciphertext_from_ints(ct, client._pk()), client._keys()) + for ct in reference_response + ), + "identical_ciphertexts": True, + "all_distances_correct": True, + "top3": [ + {"index": i, "distance": expected[i]} + for i in sorted(range(len(vectors)), key=lambda i: (expected[i], i))[:3] + ], + "source_sha256": { + str(path.relative_to(REPO_ROOT)): hashlib.sha256(path.read_bytes()).hexdigest() + for path in source_paths + }, + } + print(json.dumps(output, indent=2)) + if args.json_out: + args.json_out.parent.mkdir(parents=True, exist_ok=True) + args.json_out.write_text(json.dumps(output, indent=2) + "\n") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/results/native_bfv_server_1024.json b/benchmarks/results/native_bfv_server_1024.json new file mode 100644 index 0000000..0a3c7ca --- /dev/null +++ b/benchmarks/results/native_bfv_server_1024.json @@ -0,0 +1,128 @@ +{ + "environment": { + "utc": "2026-09-10T22:25:48.950450+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "cf454e5eb6e053822501645bd44c6f23aa832a19", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "1024", + "--repeats", + "3", + "--json-out", + "benchmarks/results/native_bfv_server_1024.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for both backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "response_modulus_bits": 50, + "device": "cpu" + }, + "seed": 1337, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 4.636117450892925, + "index_encrypt_s": 7.034667205996811, + "query_encrypt_s": 0.10684434906579554, + "public_key_export_s": 0.8335895461495966, + "reference_public_key_import_s": 0.850766752846539, + "optimized_public_key_import_s": 0.8723104181699455 + }, + "runs": [ + { + "backend": "reference", + "repeat": 0, + "server_compute_s": 121.56445405795239 + }, + { + "backend": "optimized", + "repeat": 0, + "server_compute_s": 60.35474809096195 + }, + { + "backend": "optimized", + "repeat": 1, + "server_compute_s": 60.588989215902984 + }, + { + "backend": "reference", + "repeat": 1, + "server_compute_s": 121.66320819617249 + }, + { + "backend": "reference", + "repeat": 2, + "server_compute_s": 114.82569716288708 + }, + { + "backend": "optimized", + "repeat": 2, + "server_compute_s": 56.66450631082989 + } + ], + "summary": { + "reference": { + "first_search_s": 121.56445405795239, + "warm_searches_s": [ + 121.66320819617249, + 114.82569716288708 + ], + "warm_median_s": 118.24445267952979 + }, + "optimized": { + "first_search_s": 60.35474809096195, + "warm_searches_s": [ + 60.588989215902984, + 56.66450631082989 + ], + "warm_median_s": 58.62674776336644 + } + }, + "first_search_speedup": 2.014165544602058, + "warm_median_speedup": 2.0169028163867573, + "sizes": { + "encrypted_index_bytes": 23598492, + "public_keys_bytes": 85603614, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "optimized_cached_switch_keys": 16, + "optimized_packed_cache_payload_bytes": 52083728, + "cache_size_note": "sys.getsizeof of packed GMP values, including folding constants; additional to the existing public key. Excludes small Python containers and transient arithmetic allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "c0523cfbd4100657fc7f9eae9ac117d5fa86fc57e9983a7e2dd840920a2c2298", + "benchmarks/bfv_client_matrix.py": "d7c4c4242ad2cc537b8c39cf990944b83c3a2f178ce317e23dd415e5c12f7283", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "1a9927c6e15937ee39e6e84131890e249e14352795c21e8b8930cfe1007dd674", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "6f03ac75f42218170da2bd785605ee394940a202afdd5471545a8ae3fdc8c3d8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "6e2c84e7882bb86f8e623c2eb2d104e0d4cab85090aed17840adc0809a5cd9a3", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "7a10c977ee648eb1975286b313f009b3508183efe9e942cd710ae6ad5ae279a3" + } +} diff --git a/benchmarks/results/native_bfv_server_32.json b/benchmarks/results/native_bfv_server_32.json new file mode 100644 index 0000000..207bd25 --- /dev/null +++ b/benchmarks/results/native_bfv_server_32.json @@ -0,0 +1,142 @@ +{ + "environment": { + "utc": "2026-09-10T22:15:47.796934+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "cf454e5eb6e053822501645bd44c6f23aa832a19", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "32", + "--repeats", + "4", + "--profile-dir", + "/tmp/xtrace-bfv-server-profiles", + "--json-out", + "/tmp/xtrace-bfv-server-32.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for both backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "response_modulus_bits": 50, + "device": "cpu" + }, + "seed": 1337, + "vectors": 32, + "setup_timings": { + "key_generation_s": 4.557773574022576, + "index_encrypt_s": 0.21618583402596414, + "query_encrypt_s": 0.10164761217311025, + "public_key_export_s": 0.8099749418906868, + "reference_public_key_import_s": 0.8771210089325905, + "optimized_public_key_import_s": 0.8737351400777698 + }, + "runs": [ + { + "backend": "reference", + "repeat": 0, + "server_compute_s": 3.3908394610043615 + }, + { + "backend": "optimized", + "repeat": 0, + "server_compute_s": 1.877934766933322 + }, + { + "backend": "optimized", + "repeat": 1, + "server_compute_s": 1.8629616550169885 + }, + { + "backend": "reference", + "repeat": 1, + "server_compute_s": 3.6480522511992604 + }, + { + "backend": "reference", + "repeat": 2, + "server_compute_s": 3.6024102109950036 + }, + { + "backend": "optimized", + "repeat": 2, + "server_compute_s": 1.865732975071296 + }, + { + "backend": "optimized", + "repeat": 3, + "server_compute_s": 1.871090185130015 + }, + { + "backend": "reference", + "repeat": 3, + "server_compute_s": 3.708768686046824 + } + ], + "summary": { + "reference": { + "first_search_s": 3.3908394610043615, + "warm_searches_s": [ + 3.6480522511992604, + 3.6024102109950036, + 3.708768686046824 + ], + "warm_median_s": 3.6480522511992604 + }, + "optimized": { + "first_search_s": 1.877934766933322, + "warm_searches_s": [ + 1.8629616550169885, + 1.865732975071296, + 1.871090185130015 + ], + "warm_median_s": 1.865732975071296 + } + }, + "first_search_speedup": 1.8056215373985656, + "warm_median_speedup": 1.9552917271346697, + "sizes": { + "encrypted_index_bytes": 737480, + "public_keys_bytes": 85603818, + "query_bytes": 368754, + "response_bytes": 102494, + "query_plus_response_bytes": 471248 + }, + "optimized_cached_switch_keys": 11, + "optimized_packed_cache_payload_bytes": 35807568, + "cache_size_note": "sys.getsizeof of packed GMP values, including folding constants; additional to the existing public key. Excludes small Python containers and transient arithmetic allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 31, + "distance": 0 + }, + { + "index": 19, + "distance": 241 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "c0523cfbd4100657fc7f9eae9ac117d5fa86fc57e9983a7e2dd840920a2c2298", + "benchmarks/bfv_client_matrix.py": "d7c4c4242ad2cc537b8c39cf990944b83c3a2f178ce317e23dd415e5c12f7283", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "1a9927c6e15937ee39e6e84131890e249e14352795c21e8b8930cfe1007dd674", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "6f03ac75f42218170da2bd785605ee394940a202afdd5471545a8ae3fdc8c3d8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "6e2c84e7882bb86f8e623c2eb2d104e0d4cab85090aed17840adc0809a5cd9a3", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "7a10c977ee648eb1975286b313f009b3508183efe9e942cd710ae6ad5ae279a3" + } +} diff --git a/benchmarks/results/native_bfv_server_profile_32.txt b/benchmarks/results/native_bfv_server_profile_32.txt new file mode 100644 index 0000000..d5ba38b --- /dev/null +++ b/benchmarks/results/native_bfv_server_profile_32.txt @@ -0,0 +1,75 @@ +Native BFV server profiles, 32 vectors x 512 bits, default parameters. +Additional warm searches from bfv_server.py, after four unprofiled searches per backend. +Timing comparisons are in native_bfv_server_32.json; cProfile timings are distorted by instrumentation. + +reference +Thu Sep 10 18:15:45 2026 /tmp/xtrace-bfv-server-profiles/reference.prof + + 19838069 function calls in 5.637 seconds + + Ordered by: internal time + List reduced from 72 to 25 due to restriction <25> + + ncalls tottime percall cumtime percall filename:lineno(function) + 262 1.335 0.005 2.976 0.011 bfv.py:39(_poly_product) + 1228950 0.468 0.000 1.338 0.000 bfv.py:300() + 21 0.431 0.021 3.992 0.190 bfv.py:411(_switch) + 2097408 0.338 0.000 0.338 0.000 bfv.py:60() + 2146566 0.303 0.000 0.303 0.000 bfv.py:56() + 268 0.271 0.001 0.271 0.001 {built-in method gmpy2.gmpy2.unpack} + 4292623 0.269 0.000 0.269 0.000 {method 'bit_length' of 'gmpy2.mpz' objects} + 524 0.249 0.000 0.968 0.002 {built-in method builtins.max} + 1245208 0.239 0.000 0.561 0.000 {built-in method _abc._abc_instancecheck} + 2146566 0.220 0.000 0.354 0.000 bfv.py:49() + 256 0.206 0.001 3.442 0.013 bfv.py:59(_ring_product) + 1245208 0.202 0.000 0.202 0.000 {built-in method _abc._abc_subclasscheck} + 1245233 0.195 0.000 0.881 0.000 {built-in method builtins.isinstance} + 1032318 0.176 0.000 0.176 0.000 bfv.py:420() + 1245208 0.125 0.000 0.686 0.000 :117(__instancecheck__) + 1245208 0.120 0.000 0.322 0.000 :121(__subclasscheck__) + 522 0.102 0.000 0.102 0.000 {built-in method gmpy2.gmpy2.pack} + 158 0.074 0.000 1.427 0.009 {built-in method builtins.any} + 311334 0.072 0.000 0.072 0.000 bfv.py:366() + 38 0.064 0.002 0.064 0.002 bfv.py:63(_automorphism) + 155667 0.040 0.000 0.040 0.000 bfv.py:501() + 19 0.019 0.001 4.117 0.217 bfv_evaluator.py:196(rotate_rows) + 2 0.019 0.010 5.502 2.751 bfv_client.py:237(_distance_tile) + 57 0.016 0.000 0.089 0.002 bfv.py:365() + 65536 0.016 0.000 0.016 0.000 bfv.py:34(_round_div) + + + +optimized +Thu Sep 10 18:15:47 2026 /tmp/xtrace-bfv-server-profiles/optimized.prof + + 3581974 function calls in 2.220 seconds + + Ordered by: internal time + List reduced from 73 to 25 due to restriction <25> + + ncalls tottime percall cumtime percall filename:lineno(function) + 21 1.293 0.062 1.426 0.068 bfv_evaluator.py:53(apply) + 1048704 0.102 0.000 0.102 0.000 bfv_evaluator.py:110() + 344106 0.071 0.000 0.071 0.000 bfv_evaluator.py:67() + 180246 0.065 0.000 0.193 0.000 bfv.py:300() + 158 0.061 0.000 0.371 0.002 {built-in method builtins.any} + 10 0.056 0.006 0.122 0.012 bfv.py:39(_poly_product) + 38 0.055 0.001 0.055 0.001 bfv.py:63(_automorphism) + 311334 0.050 0.000 0.050 0.000 bfv_evaluator.py:139() + 155667 0.039 0.000 0.039 0.000 bfv_evaluator.py:213() + 196632 0.038 0.000 0.090 0.000 {built-in method _abc._abc_instancecheck} + 58 0.035 0.001 0.035 0.001 {built-in method gmpy2.gmpy2.unpack} + 196632 0.032 0.000 0.032 0.000 {built-in method _abc._abc_subclasscheck} + 196657 0.031 0.000 0.140 0.000 {built-in method builtins.isinstance} + 148 0.028 0.000 0.057 0.000 {built-in method builtins.max} + 2 0.027 0.013 2.045 1.022 bfv_client.py:237(_distance_tile) + 144 0.025 0.000 0.025 0.000 {built-in method gmpy2.gmpy2.pack} + 196632 0.020 0.000 0.109 0.000 :117(__instancecheck__) + 42 0.019 0.000 0.111 0.003 bfv_evaluator.py:61(finish) + 130 0.019 0.000 0.019 0.000 {built-in method builtins.min} + 196632 0.019 0.000 0.051 0.000 :121(__subclasscheck__) + 57 0.016 0.000 0.066 0.001 bfv_evaluator.py:138() + 65536 0.016 0.000 0.016 0.000 bfv.py:34(_round_div) + 81930 0.013 0.000 0.013 0.000 bfv.py:56() + 163855 0.011 0.000 0.011 0.000 {method 'bit_length' of 'gmpy2.mpz' objects} + 81930 0.009 0.000 0.014 0.000 bfv.py:49() diff --git a/docs/api_rst/xtrace_sdk.crypto.encryption.rst b/docs/api_rst/xtrace_sdk.crypto.encryption.rst index 8c33cfd..3ad0f74 100644 --- a/docs/api_rst/xtrace_sdk.crypto.encryption.rst +++ b/docs/api_rst/xtrace_sdk.crypto.encryption.rst @@ -12,6 +12,14 @@ xtrace\_sdk.crypto.encryption.bfv module :show-inheritance: :undoc-members: +xtrace\_sdk.crypto.encryption.bfv\_evaluator module +------------------------------------------------- + +.. automodule:: xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator + :members: + :show-inheritance: + :undoc-members: + xtrace\_sdk.crypto.encryption.aes module ---------------------------------------- diff --git a/docs/research/native-bfv.md b/docs/research/native-bfv.md index 2e44003..0237b64 100644 --- a/docs/research/native-bfv.md +++ b/docs/research/native-bfv.md @@ -16,11 +16,14 @@ All paths are relative to the repository root. | File | Responsibility | | --- | --- | | `src/xtrace_sdk/x_vec/crypto/encryption/bfv.py` | Polynomial arithmetic, key generation, encryption/decryption, batch encoding, homomorphic operations, evaluation keys, modulus switching, serialization | +| `src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py` | Cached public-key server arithmetic, with optimized and reference backends | | `src/xtrace_sdk/x_vec/crypto/bfv_client.py` | `HammingClientBase` interface, binary-vector layout, public-only server evaluation, packed responses, client decoding | | `src/xtrace_sdk/x_vec/utils/xtrace_types.py` | BFV parameter, polynomial, ciphertext and key types alongside the existing scheme types | | `tests/x_vec/test_bfv_encryption.py` | Independent polynomial oracle and cryptographic arithmetic tests | +| `tests/x_vec/test_bfv_evaluator.py` | Fused key-switch oracle, exact reference comparisons, validation and cache lifecycle tests | | `tests/x_vec/test_bfv_client.py` | Client, persistence, process separation, packing boundaries, default settings and optional SEAL comparison | | `benchmarks/bfv_client_matrix.py` | Native BFV and existing CPU Paillier/Lookup timing and wire-size comparison | +| `benchmarks/bfv_server.py` | Paired native server comparison on identical encrypted inputs, with first/warm searches and optional profiles | The only shared cryptographic interface correction is to the return annotations in `HomomorphicBase`: encryption returns a ciphertext, and decryption returns a @@ -39,7 +42,7 @@ query = [0, 1] * 256 vectors = [query[:], [1 - bit for bit in query], [1 - query[0], *query[1:]]] # The server gets configuration and public/evaluation keys only. -server = BFVClient(skip_key_gen=True) +server = BFVClient(skip_key_gen=True, server_backend="optimized") server.load_config(json.loads(client.stringify_config())) server.load_stringified_keys(client.stringify_pk()) assert server.keys is None @@ -69,6 +72,13 @@ distance per vector. `encrypt_vec_batch` deliberately keeps that contract; packing is exposed through separate methods. `device="auto"` currently resolves to CPU. `device="gpu"` raises a clear error. +`server_backend="optimized"` is the default; use `"reference"` for the original +server arithmetic. This is a runtime setting, so it is not stored in +`stringify_config()` and is preserved when loading a crypto configuration. +Reuse a server instance across queries to reuse its prepared public keys. +`load_stringified_keys()` discards the evaluator cache when loading keys. Change +keys through this method rather than mutating an initialized public-key dictionary. + BFV is available directly as a local client; `ExecutionContext`, `DataLoader`, and the current XTrace HTTP endpoints do not implement its index/wire protocol. Adding a production server path is separate work. Distances are computed at the @@ -129,10 +139,58 @@ arithmetic uses a single large modulus; only the plaintext batch transform uses an NTT. The small-ring schoolbook oracle tests products through 512-bit coefficients, including worst-case all-maximum inputs that exercise radix carries. -Natural next optimizations are cached transform plans, RNS/NTT polynomial -products, faster gadget decomposition/key switching, then CUDA kernels behind -the same primitive operations. Any replacement must preserve the unreduced -product needed by BFV's scale-and-round step. +The optimized evaluator accelerates gadget decomposition/key switching as +described below. RNS/NTT polynomial products and then CUDA kernels are possible +next steps. Any replacement must preserve the unreduced product needed by +BFV's scale-and-round step. + +### Cached and fused server evaluation + +The original server profile is dominated by key switching for rotations and +relinearization. With the default six gadget digits, each switch originally +made twelve polynomial products. Each product scanned coefficient widths, +packed both operands, unpacked the convolution, folded its upper half and +reduced coefficients. Intermediate results were added and reduced again. + +`BFVEvaluator` keeps the same arithmetic but performs less repeated work: + +1. Lazily pack each public evaluation key once, then reuse it across tiles and + queries. Cache entries belong to one public-key snapshot and are limited to + the number of evaluation keys in that snapshot. +2. Pack each ciphertext digit once for both output components. Accumulate all + six products for each component inside GMP, before unpacking or reducing. +3. Fold the accumulated product inside GMP using a guarded radix subtraction. + Unpack just two N-coefficient polynomials per switch, instead of twelve + ordinary convolutions with up to 2N coefficients each. +4. Check native integer coefficient types without repeated `Integral` ABC + lookups or conversions to Python integers, and check ranges using `min` and + `max`. Other integer types still use the reference validator. Wire parsing + and validation remain in place. + +This follows the same digit/key dot product as the textbook's +[BFV key switching](https://fhetextbook.github.io/HomomorphicKeySwitching1.html). +It changes how that dot product is evaluated, not the keys or error terms. +The packing uses the existing +[GMP `pack` and `unpack` operations](https://gmpy2.readthedocs.io/en/latest/mpz.html). +Ciphertext multiplication still uses the reference exact tensor product and +scale-and-round, followed by the optimized relinearization. + +For the packing bound, let L be the number of gadget digits and b their width. +An ordinary convolution coefficient in the accumulated dot product is less +than `L*N*2^b*q`. The radix width is +`w = bit_length(q) + b + bit_length(L*N) + 1`. Thus each coefficient is below +`H = 2^(w-1)`. Split the packed product at N radix digits, subtract its upper +half from its lower half, and add H in every digit. Each resulting digit lies +strictly between zero and `2^w`, so there is no borrow or carry between lanes. +Unpack N digits, subtract H, and reduce modulo q. Linearity makes this exactly +the same as folding and reducing every individual product before summation. +Tests cover zero, sparse, random and maximum coefficients, including a one-bit +gadget radix, a single gadget digit and 512-bit coefficients. + +The tradeoff is extra server RAM for prepared public keys. These are local +caches; no cache data is serialized or sent to the client. Both backends produce +identical ciphertext coefficients for identical inputs, including after response +compaction, so their noise budgets and query/response sizes also match. ## Packed Hamming layout @@ -188,9 +246,18 @@ occur if an unsupported circuit exhausts its noise. ## Tests and benchmarks ```bash -.venv/bin/python -m pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_client.py -q +.venv/bin/python -m pytest tests/x_vec/test_bfv_encryption.py \ + tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_client.py -q .venv/bin/python benchmarks/bfv_client_matrix.py --num-vectors 1024 \ - --json-out benchmarks/results/native_bfv_1024.json + --json-out /tmp/native_bfv_matrix.json + +# Compare original and optimized native servers using the same ciphertexts. +.venv/bin/python benchmarks/bfv_server.py --num-vectors 1024 --repeats 3 \ + --json-out /tmp/native_bfv_server_1024.json + +# Smaller comparison, plus separate profiles (not included in timing samples). +.venv/bin/python benchmarks/bfv_server.py --num-vectors 32 --repeats 4 \ + --profile-dir /tmp/bfv-profiles --json-out /tmp/native_bfv_server_32.json # Compare both BFV layouts and both existing CPU Paillier clients on a small batch. .venv/bin/python benchmarks/bfv_client_matrix.py --num-vectors 16 \ @@ -207,6 +274,13 @@ vectors. No account, network connection or GPU is needed. The benchmark reports key generation/export/import, index and query encryption, server evaluation, client decryption/decoding, and serialization separately. `--repeats` generates independent runs; raw results and median timings are saved. +Use `--server-backend reference` in the matrix benchmark to run the original +native evaluator. Its server time includes first-use cache preparation for the +optimized backend. The focused `bfv_server.py` benchmark instead shares one key +set and encrypted input between both backends, alternates their execution order, +and reports first-search and subsequent warm-search times separately. Every +search must return the same ciphertexts and all plaintext-oracle distances. +Its profiles are additional runs excluded from reported timings. The four size categories are: | Measurement | What is included | When it is paid | @@ -238,7 +312,8 @@ inputs. Each decrypted all 1,024 distances correctly and selected positions run, not a statistically established performance claim. The JSON includes source-file SHA-256 hashes because the implementation was uncommitted during measurement; no other test or benchmark was running concurrently. All SDK source -hashes match this implementation. The benchmark harness received formatting and +hashes identify the initial reference implementation; the server optimization +is measured separately below. The benchmark harness received formatting and lint fixes outside the measured operations afterward. | Bytes | Native BFV, packed | Existing Paillier CPU, key_len=1024 | @@ -267,7 +342,60 @@ Server compute and larger index/key storage are the present costs of moving distance decoding off the client. The result is a working optimization baseline, not an end-to-end latency improvement. -Validation for this change: 121 offline tests passed across `experiments/bfv`, +Validation for the initial implementation: 121 offline tests passed across `experiments/bfv`, `tests/cli`, and `tests/x_vec`, with the live Hamming/metadata service tests and CUDA tests excluded. The new native modules account for 39 of those tests, including the optional SEAL oracle in this environment. + +### Server optimization measurements + +The paired benchmark keeps the same native BFV parameters, public keys, query +and encrypted index for both backends. Its first search includes lazy cache +preparation; later searches reuse the prepared keys and plaintext masks. Each +search checks both exact ciphertext equality and every decoded Hamming distance. +The server uses only public keys. These are local CPU measurements with no +concurrent tests or other benchmarks, not network or production latency results. + +Saved data and a profile summary: + +- [32-vector comparison](../../benchmarks/results/native_bfv_server_32.json): + one first search and three warm searches per backend. +- [1,024-vector comparison](../../benchmarks/results/native_bfv_server_1024.json): + one first search and two warm searches per backend. +- [32-vector profiles](../../benchmarks/results/native_bfv_server_profile_32.txt): + separate extra searches after warming both servers. Instrumented timings are + excluded from the speedup calculations. + +| 512-bit vectors | Search | Reference seconds | Optimized seconds | Speedup | +| ---: | --- | ---: | ---: | ---: | +| 32 | First, including preparation | 3.391 | 1.878 | 1.81x | +| 32 | Warm median, 3 searches | 3.648 | 1.866 | 1.96x | +| 1,024 | First, including preparation | 121.564 | 60.355 | 2.01x | +| 1,024 | Warm median, 2 searches | 118.244 | 58.627 | 2.02x | + +The earlier 108.296-second result is retained as historical data. Ratios here +use reference and optimized runs from the same benchmark session, so variation +between sessions is not counted as an optimization benefit. The saved JSON +includes the environment, all timing samples, execution order, source hashes, +cache payload size, wire sizes and correctness results. + +In the 32-vector profiles, Python function calls fall from 19,838,069 to +3,581,974, and GMP unpack calls from 268 to 58. Most remaining evaluation time +is in the fused gadget dot products, primarily the large GMP integer products. +The cache trades extra server memory for less repeated work; its recorded byte +count covers prepared GMP integers and folding constants, not total process +memory or peak temporary allocations. Ciphertext outputs, noise and transmitted +sizes match exactly between backends for the same input. + +The 1,024-vector run prepared 16 switch keys with 52,083,728 bytes of cached +GMP values (about 52.1 MB, additional to the original public key). Both backends +returned one 102,496-byte response; query plus response was 471,250 bytes. All +1,024 distances matched, top-three positions were `[0, 1023, 81]`, and the +compacted result retained a 26-bit diagnostic noise budget. These native settings +remain distinct from SEAL's; this benchmark compares native backends only. + +Validation after optimization: 130 offline tests passed, including 48 native BFV +tests and the optional SEAL oracle. Ruff and mypy passed for the SDK and BFV +benchmark files. The live service and CUDA tests remain excluded from this CPU +change. An offline Sphinx build retains the preexisting README cross-reference +warning in the earlier SEAL research report. diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_client.py index 186acab..9c72dd8 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_client.py +++ b/src/xtrace_sdk/x_vec/crypto/bfv_client.py @@ -12,6 +12,7 @@ from xtrace_sdk.x_vec.crypto.device import DeviceMode from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVEvaluator, BFVServerBackend from xtrace_sdk.x_vec.crypto.hamming_client_base import HammingClientBase from xtrace_sdk.x_vec.utils.xtrace_types import ( BFVCiphertext, @@ -37,6 +38,8 @@ class BFVClient(HammingClientBase): :param response_modulus_bits: Smaller modulus used after the Hamming circuit. :param skip_key_gen: Construct an empty client for loading saved keys. :param device: ``cpu`` or ``auto``; a CUDA implementation is not yet available. + :param server_backend: ``optimized`` (cached GMP) or ``reference`` server arithmetic. + This runtime choice is independent of the serialized crypto configuration. """ def __init__( @@ -50,12 +53,18 @@ def __init__( response_modulus_bits: int = 50, skip_key_gen: bool = False, device: DeviceMode = "auto", + server_backend: BFVServerBackend = "optimized", ) -> None: if device not in ("cpu", "auto", "gpu"): raise ValueError("device must be 'auto', 'cpu', or 'gpu'") if device == "gpu": raise NotImplementedError("Native BFV currently supports only the CPU backend") self.device = "cpu" + if server_backend not in ("optimized", "reference"): + raise ValueError("server_backend must be 'optimized' or 'reference'") + self.server_backend = server_backend + self._server_evaluator: BFVEvaluator | None = None + self._evaluator_public_key: BFVPublicKey | None = None self.params = BFVParameters( poly_modulus_degree, plain_modulus, coeff_modulus_bits, decomposition_bits, error_eta ) @@ -109,6 +118,17 @@ def _keys(self) -> BFVKeyPair: raise RuntimeError("Secret key not initialized; public-only clients cannot decrypt") return self.keys + def _evaluator(self) -> BFVEvaluator: + pk = self._pk() + if ( + self._server_evaluator is None + or self._evaluator_public_key is not pk + or self._server_evaluator.backend != self.server_backend + ): + self._server_evaluator = BFVEvaluator(pk, self.server_backend) + self._evaluator_public_key = pk + return self._server_evaluator + @staticmethod def has_gpu() -> bool: return False @@ -162,6 +182,8 @@ def load_stringified_keys(self, pk: str, sk: str | None = None) -> None: keys = {"pk": public_key, "sk": BFV.deserialize_secret_key(sk, public_key)} self.public_key = public_key self.keys = keys + self._server_evaluator = None + self._evaluator_public_key = None def _validate_vector(self, embd: Sequence[int]) -> None: if len(embd) != self.embed_len: @@ -215,11 +237,11 @@ def encrypt_vec_packed(self, embds: list[list[int]]) -> list[EncryptedVector]: def _distance_tile( self, query: BFVCiphertext, tile: BFVCiphertext, count: int ) -> BFVCiphertext: - pk = self._pk() - distance = BFV.xor(query, tile, pk) + evaluator = self._evaluator() + distance = evaluator.xor(query, tile) for i in range(self.padded_embed_len.bit_length() - 1): step = self.lanes_per_row * (1 << i) - distance = BFV.add(distance, BFV.rotate_rows(distance, step, pk), pk) + distance = evaluator.add(distance, evaluator.rotate_rows(distance, step)) # The rotate/add tree repeats sums in each dimension block. Retain only # the first block in each row and clear unused lanes in a partial tile. if count not in self._mask_cache: @@ -230,7 +252,7 @@ def _distance_tile( row, col = divmod(lane, self.lanes_per_row) mask[row * (len(mask) // 2) + col] = 1 self._mask_cache[count] = BFV.batch_encode(mask, self.params) - return BFV.multiply_plain(distance, self._mask_cache[count], pk) + return evaluator.multiply_plain(distance, self._mask_cache[count]) def encode_hamming_server( self, @@ -266,6 +288,7 @@ def encode_hamming_server_packed( capacity = self.vectors_per_ciphertext self._validate_count(vector_count, len(index), capacity) query_ct = BFV.ciphertext_from_ints(query, pk) + evaluator = self._evaluator() result = [] tiles_per_response = self.padded_embed_len for start in range(0, len(index), tiles_per_response): @@ -276,7 +299,9 @@ def encode_hamming_server_packed( size = 1 while stack and stack[-1][1] == size: left, _ = stack.pop() - tile = BFV.add(left, BFV.rotate_rows(tile, -self.lanes_per_row * size, pk), pk) + tile = evaluator.add( + left, evaluator.rotate_rows(tile, -self.lanes_per_row * size) + ) size *= 2 stack.append((tile, size)) # Partial groups have a decreasing list of powers of two. Combining @@ -284,8 +309,8 @@ def encode_hamming_server_packed( combined, _ = stack.pop() while stack: left, size = stack.pop() - combined = BFV.add( - left, BFV.rotate_rows(combined, -self.lanes_per_row * size, pk), pk + combined = evaluator.add( + left, evaluator.rotate_rows(combined, -self.lanes_per_row * size) ) if compact: combined = BFV.modulus_switch(combined, self.response_modulus_bits, pk) diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py new file mode 100644 index 0000000..6e0841e --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py @@ -0,0 +1,214 @@ +"""Cached GMP evaluation of native BFV ciphertexts using public keys only. + +The BFV scheme in ``bfv.py`` remains the reference implementation. This module +accelerates its key-switch dot products without changing parameters, rounding, +noise, or wire formats. It shares the scheme's experimental, variable-time status. +""" + +from dataclasses import dataclass +from typing import Literal + +import gmpy2 +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _automorphism +from xtrace_sdk.x_vec.utils.xtrace_types import ( + BFVCiphertext, + BFVPolynomial, + BFVPublicKey, + BFVSwitchKey, +) + +BFVServerBackend = Literal["optimized", "reference"] + + +@dataclass(frozen=True) +class _PackedSwitchKey: + """Public gadget key prepared for an exact, fused Kronecker dot product.""" + + pairs: tuple[tuple[mpz, mpz], ...] + width: int + span: int + low_mask: mpz + bias_coefficient: mpz + bias: mpz + + @staticmethod + def compile(key: BFVSwitchKey, n: int, q: mpz, bits: int) -> "_PackedSwitchKey": + # Each coefficient in the sum of ordinary convolutions is strictly + # below len(key)*N*2^bits*q. One additional guard bit allows a signed + # negacyclic fold inside GMP, without borrows between coefficient lanes. + width = q.bit_length() + bits + (n * len(key)).bit_length() + 1 + span = n * width + bias_coefficient = mpz(1) << (width - 1) + return _PackedSwitchKey( + tuple((gmpy2.pack(list(b), width), gmpy2.pack(list(a), width)) for b, a in key), + width, + span, + (mpz(1) << span) - 1, + bias_coefficient, + gmpy2.pack([bias_coefficient] * n, width), + ) + + def apply(self, poly: BFVPolynomial, q: mpz, bits: int) -> tuple[BFVPolynomial, BFVPolynomial]: + mask = (mpz(1) << bits) - 1 + accum0, accum1 = mpz(0), mpz(0) + for i, (b, a) in enumerate(self.pairs): + digits = gmpy2.pack([(c >> (i * bits)) & mask for c in poly], self.width) + accum0 += digits * b + accum1 += digits * a + + def finish(value: mpz) -> BFVPolynomial: + # Fold X^N = -1 before unpacking. Every low/high coefficient is + # < bias_coefficient, so low - high + bias lies strictly in (0,2^w). + # Thus the packed subtraction has no cross-lane borrow, and unpack + # returns exactly N coefficients, even for the zero polynomial. + folded = (value & self.low_mask) - (value >> self.span) + self.bias + return tuple((c - self.bias_coefficient) % q for c in gmpy2.unpack(folded, self.width)) + + return finish(accum0), finish(accum1) + + +class BFVEvaluator: + """Reusable native BFV server arithmetic, bound to one public key. + + ``optimized`` lazily caches packed evaluation keys and fuses gadget products; + ``reference`` delegates to the original BFV routines. Both return identical + ciphertexts. Reuse one evaluator across queries to amortize preparation. + + :param pk: Valid public key from key generation or public-key deserialization. + :param backend: ``optimized`` (default) or ``reference``. + + The key dictionaries are snapshotted; immutable polynomial tuples are shared. + Construct a new evaluator when changing keys. The cache is bounded by the + number of evaluation keys in this snapshot and never stores query results. + """ + + def __init__(self, pk: BFVPublicKey, backend: BFVServerBackend = "optimized") -> None: + if backend not in ("optimized", "reference"): + raise ValueError("server_backend must be 'optimized' or 'reference'") + self.backend = backend + self._pk: BFVPublicKey = {**pk, "galois_keys": dict(pk["galois_keys"])} + # Exponent zero identifies relinearization; Galois exponents are odd. + self._switch_keys: dict[int, _PackedSwitchKey] = {} + + def _validate(self, ciphertext: BFVCiphertext) -> None: + pk = self._pk + n, q = pk["params"].poly_modulus_degree, ciphertext.modulus + if ciphertext.key_id != pk["key_id"]: + raise ValueError("Ciphertext belongs to a different BFV key") + if not pk["params"].plain_modulus < q <= pk["q"]: + raise ValueError("Invalid ciphertext modulus") + if len(ciphertext.components) not in (2, 3): + raise ValueError("BFV ciphertext must have two or three components") + for poly in ciphertext.components: + if len(poly) != n: + raise ValueError("Invalid ciphertext polynomial") + # Avoid Integral's Python ABC checks and int(mpz) conversions for + # every native coefficient. Retain support for other Integral types + # through the reference validator; never silently accept floats. + if any(type(c) is not mpz and type(c) is not int for c in poly): + BFV.validate_ciphertext(ciphertext, pk) + return + if min(poly) < 0 or max(poly) >= q: + raise ValueError("Invalid ciphertext polynomial") + + def _pair(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> None: + self._validate(lhs) + if rhs is not lhs: + self._validate(rhs) + if lhs.modulus != rhs.modulus or len(lhs.components) != len(rhs.components): + raise ValueError("Ciphertexts must have the same modulus and component count") + + def _switch(self, poly: BFVPolynomial, exponent: int) -> tuple[BFVPolynomial, BFVPolynomial]: + pk, params = self._pk, self._pk["params"] + if exponent not in self._switch_keys: + key = pk["relin_key"] if exponent == 0 else pk["galois_keys"][exponent] + self._switch_keys[exponent] = _PackedSwitchKey.compile( + key, params.poly_modulus_degree, pk["q"], params.decomposition_bits + ) + return self._switch_keys[exponent].apply(poly, pk["q"], params.decomposition_bits) + + def add(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> BFVCiphertext: + """Add two ciphertexts with the same modulus and component count.""" + if self.backend == "reference": + return BFV.add(lhs, rhs, self._pk) + self._pair(lhs, rhs) + return BFVCiphertext( + tuple( + tuple((a + b) % lhs.modulus for a, b in zip(p1, p2, strict=True)) + for p1, p2 in zip(lhs.components, rhs.components, strict=True) + ), + lhs.modulus, + lhs.key_id, + ) + + def subtract(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> BFVCiphertext: + """Subtract ciphertexts before squaring for binary XOR.""" + if self.backend == "reference": + return BFV.subtract(lhs, rhs, self._pk) + self._pair(lhs, rhs) + return BFVCiphertext( + tuple( + tuple((a - b) % lhs.modulus for a, b in zip(p1, p2, strict=True)) + for p1, p2 in zip(lhs.components, rhs.components, strict=True) + ), + lhs.modulus, + lhs.key_id, + ) + + def multiply_plain(self, ciphertext: BFVCiphertext, plaintext: BFVPolynomial) -> BFVCiphertext: + """Apply an unscaled plaintext polynomial, e.g. a distance slot mask.""" + return BFV.multiply_plain(ciphertext, plaintext, self._pk) + + def relinearize(self, ciphertext: BFVCiphertext) -> BFVCiphertext: + """Replace the quadratic component using the cached public gadget key.""" + if self.backend == "reference": + return BFV.relinearize(ciphertext, self._pk) + self._validate(ciphertext) + if len(ciphertext.components) == 2: + return ciphertext + pk = self._pk + if ciphertext.modulus != pk["q"]: + raise ValueError("Relinearization requires the original ciphertext modulus") + if not pk["relin_key"]: + raise ValueError("Public key has no relinearization key") + c0, c1, c2 = ciphertext.components + k0, k1 = self._switch(c2, 0) + return BFVCiphertext( + ( + tuple((a + b) % pk["q"] for a, b in zip(c0, k0, strict=True)), + tuple((a + b) % pk["q"] for a, b in zip(c1, k1, strict=True)), + ), + pk["q"], + pk["key_id"], + ) + + def multiply(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> BFVCiphertext: + """Keep the reference exact tensor product and scale; accelerate relinearization.""" + return self.relinearize(BFV.multiply(lhs, rhs, self._pk, relinearize=False)) + + def xor(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> BFVCiphertext: + """Slotwise binary XOR, (a-b)^2; encrypted inputs cannot be checked for bits.""" + difference = self.subtract(lhs, rhs) + return self.multiply(difference, difference) + + def rotate_rows(self, ciphertext: BFVCiphertext, steps: int) -> BFVCiphertext: + """Rotate both slot rows left (right for negative steps) using public keys.""" + if self.backend == "reference": + return BFV.rotate_rows(ciphertext, steps, self._pk) + self._validate(ciphertext) + pk = self._pk + if ciphertext.modulus != pk["q"] or len(ciphertext.components) != 2: + raise ValueError("Rotation requires two components at the original modulus") + n, q = pk["params"].poly_modulus_degree, pk["q"] + exponent = pow(3, steps % (n // 2), 2 * n) + if exponent == 1: + return ciphertext + if exponent not in pk["galois_keys"]: + raise ValueError(f"Public key has no rotation key for steps={steps}") + c0, c1 = (_automorphism(c, exponent, q) for c in ciphertext.components) + k0, k1 = self._switch(c1, exponent) + return BFVCiphertext( + (tuple((a + b) % q for a, b in zip(c0, k0, strict=True)), k1), q, pk["key_id"] + ) diff --git a/tests/x_vec/test_bfv_client.py b/tests/x_vec/test_bfv_client.py index 24b2e77..ca68f5c 100644 --- a/tests/x_vec/test_bfv_client.py +++ b/tests/x_vec/test_bfv_client.py @@ -18,8 +18,8 @@ def small_client(dimension: int = 3) -> BFVClient: return BFVClient(dimension, 32, 193, 120, 20, response_modulus_bits=40) -def public_server(client: BFVClient) -> BFVClient: - server = BFVClient(skip_key_gen=True) +def public_server(client: BFVClient, backend: str = "optimized") -> BFVClient: + server = BFVClient(skip_key_gen=True, server_backend=backend) server.load_config(json.loads(client.stringify_config())) server.load_stringified_keys(client.stringify_pk()) assert server.keys is None @@ -70,11 +70,14 @@ def test_packed_tile_row_and_response_boundaries(dimension: int) -> None: client = small_client(dimension) server = public_server(client) c = client.vectors_per_ciphertext + reference = public_server(client, "reference") for count in sorted({0, 1, c - 1, c, c + 1, 31, 32, 33, 67}): vectors, query, expected = data(count, dimension) index = client.encrypt_vec_packed(vectors) assert len(index) == (count + c - 1) // c - result = server.encode_hamming_server_packed(client.encrypt_vec_one(query), index, count) + encrypted_query = client.encrypt_vec_one(query) + result = server.encode_hamming_server_packed(encrypted_query, index, count) + assert result == reference.encode_hamming_server_packed(encrypted_query, index, count) assert len(result) == (count + 31) // 32 distances = client.decode_hamming_client_packed(result, count) assert distances == expected @@ -102,6 +105,8 @@ def test_partial_tile_padding_is_zero_and_compaction_preserves_distances() -> No assert all(slots[i] == 0 for i in range(32) if i not in used) compact = BFV.ciphertext_to_ints(BFV.modulus_switch(ct, 40, pk), pk) assert client.decode_hamming_client_packed([compact], 13) == expected + reference = public_server(client, "reference") + assert reference.encode_hamming_server_packed(query_ct, index, 13, compact=False) == response assert sum(v.bit_length() for v in compact) < sum(v.bit_length() for v in response[0]) / 2 @@ -168,17 +173,45 @@ def test_invalid_vectors_keys_configuration_and_counts() -> None: with pytest.raises(NotImplementedError, match="CPU"): BFVClient(device="gpu") assert not client.has_gpu() + with pytest.raises(ValueError, match="server_backend"): + BFVClient(skip_key_gen=True, server_backend="invalid") for dimension in (0, -1, 17): with pytest.raises(ValueError, match="embed_len"): small_client(dimension) +def test_backend_and_key_reload_reset_public_evaluator() -> None: + client = small_client() + server = public_server(client, "reference") + assert server.server_backend == "reference" + assert "server_backend" not in json.loads(server.stringify_config()) + query = client.encrypt_vec_one([0, 1, 0]) + reference = server.encode_hamming_server(query, query) + server.server_backend = "optimized" + assert server.encode_hamming_server(query, query) == reference + old_evaluator = server._evaluator() + assert old_evaluator._switch_keys + replacement = small_client() + server.load_stringified_keys(replacement.stringify_pk()) + assert server._server_evaluator is None + assert server._evaluator() is not old_evaluator + assert not server._evaluator()._switch_keys + with pytest.raises(ValueError, match="header or key"): + server.encode_hamming_server(query, query) + query = replacement.encrypt_vec_one([0, 1, 0]) + assert replacement.decode_hamming_client_one(server.encode_hamming_server(query, query)) == 0 + + @pytest.fixture(scope="module") def default_case() -> tuple[BFVClient, list[list[int]], list[int], list[int]]: client = BFVClient() vectors, query, expected = data(33, 512) index = client.encrypt_vec_packed(vectors) - result = client.encode_hamming_server_packed(client.encrypt_vec_one(query), index, len(vectors)) + encrypted_query = client.encrypt_vec_one(query) + result = client.encode_hamming_server_packed(encrypted_query, index, len(vectors)) + reference = BFVClient(skip_key_gen=True, server_backend="reference") + reference.public_key = client._pk() + assert reference.encode_hamming_server_packed(encrypted_query, index, len(vectors)) == result distances = client.decode_hamming_client_packed(result, len(vectors)) assert distances == expected assert BFV.noise_budget(BFV.ciphertext_from_ints(result[0], client._pk()), client._keys()) >= 10 diff --git a/tests/x_vec/test_bfv_evaluator.py b/tests/x_vec/test_bfv_evaluator.py new file mode 100644 index 0000000..30797d9 --- /dev/null +++ b/tests/x_vec/test_bfv_evaluator.py @@ -0,0 +1,145 @@ +"""Exact coefficient comparisons and boundary oracles for optimized BFV evaluation.""" + +import random +from dataclasses import replace + +import numpy as np +import pytest +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVEvaluator, _PackedSwitchKey +from xtrace_sdk.x_vec.utils.xtrace_types import BFVCiphertext, BFVKeyPair + + +@pytest.fixture(scope="module") +def keys() -> BFVKeyPair: + # Insecure toy ring; only the arithmetic is under test. + return BFV.key_gen(16, 97, 100, 15, rotation_steps=[1, -1, 2, 4]) + + +@pytest.mark.parametrize( + "q_bits,digit_bits", [(31, 1), (100, 15), (180, 30), (512, 127), (100, 100)] +) +def test_fused_switch_against_schoolbook(q_bits: int, digit_bits: int) -> None: + n, q = 8, (mpz(1) << q_bits) - 1 + digits = (q_bits + digit_bits - 1) // digit_bits + rng = random.Random(q_bits + digit_bits) + # All-maximum keys stress the bound on the *sum* of digit convolutions. + for key in ( + tuple(((q - 1,) * n, (q - 1,) * n) for _ in range(digits)), + tuple( + tuple(tuple(mpz(rng.randrange(q)) for _ in range(n)) for _ in range(2)) + for _ in range(digits) + ), + ): + compiled = _PackedSwitchKey.compile(key, n, q, digit_bits) + for poly in ( + (mpz(0),) * n, + (q - 1,) * n, + (mpz(0),) * (n - 1) + (q - 1,), + tuple(mpz(rng.randrange(q)) for _ in range(n)), + ): + expected = [[0] * n for _ in range(2)] + # Independent O(N^2) signed negacyclic convolution and digit sum. + # No pack/unpack or optimized/reference polynomial helper is used. + for i, pair in enumerate(key): + for j, coefficient in enumerate(poly): + digit = (int(coefficient) >> (i * digit_bits)) % (1 << digit_bits) + for component in range(2): + for k, value in enumerate(pair[component]): + sign = 1 if j + k < n else -1 + expected[component][(j + k) % n] += sign * digit * int(value) + actual = compiled.apply(poly, q, digit_bits) + assert actual == tuple(tuple(value % q for value in row) for row in expected) + + +def test_operations_identical_to_reference(keys: BFVKeyPair) -> None: + pk, rng = keys["pk"], random.Random(20) + evaluator = BFVEvaluator(pk) + for _ in range(3): + pa, pb = ( + BFV.batch_encode([rng.randrange(97) for _ in range(16)], pk["params"]) for _ in range(2) + ) + ca, cb = BFV.encrypt(pa, pk), BFV.encrypt(pb, pk) + assert evaluator.add(ca, cb) == BFV.add(ca, cb, pk) + assert evaluator.subtract(ca, cb) == BFV.subtract(ca, cb, pk) + assert evaluator.multiply_plain(ca, pb) == BFV.multiply_plain(ca, pb, pk) + assert evaluator.multiply(ca, cb) == BFV.multiply(ca, cb, pk) + assert evaluator.xor(ca, cb) == BFV.xor(ca, cb, pk) + quadratic = BFV.multiply(ca, cb, pk, relinearize=False) + assert evaluator.relinearize(quadratic) == BFV.relinearize(quadratic, pk) + assert evaluator.relinearize(ca) is ca + assert evaluator.add(quadratic, quadratic) == BFV.add(quadratic, quadratic, pk) + for step in (0, 1, -1, 2, 4, 9): + assert evaluator.rotate_rows(ca, step) == BFV.rotate_rows(ca, step, pk) + compact = BFV.modulus_switch(ca, 40, pk) + assert evaluator.add(compact, compact) == BFV.add(compact, compact, pk) + assert evaluator.subtract(compact, compact) == BFV.subtract(compact, compact, pk) + + +def test_reject_malformed_ciphertexts_and_keep_integral_support(keys: BFVKeyPair) -> None: + pk = keys["pk"] + evaluator = BFVEvaluator(pk) + ct = BFV.encrypt(BFV.batch_encode([1], pk["params"]), pk) + malformed = [ + replace(ct, key_id="wrong"), + replace(ct, modulus=mpz(97)), + replace(ct, modulus=pk["q"] + 1), + replace(ct, components=ct.components[:1]), + replace(ct, components=(ct.components[0][:-1], ct.components[1])), + ] + for value in (-1, pk["q"], 0.0, "0", None): + malformed.append(replace(ct, components=((value, *ct.components[0][1:]), ct.components[1]))) + for bad in malformed: + for validate in (evaluator._validate, lambda value: BFV.validate_ciphertext(value, pk)): + with pytest.raises(ValueError): + validate(bad) + for operation in (evaluator.add, evaluator.subtract, evaluator.multiply): + with pytest.raises(ValueError): + operation(ct, bad) + with pytest.raises(ValueError): + evaluator.rotate_rows(bad, 1) + with pytest.raises(ValueError): + evaluator.relinearize(bad) + for value in (0, False, np.int64(0)): + candidate = replace(ct, components=((value, *ct.components[0][1:]), ct.components[1])) + assert evaluator.add(ct, candidate) == BFV.add(ct, candidate, pk) + quadratic = BFV.multiply(ct, ct, pk, relinearize=False) + compact = BFV.modulus_switch(ct, 40, pk) + for incompatible in (quadratic, compact): + with pytest.raises(ValueError): + evaluator.add(ct, incompatible) + with pytest.raises(ValueError): + evaluator.subtract(ct, incompatible) + with pytest.raises(ValueError): + evaluator.multiply(incompatible, incompatible) + with pytest.raises(ValueError): + evaluator.rotate_rows(incompatible, 1) + with pytest.raises(ValueError): + evaluator.relinearize(BFV.modulus_switch(quadratic, 40, pk)) + + +def test_missing_keys_and_snapshot_cache(keys: BFVKeyPair) -> None: + pk = {**keys["pk"], "galois_keys": dict(keys["pk"]["galois_keys"])} + ct = BFV.encrypt(BFV.batch_encode([1], pk["params"]), pk) + evaluator = BFVEvaluator(pk) + assert not evaluator._switch_keys + first = evaluator.rotate_rows(ct, 1) + prepared = evaluator._switch_keys[3] + assert evaluator.rotate_rows(ct, 9) == first + assert evaluator._switch_keys[3] is prepared + assert len(evaluator._switch_keys) == 1 + # Mutation of the caller's dictionary cannot leave a stale mixed cache. + pk["galois_keys"].clear() + assert evaluator.rotate_rows(ct, 1) == first + missing = BFVEvaluator({**pk, "relin_key": ()}) + with pytest.raises(ValueError, match="rotation key"): + missing.rotate_rows(ct, 1) + with pytest.raises(ValueError, match="relinearization key"): + missing.multiply(ct, ct) + reference = BFVEvaluator(keys["pk"], "reference") + assert reference.rotate_rows(ct, 1) == first + assert not reference._switch_keys + with pytest.raises(ValueError, match="server_backend"): + BFVEvaluator(keys["pk"], "invalid") From 79ad443f2cc4606af8f142af075ecdcf8e62d8a2 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Thu, 10 Sep 2026 19:17:57 -0400 Subject: [PATCH 04/12] Add native RNS/NTT BFV CPU server backend --- .github/workflows/ci.yml | 13 + CHANGELOG.md | 1 + benchmarks/bfv_client_matrix.py | 4 +- benchmarks/bfv_server.py | 141 +++++-- benchmarks/results/native_bfv_rns_1024.json | 222 ++++++++++ benchmarks/results/native_bfv_rns_32.json | 242 +++++++++++ .../results/native_bfv_rns_profile_32.txt | 83 ++++ docs/api_rst/xtrace_sdk.crypto.encryption.rst | 8 + docs/index.rst | 1 + docs/research/native-bfv-ntt.md | 228 ++++++++++ docs/research/native-bfv.md | 22 +- hatch_build.py | 7 +- pyproject.toml | 5 +- src/xtrace_sdk/x_vec/crypto/bfv_client.py | 16 +- .../x_vec/crypto/bfv_cpu_ext/Makefile | 23 + .../x_vec/crypto/bfv_cpu_ext/__init__.py | 1 + .../x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi | 24 ++ .../x_vec/crypto/bfv_cpu_ext/bindings.cpp | 279 ++++++++++++ .../x_vec/crypto/bfv_cpu_ext/rns_ntt.h | 397 ++++++++++++++++++ .../x_vec/crypto/encryption/bfv_evaluator.py | 93 +++- .../x_vec/crypto/encryption/bfv_rns.py | 111 +++++ tests/x_vec/native/test_bfv_rns.cpp | 85 ++++ tests/x_vec/test_bfv_client.py | 16 +- tests/x_vec/test_bfv_evaluator.py | 37 +- tests/x_vec/test_bfv_rns.py | 193 +++++++++ 25 files changed, 2187 insertions(+), 65 deletions(-) create mode 100644 benchmarks/results/native_bfv_rns_1024.json create mode 100644 benchmarks/results/native_bfv_rns_32.json create mode 100644 benchmarks/results/native_bfv_rns_profile_32.txt create mode 100644 docs/research/native-bfv-ntt.md create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/__init__.py create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h create mode 100644 src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py create mode 100644 tests/x_vec/native/test_bfv_rns.cpp create mode 100644 tests/x_vec/test_bfv_rns.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6569790..a54fb49 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -62,6 +62,19 @@ jobs: - name: Pytest (test_paillier_encryption) run: uv run pytest tests/x_vec/test_paillier_encryption.py + - name: Build and test native BFV CPU kernels + run: | + sudo apt-get update + sudo apt-get install -y libgmp-dev + make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" + uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 1' + uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_client.py + + - name: BFV native arithmetic sanitizers + run: | + g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext tests/x_vec/native/test_bfv_rns.cpp -lgmpxx -lgmp -o /tmp/test_bfv_rns + /tmp/test_bfv_rns + # ── Integration tests (require XTrace API credentials) ────────── # Triggers: # - PRs: only when explicitly labeled `run-integration` (avoids quota diff --git a/CHANGELOG.md b/CHANGELOG.md index 0fd15f2..5ec3051 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ All notable changes to this project will be documented in this file. ### Added +- Optional native BFV `server_backend="rns"`, with independently implemented C++ RNS/NTT polynomial arithmetic, cached transformed evaluation keys, exact GMP reconstruction, and a native Hamming tile circuit. Preserves native BFV keys, ciphertext outputs and wire format. Includes native-boundary tests, sanitizer oracles, optional SEAL benchmarks and correctly tagged binary wheels. The previous GMP and reference backends remain available. - Native GMP BFV primitives in `crypto/encryption/bfv.py` and a `BFVClient` in `crypto/bfv_client.py`, with CRT batching, ciphertext multiplication/relinearization, public-key Hamming evaluation, packed distance responses, and terminal modulus compaction. Includes CPU benchmarks and tests against plaintext algebra and an optional SEAL oracle. This is an experimental local implementation; the existing Paillier clients and SEAL research example are preserved. - Offline BFV packed Hamming experiment under `experiments/bfv`, with an isolated public-key server evaluator, correctness tests, a comparison against an exported Paillier Git revision, and a research report. The production Paillier path and dependencies are unchanged. - `xtrace_sdk.x_vec.crypto.device.resolve_device` — shared device-resolution helper used by both Paillier clients and intended for future homomorphic clients with optional GPU backends. diff --git a/benchmarks/bfv_client_matrix.py b/benchmarks/bfv_client_matrix.py index de6fe03..09d468f 100644 --- a/benchmarks/bfv_client_matrix.py +++ b/benchmarks/bfv_client_matrix.py @@ -236,7 +236,9 @@ def main() -> None: parser.add_argument("--decomposition-bits", type=int, default=30) parser.add_argument("--response-modulus-bits", type=int, default=50) parser.add_argument("--no-compact", action="store_true") - parser.add_argument("--server-backend", choices=("optimized", "reference"), default="optimized") + parser.add_argument( + "--server-backend", choices=("optimized", "reference", "rns"), default="optimized" + ) parser.add_argument("--seed", type=int, default=1337) parser.add_argument( "--repeats", diff --git a/benchmarks/bfv_server.py b/benchmarks/bfv_server.py index 151158c..693960b 100644 --- a/benchmarks/bfv_server.py +++ b/benchmarks/bfv_server.py @@ -11,6 +11,7 @@ import argparse import cProfile import hashlib +import importlib.metadata import json import platform import statistics @@ -19,7 +20,7 @@ from datetime import UTC, datetime from functools import partial from pathlib import Path -from typing import Any +from typing import Any, cast import gmpy2 @@ -46,6 +47,16 @@ def main() -> None: help="Searches per backend, including the first cold search", ) parser.add_argument("--json-out", type=Path) + parser.add_argument( + "--backends", + default="reference,optimized", + help="Comma-separated native backends: reference,optimized,rns", + ) + parser.add_argument( + "--seal", + action="store_true", + help="Also measure the preserved SEAL prototype on the same plaintext workload", + ) parser.add_argument( "--profile-dir", type=Path, @@ -54,6 +65,16 @@ def main() -> None: args = parser.parse_args() if args.num_vectors < 1 or args.repeats < 1 or args.embed_len < 1: parser.error("num-vectors, embed-len and repeats must be positive") + names = args.backends.split(",") + if ( + not names + or len(set(names)) != len(names) + or any(b not in ("reference", "optimized", "rns") for b in names) + ): + parser.error("backends must be distinct native backend names") + if args.seal and (args.poly_modulus_degree != 8192 or args.plain_modulus != 65537): + parser.error("the SEAL comparison requires N=8192 and t=65537") + backends = tuple(cast(BFVServerBackend, name) for name in names) vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) print("Generating one key set and encrypting the shared input", file=sys.stderr, flush=True) setup: dict[str, float] = {} @@ -74,7 +95,6 @@ def main() -> None: server_index, server_query = unpack_packet(index_wire), unpack_packet(query_wire)[0] setup["public_key_export_s"], public_json = time_call(client.stringify_pk) config = json.loads(client.stringify_config()) - backends: tuple[BFVServerBackend, ...] = ("reference", "optimized") servers = {} for backend in backends: server = BFVClient(skip_key_gen=True, server_backend=backend, **config) @@ -91,22 +111,76 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: server_query, server_index, len(vectors) ) - timings: dict[str, list[float]] = {backend: [] for backend in backends} + seal_info: dict[str, Any] | None = None + if args.seal: + sys.path.insert(0, str(REPO_ROOT / "experiments/bfv")) + from packed_hamming import BfvClient, BfvServer + + print( + "Preparing the preserved SEAL prototype on the same vectors", + file=sys.stderr, + flush=True, + ) + seal_client = BfvClient(args.embed_len) + seal_bundle = seal_client.server_bundle() + seal_server = BfvServer(seal_bundle) + seal_index, seal_query = ( + seal_client.encrypt_index(vectors), + seal_client.encrypt_query(query), + ) + seal_info = { + "tenseal": importlib.metadata.version("tenseal"), + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + p.bit_count() + for p in seal_client.context.first_context_data().parms().coeff_modulus() + ], + "key_coeff_modulus_bits": [ + p.bit_count() + for p in seal_client.context.key_context_data().parms().coeff_modulus() + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": len(seal_index), + "query_bytes": len(seal_query), + "public_key_bytes": len(seal_bundle), + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + } + + measured_backends = (*backends, "seal") if args.seal else backends + timings: dict[str, list[float]] = {backend: [] for backend in measured_backends} order = [] reference_response: list[list[int]] | None = None for repeat in range(args.repeats): # Alternate ordering to reduce systematic first/last-run effects. - for backend in backends if repeat % 2 == 0 else backends[::-1]: - print(f"{backend}: search {repeat + 1}/{args.repeats}", file=sys.stderr, flush=True) - elapsed, response = time_call(partial(evaluate, backend)) - timings[backend].append(elapsed) - order.append({"backend": backend, "repeat": repeat, "server_compute_s": elapsed}) - if reference_response is None: - reference_response = response - assert response == reference_response, "Backend ciphertexts differ" - assert client.decode_hamming_client_packed(response, len(vectors)) == expected + for backend_name in measured_backends if repeat % 2 == 0 else measured_backends[::-1]: + print( + f"{backend_name}: search {repeat + 1}/{args.repeats}", file=sys.stderr, flush=True + ) + if backend_name == "seal": + elapsed, seal_response = time_call( + partial(seal_server.search, seal_index, seal_query) + ) + assert [d for _, d in seal_client.decrypt_distances(seal_response)] == expected + assert seal_info is not None + seal_info["response_bytes"] = len(seal_response) + seal_info["all_distances_correct"] = True + seal_info["minimum_response_noise_budget_bits"] = min( + seal_client.noise_budgets(seal_response) + ) + else: + elapsed, response = time_call( + partial(evaluate, cast(BFVServerBackend, backend_name)) + ) + if reference_response is None: + reference_response = response + assert response == reference_response, "Native backend ciphertexts differ" + assert client.decode_hamming_client_packed(response, len(vectors)) == expected + timings[backend_name].append(elapsed) + order.append({"backend": backend_name, "repeat": repeat, "server_compute_s": elapsed}) print( - f"{backend}: {elapsed:.6f} s; ciphertexts and all distances verified", + f"{backend_name}: {elapsed:.6f} s; all distances verified" + + ("; identical native ciphertexts" if backend_name != "seal" else ""), file=sys.stderr, flush=True, ) @@ -137,8 +211,16 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_client.py", REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py", REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py", + REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py", + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.cpp"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.h"), + REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile", REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", ] + if "rns" in backends: + source_paths.extend((REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.so")) + if args.seal: + source_paths.append(REPO_ROOT / "experiments/bfv/packed_hamming.py") output: dict[str, Any] = { "environment": { "utc": datetime.now(UTC).isoformat(), @@ -171,11 +253,24 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: "setup_timings": setup, "runs": order, "summary": summary, - "first_search_speedup": timings["reference"][0] / timings["optimized"][0], + "first_search_speedup": timings["reference"][0] / timings["optimized"][0] + if "reference" in timings and "optimized" in timings + else None, "warm_median_speedup": statistics.median(timings["reference"][1:]) / statistics.median(timings["optimized"][1:]) - if args.repeats > 1 + if args.repeats > 1 and "reference" in timings and "optimized" in timings else None, + "speedups_relative_to_first_native_backend": { + backend: { + "first": timings[backends[0]][0] / values[0], + "warm_median": statistics.median(timings[backends[0]][1:]) + / statistics.median(values[1:]) + if args.repeats > 1 + else None, + } + for backend, values in timings.items() + }, + "seal_comparison": seal_info, "sizes": { "encrypted_index_bytes": len(index_wire), "public_keys_bytes": key_bytes, @@ -183,18 +278,10 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: "response_bytes": len(response_wire), "query_plus_response_bytes": len(query_wire) + len(response_wire), }, - "optimized_cached_switch_keys": len(servers["optimized"]._evaluator()._switch_keys), - "optimized_packed_cache_payload_bytes": sum( - sys.getsizeof(value) - for key in servers["optimized"]._evaluator()._switch_keys.values() - for value in ( - key.low_mask, - key.bias_coefficient, - key.bias, - *(v for pair in key.pairs for v in pair), - ) - ), - "cache_size_note": "sys.getsizeof of packed GMP values, including folding constants; additional to the existing public key. Excludes small Python containers and transient arithmetic allocations; not peak RSS.", + "caches": { + backend: server._evaluator().cache_info() for backend, server in servers.items() + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", "minimum_response_noise_budget_bits": min( BFV.noise_budget(BFV.ciphertext_from_ints(ct, client._pk()), client._keys()) for ct in reference_response diff --git a/benchmarks/results/native_bfv_rns_1024.json b/benchmarks/results/native_bfv_rns_1024.json new file mode 100644 index 0000000..c3b34eb --- /dev/null +++ b/benchmarks/results/native_bfv_rns_1024.json @@ -0,0 +1,222 @@ +{ + "environment": { + "utc": "2026-09-10T23:15:20.306784+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "ff8d5a29a58874e6498e1cdae6b3fcde21e19a5d", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "1024", + "--backends", + "optimized,rns", + "--seal", + "--repeats", + "3", + "--json-out", + "benchmarks/results/native_bfv_rns_1024.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for both backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "response_modulus_bits": 50, + "device": "cpu" + }, + "seed": 1337, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 4.33735747798346, + "index_encrypt_s": 6.699632134055719, + "query_encrypt_s": 0.1024328120984137, + "public_key_export_s": 0.7893146618735045, + "optimized_public_key_import_s": 0.8305047010071576, + "rns_public_key_import_s": 0.8661916260607541 + }, + "runs": [ + { + "backend": "optimized", + "repeat": 0, + "server_compute_s": 55.7612584088929 + }, + { + "backend": "rns", + "repeat": 0, + "server_compute_s": 8.329127850010991 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 2.4031696289312094 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 2.387246180092916 + }, + { + "backend": "rns", + "repeat": 1, + "server_compute_s": 8.029489122098312 + }, + { + "backend": "optimized", + "repeat": 1, + "server_compute_s": 55.893618842121214 + }, + { + "backend": "optimized", + "repeat": 2, + "server_compute_s": 55.91135185002349 + }, + { + "backend": "rns", + "repeat": 2, + "server_compute_s": 8.04160486208275 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 2.3929480500519276 + } + ], + "summary": { + "optimized": { + "first_search_s": 55.7612584088929, + "warm_searches_s": [ + 55.893618842121214, + 55.91135185002349 + ], + "warm_median_s": 55.90248534607235 + }, + "rns": { + "first_search_s": 8.329127850010991, + "warm_searches_s": [ + 8.029489122098312, + 8.04160486208275 + ], + "warm_median_s": 8.03554699209053 + }, + "seal": { + "first_search_s": 2.4031696289312094, + "warm_searches_s": [ + 2.387246180092916, + 2.3929480500519276 + ], + "warm_median_s": 2.3900971150724217 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "optimized": { + "first": 1.0, + "warm_median": 1.0 + }, + "rns": { + "first": 6.694729557887542, + "warm_median": 6.956898565971642 + }, + "seal": { + "first": 23.20321367979849, + "warm_median": 23.389210837308788 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 27680661, + "query_bytes": 432556, + "public_key_bytes": 39587127, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 105915, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "sizes": { + "encrypted_index_bytes": 23598495, + "public_keys_bytes": 85603545, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "caches": { + "optimized": { + "switch_keys": 16, + "key_payload_bytes": 52083728, + "transform_table_bytes": 0 + }, + "rns": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "switch_keys": 16, + "key_payload_bytes": 50331648 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "839279a7bfd41e46e02a4f49f51afc6e3fbe2e3c33d5a4f9745eb3107bf399c1", + "benchmarks/bfv_client_matrix.py": "b5539dd816f6dbd8f26fadde4f72c9582a8fa4f67bf8a64b3a874d38f576216c", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "03205e7775265873da6da51a2d26aa0e63a59c2961dab31f29e71e8b9b92a84f", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "6f03ac75f42218170da2bd785605ee394940a202afdd5471545a8ae3fdc8c3d8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "de206a348b704982cfa58526189af7c61cf0628807457f7beddb00498a2ea78a", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "6f97a52f5b4b9981533fbed5d043e4e2179906491d0805609fd76e025e38bbc9", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "fb72e876eb585874c8dd42a6ecba89ed8f85b1b509f18171cfa15119384ce801", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "13f4e8aab63ce7cbab5326cab67676ee3c2a9c3cbb9786baa18bed9833c4a90e", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "3263d9f575c947aaa3a9199b542dac7e607afdfbb223d70449a2f173be3df78c", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "7a10c977ee648eb1975286b313f009b3508183efe9e942cd710ae6ad5ae279a3", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "6fc7f25165300eda1ecd5bd9d516ba958f045de8951d81ff296c72ce461b5974", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/benchmarks/results/native_bfv_rns_32.json b/benchmarks/results/native_bfv_rns_32.json new file mode 100644 index 0000000..f6b06ca --- /dev/null +++ b/benchmarks/results/native_bfv_rns_32.json @@ -0,0 +1,242 @@ +{ + "environment": { + "utc": "2026-09-10T23:06:32.636539+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "ff8d5a29a58874e6498e1cdae6b3fcde21e19a5d", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "32", + "--backends", + "optimized,rns", + "--seal", + "--repeats", + "4", + "--profile-dir", + "/tmp/xtrace-bfv-rns-profiles", + "--json-out", + "benchmarks/results/native_bfv_rns_32.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for both backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "response_modulus_bits": 50, + "device": "cpu" + }, + "seed": 1337, + "vectors": 32, + "setup_timings": { + "key_generation_s": 4.319804938975722, + "index_encrypt_s": 0.21808502497151494, + "query_encrypt_s": 0.1036710909102112, + "public_key_export_s": 0.7804761249572039, + "optimized_public_key_import_s": 0.8426786060445011, + "rns_public_key_import_s": 0.8472385408822447 + }, + "runs": [ + { + "backend": "optimized", + "repeat": 0, + "server_compute_s": 1.7459196471609175 + }, + { + "backend": "rns", + "repeat": 0, + "server_compute_s": 0.5007300598081201 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 0.07752445782534778 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 0.07478765211999416 + }, + { + "backend": "rns", + "repeat": 1, + "server_compute_s": 0.26233402313664556 + }, + { + "backend": "optimized", + "repeat": 1, + "server_compute_s": 1.6846589841879904 + }, + { + "backend": "optimized", + "repeat": 2, + "server_compute_s": 1.6894245909061283 + }, + { + "backend": "rns", + "repeat": 2, + "server_compute_s": 0.2628466538153589 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 0.07481790310703218 + }, + { + "backend": "seal", + "repeat": 3, + "server_compute_s": 0.07475750101730227 + }, + { + "backend": "rns", + "repeat": 3, + "server_compute_s": 0.2623735030647367 + }, + { + "backend": "optimized", + "repeat": 3, + "server_compute_s": 1.6904642400331795 + } + ], + "summary": { + "optimized": { + "first_search_s": 1.7459196471609175, + "warm_searches_s": [ + 1.6846589841879904, + 1.6894245909061283, + 1.6904642400331795 + ], + "warm_median_s": 1.6894245909061283 + }, + "rns": { + "first_search_s": 0.5007300598081201, + "warm_searches_s": [ + 0.26233402313664556, + 0.2628466538153589, + 0.2623735030647367 + ], + "warm_median_s": 0.2623735030647367 + }, + "seal": { + "first_search_s": 0.07752445782534778, + "warm_searches_s": [ + 0.07478765211999416, + 0.07481790310703218, + 0.07475750101730227 + ], + "warm_median_s": 0.07478765211999416 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "optimized": { + "first": 1.0, + "warm_median": 1.0 + }, + "rns": { + "first": 3.4867482248418527, + "warm_median": 6.43900611598454 + }, + "seal": { + "first": 22.520888196267567, + "warm_median": 22.5896193157061 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 865100, + "query_bytes": 432689, + "public_key_bytes": 39584869, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 103240, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "sizes": { + "encrypted_index_bytes": 737480, + "public_keys_bytes": 85602442, + "query_bytes": 368754, + "response_bytes": 102494, + "query_plus_response_bytes": 471248 + }, + "caches": { + "optimized": { + "switch_keys": 11, + "key_payload_bytes": 35807568, + "transform_table_bytes": 0 + }, + "rns": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "switch_keys": 11, + "key_payload_bytes": 34603008 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 31, + "distance": 0 + }, + { + "index": 19, + "distance": 241 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "839279a7bfd41e46e02a4f49f51afc6e3fbe2e3c33d5a4f9745eb3107bf399c1", + "benchmarks/bfv_client_matrix.py": "b5539dd816f6dbd8f26fadde4f72c9582a8fa4f67bf8a64b3a874d38f576216c", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "03205e7775265873da6da51a2d26aa0e63a59c2961dab31f29e71e8b9b92a84f", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "6f03ac75f42218170da2bd785605ee394940a202afdd5471545a8ae3fdc8c3d8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "de206a348b704982cfa58526189af7c61cf0628807457f7beddb00498a2ea78a", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "6f97a52f5b4b9981533fbed5d043e4e2179906491d0805609fd76e025e38bbc9", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "fb72e876eb585874c8dd42a6ecba89ed8f85b1b509f18171cfa15119384ce801", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "13f4e8aab63ce7cbab5326cab67676ee3c2a9c3cbb9786baa18bed9833c4a90e", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "3263d9f575c947aaa3a9199b542dac7e607afdfbb223d70449a2f173be3df78c", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "7a10c977ee648eb1975286b313f009b3508183efe9e942cd710ae6ad5ae279a3", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "6fc7f25165300eda1ecd5bd9d516ba958f045de8951d81ff296c72ce461b5974", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/benchmarks/results/native_bfv_rns_profile_32.txt b/benchmarks/results/native_bfv_rns_profile_32.txt new file mode 100644 index 0000000..742185d --- /dev/null +++ b/benchmarks/results/native_bfv_rns_profile_32.txt @@ -0,0 +1,83 @@ +Additional cProfile runs for 32 vectors x 512 bits. +These instrumented runs are excluded from benchmark timings. + +optimized +Thu Sep 10 19:06:32 2026 /tmp/xtrace-bfv-rns-profiles/optimized.prof + + 3581976 function calls in 2.024 seconds + + Ordered by: cumulative time + List reduced from 74 to 30 due to restriction <30> + + ncalls tottime percall cumtime percall filename:lineno(function) + 1 0.001 0.001 2.046 2.046 bfv_server.py:109(evaluate) + 1 0.002 0.002 2.044 2.044 bfv_client.py:273(encode_hamming_server_packed) + 2 0.001 0.000 1.860 0.930 bfv_client.py:238(_distance_tile) + 2 0.022 0.011 1.859 0.930 bfv_evaluator.py:242(hamming_tile) + 19 0.008 0.000 1.318 0.069 bfv_evaluator.py:269(rotate_rows) + 21 0.000 0.000 1.304 0.062 bfv_evaluator.py:146(_switch) + 21 1.185 0.056 1.304 0.062 bfv_evaluator.py:55(apply) + 158 0.059 0.000 0.352 0.002 {built-in method builtins.any} + 2 0.000 0.000 0.333 0.166 bfv_evaluator.py:237(xor) + 2 0.003 0.001 0.315 0.157 bfv_evaluator.py:221(multiply) + 11 0.000 0.000 0.201 0.018 bfv.py:289(validate_ciphertext) + 180246 0.063 0.000 0.190 0.000 bfv.py:300() + 2 0.003 0.001 0.172 0.086 bfv.py:447(multiply) + 63 0.000 0.000 0.170 0.003 bfv_evaluator.py:118(_validate) + 19 0.000 0.000 0.164 0.009 bfv_evaluator.py:158(add) + 2 0.002 0.001 0.139 0.070 bfv_evaluator.py:198(relinearize) + 196657 0.030 0.000 0.139 0.000 {built-in method builtins.isinstance} + 21 0.000 0.000 0.112 0.005 bfv_evaluator.py:139(_pair) + 196632 0.020 0.000 0.108 0.000 :117(__instancecheck__) + 10 0.049 0.005 0.107 0.011 bfv.py:39(_poly_product) + 2 0.000 0.000 0.102 0.051 bfv_evaluator.py:186(multiply_plain) + 2 0.000 0.000 0.102 0.051 bfv.py:398(multiply_plain) + 42 0.019 0.000 0.099 0.002 bfv_evaluator.py:63(finish) + 196632 0.038 0.000 0.088 0.000 {built-in method _abc._abc_instancecheck} + 1048704 0.088 0.000 0.088 0.000 bfv_evaluator.py:133() + 2 0.000 0.000 0.074 0.037 bfv.py:351(_pair) + 57 0.016 0.000 0.062 0.001 bfv_evaluator.py:164() + 344106 0.062 0.000 0.062 0.000 bfv_evaluator.py:69() + 3 0.001 0.000 0.059 0.020 bfv.py:566(ciphertext_from_ints) + 57 0.000 0.000 0.055 0.001 bfv_evaluator.py:289() + + +rns +Thu Sep 10 19:06:32 2026 /tmp/xtrace-bfv-rns-profiles/rns.prof + + 770665 function calls in 0.342 seconds + + Ordered by: cumulative time + List reduced from 62 to 30 due to restriction <30> + + ncalls tottime percall cumtime percall filename:lineno(function) + 1 0.001 0.001 0.342 0.342 bfv_server.py:109(evaluate) + 1 0.002 0.002 0.340 0.340 bfv_client.py:273(encode_hamming_server_packed) + 2 0.000 0.000 0.213 0.107 bfv_client.py:238(_distance_tile) + 2 0.000 0.000 0.213 0.107 bfv_evaluator.py:242(hamming_tile) + 2 0.000 0.000 0.184 0.092 bfv_rns.py:68(hamming_tile) + 2 0.178 0.089 0.178 0.089 {built-in method xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns.hamming_tile} + 32 0.011 0.000 0.121 0.004 {built-in method builtins.any} + 5 0.000 0.000 0.090 0.018 bfv.py:289(validate_ciphertext) + 81930 0.028 0.000 0.085 0.000 bfv.py:300() + 98353 0.015 0.000 0.069 0.000 {built-in method builtins.isinstance} + 3 0.001 0.000 0.059 0.020 bfv.py:566(ciphertext_from_ints) + 98328 0.010 0.000 0.054 0.000 :117(__instancecheck__) + 98328 0.019 0.000 0.044 0.000 {built-in method _abc._abc_instancecheck} + 1 0.000 0.000 0.027 0.027 bfv.py:504(modulus_switch) + 98328 0.009 0.000 0.025 0.000 :121(__subclasscheck__) + 7 0.000 0.000 0.019 0.003 bfv_evaluator.py:118(_validate) + 2 0.001 0.000 0.018 0.009 bfv.py:244(_plaintext) + 1 0.000 0.000 0.017 0.017 bfv.py:545(ciphertext_to_ints) + 3 0.000 0.000 0.016 0.005 bfv_evaluator.py:139(_pair) + 98328 0.016 0.000 0.016 0.000 {built-in method _abc._abc_subclasscheck} + 16386 0.003 0.000 0.015 0.000 bfv.py:248() + 1 0.000 0.000 0.013 0.013 bfv_evaluator.py:269(rotate_rows) + 1 0.000 0.000 0.011 0.011 bfv_rns.py:59(rotate) + 114702 0.010 0.000 0.010 0.000 bfv_evaluator.py:133() + 1 0.000 0.000 0.009 0.009 bfv_evaluator.py:158(add) + 1 0.009 0.009 0.009 0.009 {built-in method xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns.rotate_rows} + 3 0.001 0.000 0.009 0.003 bfv.py:524() + 16386 0.004 0.000 0.008 0.000 bfv.py:525() + 12 0.006 0.000 0.006 0.000 {built-in method gmpy2.gmpy2.unpack} + 6 0.000 0.000 0.004 0.001 bfv_rns.py:40(_unpack) diff --git a/docs/api_rst/xtrace_sdk.crypto.encryption.rst b/docs/api_rst/xtrace_sdk.crypto.encryption.rst index 3ad0f74..acb1be0 100644 --- a/docs/api_rst/xtrace_sdk.crypto.encryption.rst +++ b/docs/api_rst/xtrace_sdk.crypto.encryption.rst @@ -20,6 +20,14 @@ xtrace\_sdk.crypto.encryption.bfv\_evaluator module :show-inheritance: :undoc-members: +xtrace\_sdk.crypto.encryption.bfv\_rns module +------------------------------------------- + +.. automodule:: xtrace_sdk.x_vec.crypto.encryption.bfv_rns + :members: + :show-inheritance: + :undoc-members: + xtrace\_sdk.crypto.encryption.aes module ---------------------------------------- diff --git a/docs/index.rst b/docs/index.rst index be92059..06b3abc 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -82,6 +82,7 @@ Choose your path: :caption: Research research/native-bfv + research/native-bfv-ntt research/bfv-packed-hamming .. toctree:: diff --git a/docs/research/native-bfv-ntt.md b/docs/research/native-bfv-ntt.md new file mode 100644 index 0000000..443c51c --- /dev/null +++ b/docs/research/native-bfv-ntt.md @@ -0,0 +1,228 @@ +# Native BFV RNS/NTT server arithmetic + +The optional `server_backend="rns"` evaluator uses independently implemented C++ +kernels to accelerate native BFV. It retains the existing keys, ciphertext +modulus, error distribution, packing layout and wire format. The default +`"optimized"` GMP evaluator and the original `"reference"` evaluator remain +available. The earlier SEAL experiment is preserved for comparison. + +## What we learned from SEAL + +The source review used the pinned **SEAL v4.1.2** tag, rather than an unversioned +branch. The comparison executable is the separately installed TenSEAL prototype; +its package version and actual modulus sizes are recorded by the benchmark. + +| SEAL source | Relevant technique | Native implementation | +| --- | --- | --- | +| [`evaluator.cpp`, `bfv_multiply`](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/evaluator.cpp) | RNS base extension, NTT polynomial products, and scale/base conversion | Auxiliary RNS bases reconstruct the exact signed tensor product before our existing rounding rule | +| [`keygenerator.cpp`, `generate_one_kswitch_key`](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/keygenerator.cpp) | Evaluation keys are generated in NTT form | Lazily convert our existing gadget keys into NTT form and cache them | +| [`evaluator.cpp`, `switch_key_inplace`](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/evaluator.cpp) | Pointwise key products and delayed reduction in wide accumulators | Sum gadget products in unsigned 128-bit accumulators before inverse transforms | +| [`util/ntt.cpp`](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/util/ntt.cpp) | Prepared transform tables and inverse scaling | Cached roots, twists and inverse scaling; matching forward/inverse orders avoid explicit bit reversal | +| [`util/uintarithsmallmod.h`](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/util/uintarithsmallmod.h) | Precomputed quotient operands for modular multiplication | Shoup multiplication avoids division in transform butterflies | + +SEAL's multiplication also uses specialized RNS conversion routines in +[`util/rns.cpp`](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/util/rns.cpp). +Our backend deliberately keeps the original BFV scheme and exact GMP +reconstruction, so it does not implement SEAL's full BEHZ multiplication or +special-prime key-switching scheme. No SEAL source is compiled or linked into +the native extension. These are independent implementations of the arithmetic +techniques identified during that review. + +## Arithmetic and correctness bounds + +The existing q is a single large prime and need not support an NTT of the +required length. The backend instead selects distinct 60-bit auxiliary primes +`p_i = 1 mod 2N`. A polynomial is temporarily represented by its residues modulo +each `p_i`. These primes are arithmetic scratch space; they do not replace q. + +Each prime gets an N-point negacyclic transform. Multiplying coefficient i by +`psi^i`, where psi has order 2N, changes multiplication modulo `X^N+1` into a +cyclic transform calculation. The forward transform produces bit-reversed +values; the complementary inverse accepts that same order and combines division +by N with the inverse twist. Pointwise products compute the convolution. + +The Chinese Remainder Theorem reconstructs the **signed integer** result. Let +`M = product(p_i)`, b be the gadget digit width and L the number of digits. +We choose enough primes that: + +- Key switching: `M > 2*N*L*(2^b - 1)*(q - 1)`. +- Ciphertext multiplication, including the two cross products: + `M > 4*N*(q - 1)^2`. +- A general polynomial product uses the bound + `M > 2*N*max(lhs)*max(rhs)`. + +Every result coefficient then lies strictly between `-M/2` and `M/2`, making +centered CRT reconstruction unique. For BFV multiplication, the reconstructed +coefficient c is rounded with `floor((2*t*c + q)/(2*q))` before reduction modulo +q. Premature reduction modulo q would destroy information needed by that step. +The default key switch uses four auxiliary primes; the tensor product uses +seven. All transforms and rounding use integer arithmetic. + +Transform values remain canonical residues. A fixed twiddle w stores +`floor(w*2^64/p)`; its high-word product gives a quotient estimate requiring at +most one correction. Gadget products are below `2^120`, allowing batches of +256 terms in 128-bit accumulators. Longer gadgets reduce between batches. +Digits of at most 60 bits are extracted directly from GMP limbs on supported +64-bit GMP builds; larger digits retain a GMP path. + +The complete subtract/square/relinearize/rotate/add/mask circuit for one Hamming +tile stays inside C++. This avoids returning thousands of intermediate GMP +coefficients to Python after each rotation. Result merging and public client +interfaces retain their existing structure. Native calls release the GIL during +computation; immutable shared plans and per-call scratch arrays support concurrent +evaluations. Input lengths, coefficient ranges, capsule types and native plan +ownership are checked at the extension boundary. + +## Build and use + +From the repository root, using a Linux 64-bit environment with a C++17 compiler, +Python development headers and GMP development files: + +```bash +make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" +``` + +There is no additional runtime Python package, pybind11, SEAL, or CUDA dependency. +The built module needs the system GMP and C++ runtime libraries. It is optional; +the Python/GMP path works without a compiler or this extension. Wheels include +the native binary when built locally and receive a Python/OS/architecture tag. + +```python +import json +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + +client = BFVClient(embed_len=512) +server = BFVClient(skip_key_gen=True, server_backend="rns") +server.load_config(json.loads(client.stringify_config())) +server.load_stringified_keys(client.stringify_pk()) +assert server.keys is None + +query = [0, 1] * 256 +vectors = [query[:], [1 - bit for bit in query]] +response = server.encode_hamming_server_packed( + client.encrypt_vec_one(query), client.encrypt_vec_packed(vectors), len(vectors) +) +assert client.decode_hamming_client_packed(response, len(vectors)) == [0, 512] +``` + +Reuse the server to amortize transformed-key preparation. Key loading discards +the cache. Select `server_backend="optimized"` for the previous GMP optimization +or `"reference"` for the original arithmetic. `"rns"` reports a build instruction +if the native extension cannot be imported; it does not silently change the +requested backend. The setting is runtime-only and does not alter saved crypto +configurations. + +## Code map + +| File | Responsibility | +| --- | --- | +| `crypto/bfv_cpu_ext/rns_ntt.h` | NTT plans, fixed-multiplier reduction, exact CRT, gadget multiplication, BFV scaling and native tile evaluation | +| `crypto/bfv_cpu_ext/bindings.cpp` | Checked CPython boundary and native object lifetimes | +| `crypto/bfv_cpu_ext/Makefile` | Optional local C++ build | +| `crypto/encryption/bfv_rns.py` | Existing GMP polynomial types to native buffers and cached key handles | +| `crypto/encryption/bfv_evaluator.py` | Backend selection and common validation/reference arithmetic | +| `crypto/bfv_client.py` | Existing individual/packed Hamming interfaces | +| `tests/x_vec/test_bfv_rns.py` | Independent arithmetic oracles, bounds, native lifetimes, concurrency and default parameters | +| `tests/x_vec/native/test_bfv_rns.cpp` | Standalone native oracles for sanitizer runs | +| `benchmarks/bfv_server.py` | Repeated native comparisons and optional SEAL comparison | + +Crypto paths above are relative to `src/xtrace_sdk/x_vec/`. + +## Verification and benchmarks + +```bash +.venv/bin/python -m pytest tests/x_vec/test_bfv_encryption.py \ + tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py \ + tests/x_vec/test_bfv_client.py -q + +g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer \ + -I src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext \ + tests/x_vec/native/test_bfv_rns.cpp -lgmpxx -lgmp -o /tmp/test_bfv_rns +/tmp/test_bfv_rns + +.venv/bin/python benchmarks/bfv_server.py --num-vectors 1024 \ + --backends optimized,rns --seal --repeats 3 --json-out /tmp/bfv-rns-1024.json +``` + +Omit `--seal` when TenSEAL is not installed. The test suite covers zero, maximum, +sparse and random coefficients; digit boundaries at 59/60/61 bits; 512 one-bit +digits; exact signed rounding; repeated multiplication; malformed buffers; +partial tiles and multiple responses; public-only evaluation in another process; +key reload; and missing-extension fallback. Standalone tests reach N=32768. +LeakSanitizer cannot run under the local tracing sandbox: setting +`ASAN_OPTIONS=detect_leaks=0` keeps address and undefined-behavior checks enabled +there. CI runs the standalone sanitizer executable without that override. + +Native backends share identical public keys and encrypted inputs. Every timed +search must produce identical native ciphertexts and correct plaintext distances. +SEAL uses the same vectors with its own keys and parameters; its result is +checked against every plaintext distance. Search order alternates, and first-use +preparation is reported separately from subsequent warm searches. Profiles are +extra runs excluded from timing samples. + +Native timings include ciphertext decoding/encoding and terminal compaction, +but exclude the outer MessagePack envelope. SEAL timings include its envelope +and temporary-file binding I/O. Neither includes key import, index/query +encryption, client decryption, network time, or top-k content retrieval. The +comparison measures practical server calls; it does not establish equal security +or identical cryptographic work across schemes. + +Validation for this change passed 151 offline tests across the BFV experiment, +CLI and SDK crypto suites, including 69 BFV tests. API integration and GPU tests +were excluded. Ruff, formatting checks, mypy and the standalone address/undefined- +behavior sanitizer run passed. A built platform-tagged wheel loaded its own +native module and completed an encrypted Hamming search. The Sphinx build passed +with the pre-existing README cross-reference warning in `bfv-packed-hamming.md`. + +### Recorded measurements + +Measured on 2026-09-10 on an AMD Ryzen 7 5800X with Python 3.12.3, GMP 6.3.0, +GCC 13.3.0 (`-O3`, without `-march=native`) and TenSEAL 0.3.16. Each vector has +512 bits. Native parameters +are N=8192, t=65537, q=180 bits, six 30-bit gadget digits and a 50-bit terminal +response modulus. The SEAL context has four active primes of 43/43/44/44 bits, +plus a 44-bit special prime for keys, with its TC128 security setting. + +| Vectors | Server backend | First search (s) | Warm median (s) | +| ---: | --- | ---: | ---: | +| 32 | Previous GMP (`optimized`) | 1.7459 | 1.6894 | +| 32 | Native C++ RNS/NTT (`rns`) | 0.5007 | 0.2624 | +| 32 | Preserved SEAL prototype | 0.0775 | 0.0748 | +| 1,024 | Previous GMP (`optimized`) | 55.7613 | 55.9025 | +| 1,024 | Native C++ RNS/NTT (`rns`) | 8.3291 | 8.0355 | +| 1,024 | Preserved SEAL prototype | 2.4032 | 2.3901 | + +The 32-vector run has one first search and three warm searches per backend; +the 1,024-vector run has one first search and two warm searches. On the larger +workload the native backend is **6.96 times faster** than the previous GMP +backend. SEAL remains **3.36 times faster**, compared with 23.39 times faster +than GMP in the same measurement. These are local repeated measurements, not +a claim about all machines or parameter sets. + +Both native backends returned exactly the same encrypted responses on every +run. For 1,024 vectors, each response is 102,496 bytes and query plus response +is 471,250 bytes with the benchmark's MessagePack envelope. The minimum native +response noise budget is 26 bits. The arithmetic optimization changes neither +those ciphertext bytes nor the original scheme's security assumptions. + +The 1,024-vector native server caches 16 transformed keys: 50,331,648 bytes of +coefficient arrays plus 2,752,512 bytes of transform tables. The previous GMP +server reports 52,083,728 bytes of prepared-key payload. Both are additional +to the loaded public key and exclude containers, CRT constants and temporary +allocations; these figures are not peak process memory. + +Raw samples, parameters, correctness checks, source/binary hashes and timing +boundaries are saved in `benchmarks/results/native_bfv_rns_32.json` and +`benchmarks/results/native_bfv_rns_1024.json`. The additional Python profile is +`benchmarks/results/native_bfv_rns_profile_32.txt`. Profiles show native tile +evaluation alongside the remaining Python validation and serialization work; +they do not separately attribute time inside the C++ kernel. + +## Remaining differences from SEAL + +The native backend still reconstructs large GMP coefficients after key switches +and uses them between operations. SEAL has an integrated RNS representation and +specialized base conversion. Our backend also allocates scratch arrays per call +and retains Python wire parsing/validation and result merging. Those are further +optimization opportunities. Current code remains experimental, variable-time +leveled BFV with the same limitations as the [native implementation](native-bfv.md). diff --git a/docs/research/native-bfv.md b/docs/research/native-bfv.md index 0237b64..d50b432 100644 --- a/docs/research/native-bfv.md +++ b/docs/research/native-bfv.md @@ -16,14 +16,15 @@ All paths are relative to the repository root. | File | Responsibility | | --- | --- | | `src/xtrace_sdk/x_vec/crypto/encryption/bfv.py` | Polynomial arithmetic, key generation, encryption/decryption, batch encoding, homomorphic operations, evaluation keys, modulus switching, serialization | -| `src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py` | Cached public-key server arithmetic, with optimized and reference backends | +| `src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py` | Cached public-key server arithmetic, with optimized, reference and optional RNS backends | +| `src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py` and `src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/` | Optional native RNS/NTT CPU arithmetic and its Python boundary | | `src/xtrace_sdk/x_vec/crypto/bfv_client.py` | `HammingClientBase` interface, binary-vector layout, public-only server evaluation, packed responses, client decoding | | `src/xtrace_sdk/x_vec/utils/xtrace_types.py` | BFV parameter, polynomial, ciphertext and key types alongside the existing scheme types | | `tests/x_vec/test_bfv_encryption.py` | Independent polynomial oracle and cryptographic arithmetic tests | | `tests/x_vec/test_bfv_evaluator.py` | Fused key-switch oracle, exact reference comparisons, validation and cache lifecycle tests | | `tests/x_vec/test_bfv_client.py` | Client, persistence, process separation, packing boundaries, default settings and optional SEAL comparison | | `benchmarks/bfv_client_matrix.py` | Native BFV and existing CPU Paillier/Lookup timing and wire-size comparison | -| `benchmarks/bfv_server.py` | Paired native server comparison on identical encrypted inputs, with first/warm searches and optional profiles | +| `benchmarks/bfv_server.py` | Native comparisons on identical encrypted inputs, with first/warm searches, optional SEAL comparison and profiles | The only shared cryptographic interface correction is to the return annotations in `HomomorphicBase`: encryption returns a ciphertext, and decryption returns a @@ -79,6 +80,10 @@ Reuse a server instance across queries to reuse its prepared public keys. `load_stringified_keys()` discards the evaluator cache when loading keys. Change keys through this method rather than mutating an initialized public-key dictionary. +An optional `server_backend="rns"` uses our compiled CPU RNS/NTT kernels. See +[native BFV RNS/NTT arithmetic](native-bfv-ntt.md) for the SEAL source review, +build instructions, exact arithmetic bounds and benchmark comparisons. + BFV is available directly as a local client; `ExecutionContext`, `DataLoader`, and the current XTrace HTTP endpoints do not implement its index/wire protocol. Adding a production server path is separate work. Distances are computed at the @@ -134,15 +139,16 @@ convolution. Unpacking and subtracting the upper half implements `X^N = -1`. The result remains exact, including negative coefficients, until the caller scales or reduces it. Squaring reuses the packed operand and one cross product. -This is **Kronecker substitution**, not SEAL's RNS implementation. Ciphertext -arithmetic uses a single large modulus; only the plaintext batch transform uses -an NTT. The small-ring schoolbook oracle tests products through 512-bit +This is **Kronecker substitution**. The reference and optimized GMP backends use +a single large modulus; only their plaintext batch transform uses an NTT. +The small-ring schoolbook oracle tests products through 512-bit coefficients, including worst-case all-maximum inputs that exercise radix carries. The optimized evaluator accelerates gadget decomposition/key switching as -described below. RNS/NTT polynomial products and then CUDA kernels are possible -next steps. Any replacement must preserve the unreduced product needed by -BFV's scale-and-round step. +described below. The optional [RNS/NTT backend](native-bfv-ntt.md) now implements +an alternative polynomial multiplication path on the CPU. It preserves the +unreduced product needed by BFV's scale-and-round step. CUDA kernels remain +future work. ### Cached and fused server evaluation diff --git a/hatch_build.py b/hatch_build.py index 674d1be..8289bda 100644 --- a/hatch_build.py +++ b/hatch_build.py @@ -1,4 +1,4 @@ -"""Hatchling build hook: tag wheels as platform-specific when GPU `.so` +"""Hatchling build hook: tag wheels as platform-specific when native `.so` artifacts are bundled, so pip won't install a `py3-none-any` wheel onto a machine whose Python ABI / OS / arch can't actually load the extension. @@ -17,12 +17,13 @@ class CustomBuildHook(BuildHookInterface): PLUGIN_NAME = "custom" def initialize(self, version: str, build_data: dict) -> None: - gpu_so_globs = [ + native_so_globs = [ "src/xtrace_sdk/x_vec/crypto/paillier_gpu_ext/*.so", "src/xtrace_sdk/x_vec/crypto/paillier_lookup_gpu_ext/*.so", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/*.so", ] has_binary = any( - glob.glob(os.path.join(self.root, pattern)) for pattern in gpu_so_globs + glob.glob(os.path.join(self.root, pattern)) for pattern in native_so_globs ) if has_binary: build_data["pure_python"] = False diff --git a/pyproject.toml b/pyproject.toml index 6707598..fae9b0b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -41,11 +41,12 @@ source = "vcs" local_scheme = "no-local-version" [tool.hatch.build] -# Force-include the GPU extension binaries even though *.so is in .gitignore. -# These are produced by build_gpu_binaries.sh prior to `uv build` in CI. +# Force-include optional native extension binaries even though *.so is ignored. +# GPU binaries use build_gpu_binaries.sh; BFV CPU kernels use their local Makefile. artifacts = [ "src/xtrace_sdk/x_vec/crypto/paillier_gpu_ext/*.so", "src/xtrace_sdk/x_vec/crypto/paillier_lookup_gpu_ext/*.so", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/*.so", ] [tool.hatch.build.targets.wheel] diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_client.py index 9c72dd8..f0d2b23 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_client.py +++ b/src/xtrace_sdk/x_vec/crypto/bfv_client.py @@ -38,7 +38,8 @@ class BFVClient(HammingClientBase): :param response_modulus_bits: Smaller modulus used after the Hamming circuit. :param skip_key_gen: Construct an empty client for loading saved keys. :param device: ``cpu`` or ``auto``; a CUDA implementation is not yet available. - :param server_backend: ``optimized`` (cached GMP) or ``reference`` server arithmetic. + :param server_backend: ``optimized`` (cached GMP), ``reference``, or ``rns`` + (optional compiled CPU RNS/NTT extension) server arithmetic. This runtime choice is independent of the serialized crypto configuration. """ @@ -60,8 +61,8 @@ def __init__( if device == "gpu": raise NotImplementedError("Native BFV currently supports only the CPU backend") self.device = "cpu" - if server_backend not in ("optimized", "reference"): - raise ValueError("server_backend must be 'optimized' or 'reference'") + if server_backend not in ("optimized", "reference", "rns"): + raise ValueError("server_backend must be 'optimized', 'reference', or 'rns'") self.server_backend = server_backend self._server_evaluator: BFVEvaluator | None = None self._evaluator_public_key: BFVPublicKey | None = None @@ -238,10 +239,6 @@ def _distance_tile( self, query: BFVCiphertext, tile: BFVCiphertext, count: int ) -> BFVCiphertext: evaluator = self._evaluator() - distance = evaluator.xor(query, tile) - for i in range(self.padded_embed_len.bit_length() - 1): - step = self.lanes_per_row * (1 << i) - distance = evaluator.add(distance, evaluator.rotate_rows(distance, step)) # The rotate/add tree repeats sums in each dimension block. Retain only # the first block in each row and clear unused lanes in a partial tile. if count not in self._mask_cache: @@ -252,7 +249,10 @@ def _distance_tile( row, col = divmod(lane, self.lanes_per_row) mask[row * (len(mask) // 2) + col] = 1 self._mask_cache[count] = BFV.batch_encode(mask, self.params) - return evaluator.multiply_plain(distance, self._mask_cache[count]) + steps = [ + self.lanes_per_row * (1 << i) for i in range(self.padded_embed_len.bit_length() - 1) + ] + return evaluator.hamming_tile(query, tile, steps, self._mask_cache[count]) def encode_hamming_server( self, diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile new file mode 100644 index 0000000..9657a09 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile @@ -0,0 +1,23 @@ +# Optional CPU BFV extension; no SEAL, CUDA, pybind11 or Python build dependency. +# From the repository root: +# make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$(pwd)/.venv/bin/python" +# Requires a C++17 compiler, Python development headers, and GMP development files. +PYTHON ?= python3 +CXX ?= c++ +PY_INC := $(shell $(PYTHON) -c 'import sysconfig; print(sysconfig.get_paths()["include"])') +EXT := $(shell $(PYTHON) -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))') +TARGET := _bfv_rns$(EXT) +CXXFLAGS ?= -O3 +CPPFLAGS ?= +LDFLAGS ?= +LDLIBS ?= -lgmpxx -lgmp + +.PHONY: all clean +all: $(TARGET) + +$(TARGET): bindings.cpp rns_ntt.h Makefile + $(CXX) $(CPPFLAGS) $(CXXFLAGS) -std=c++17 -Wall -Wextra -fPIC -shared \ + -I$(PY_INC) bindings.cpp $(LDFLAGS) $(LDLIBS) -o $@ + +clean: + rm -f $(TARGET) diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/__init__.py b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/__init__.py new file mode 100644 index 0000000..9964a2c --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/__init__.py @@ -0,0 +1 @@ +"""Optional, independently implemented CPU kernels for native BFV arithmetic.""" diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi new file mode 100644 index 0000000..5d9bacd --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi @@ -0,0 +1,24 @@ +from typing import Any + +ABI_VERSION: int + +def create_ring(n: int, q_hex: str, digit_bits: int) -> object: ... +def compile_key(ring: object, key: tuple[tuple[bytes, bytes], ...]) -> object: ... +def apply_key(key: object, poly: bytes) -> tuple[bytes, bytes]: ... +def ring_product(ring: object, lhs: bytes, rhs: bytes) -> bytes: ... +def multiply( + ring: object, lhs: tuple[bytes, ...], rhs: tuple[bytes, ...] | None, t: int +) -> tuple[bytes, bytes, bytes]: ... +def rotate_rows( + key: object, ciphertext: tuple[bytes, ...], exponent: int +) -> tuple[bytes, bytes]: ... +def hamming_tile( + relin: object, + query: tuple[bytes, ...], + tile: tuple[bytes, ...], + rotations: tuple[tuple[int, object], ...], + mask: bytes, + t: int, +) -> tuple[bytes, bytes]: ... +def ring_info(ring: object) -> dict[str, Any]: ... +def key_bytes(key: object) -> int: ... diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp new file mode 100644 index 0000000..3debc1a --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp @@ -0,0 +1,279 @@ +// CPython boundary for the optional native BFV RNS/NTT kernels. +#define PY_SSIZE_T_CLEAN +#include +#include "rns_ntt.h" +#include + +using namespace xtrace_bfv; +namespace { +using RingPtr = std::shared_ptr; +using KeyPtr = std::shared_ptr; +constexpr const char* ring_name = "xtrace.bfv.rns.ring.v1"; +constexpr const char* key_name = "xtrace.bfv.rns.key.v1"; + +struct PythonError {}; +struct WithoutGIL { + PyThreadState* state = PyEval_SaveThread(); + ~WithoutGIL() { PyEval_RestoreThread(state); } +}; + +template T get_capsule(PyObject* object, const char* name) { + auto pointer = static_cast(PyCapsule_GetPointer(object, name)); + if (!pointer) throw PythonError{}; + return *pointer; +} +template void delete_capsule(PyObject* capsule) { + delete static_cast(PyCapsule_GetPointer(capsule, PyCapsule_GetName(capsule))); +} +template PyObject* capsule(T value, const char* name) { + auto owner = std::make_unique(std::move(value)); + PyObject* result = PyCapsule_New(owner.get(), name, delete_capsule); + if (result) owner.release(); + return result; +} + +// Only fixed-width little-endian bytes cross this private kernel boundary. +// Validate lengths and coefficient bounds before any native transform accesses +// memory or relies on its CRT reconstruction bound. +Polynomial read_poly(PyObject* object, const Ring& ring) { + if (!PyBytes_Check(object)) { + PyErr_SetString(PyExc_TypeError, "Polynomial must be fixed-width bytes"); + throw PythonError{}; + } + if (PyBytes_GET_SIZE(object) != static_cast(ring.n * ring.coefficient_bytes)) + throw std::invalid_argument("Incorrect polynomial byte length"); + const auto* data = PyBytes_AS_STRING(object); + Polynomial result(ring.n); + for (std::size_t i = 0; i < ring.n; ++i) { + mpz_import(result[i].get_mpz_t(), ring.coefficient_bytes, -1, 1, 0, 0, + data + i * ring.coefficient_bytes); + if (result[i] >= ring.q) throw std::invalid_argument("Noncanonical polynomial coefficient"); + } + return result; +} + +std::array read_pair(PyObject* object, const Ring& ring) { + if (!PyTuple_Check(object) || PyTuple_GET_SIZE(object) != 2) + throw std::invalid_argument("Expected two ciphertext components"); + return {read_poly(PyTuple_GET_ITEM(object, 0), ring), read_poly(PyTuple_GET_ITEM(object, 1), ring)}; +} + +PyObject* write_poly(const Polynomial& poly, const Ring& ring) { + if (poly.size() != ring.n) throw std::logic_error("Native kernel returned an incorrect polynomial length"); + PyObject* result = PyBytes_FromStringAndSize(nullptr, ring.n * ring.coefficient_bytes); + if (!result) return nullptr; + auto* output = PyBytes_AS_STRING(result); + std::fill(output, output + ring.n * ring.coefficient_bytes, 0); + for (std::size_t i = 0; i < ring.n; ++i) { + // All public return paths first reduce into [0,q); keep the guard here + // so a future kernel mistake cannot overrun a fixed-width buffer. + if (poly[i] < 0 || poly[i] >= ring.q) { + Py_DECREF(result); + throw std::logic_error("Native kernel returned a noncanonical coefficient"); + } + mpz_export(output + i * ring.coefficient_bytes, nullptr, -1, 1, 0, 0, poly[i].get_mpz_t()); + } + return result; +} + +template +PyObject* write_components(const std::array& components, const Ring& ring) { + PyObject* result = PyTuple_New(Count); + if (!result) return nullptr; + try { + for (std::size_t i = 0; i < Count; ++i) { + PyObject* poly = write_poly(components[i], ring); + if (!poly) { Py_DECREF(result); return nullptr; } + PyTuple_SET_ITEM(result, i, poly); + } + } catch (...) { Py_DECREF(result); throw; } + return result; +} + +template PyObject* checked(Function&& function) { + try { return function(); } + catch (const PythonError&) { return nullptr; } + catch (const std::bad_alloc&) { return PyErr_NoMemory(); } + catch (const std::invalid_argument& error) { PyErr_SetString(PyExc_ValueError, error.what()); } + catch (const std::exception& error) { PyErr_SetString(PyExc_RuntimeError, error.what()); } + return nullptr; +} + +PyObject* create_ring(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *n_object, *bits_object; + const char* q_hex; + if (!PyArg_ParseTuple(args, "OsO", &n_object, &q_hex, &bits_object)) return nullptr; + auto n = PyLong_AsUnsignedLongLong(n_object), bits = PyLong_AsUnsignedLongLong(bits_object); + if (PyErr_Occurred()) return nullptr; + if (n > 32768 || bits > 512) throw std::invalid_argument("Invalid native BFV ring parameters"); + mpz_class q; + if (q.set_str(q_hex, 16) != 0) throw std::invalid_argument("Invalid ciphertext modulus"); + RingPtr ring; + { WithoutGIL release; ring = std::make_shared(n, q, bits); } + return capsule(std::move(ring), ring_name); + }); +} + +Word read_t(PyObject* value, const Ring& ring) { + Word t = PyLong_AsUnsignedLongLong(value); + if (PyErr_Occurred()) throw PythonError{}; + if (t < 2 || t >= (Word(1) << 60) || mpz_cmp_ui(ring.q.get_mpz_t(), t) <= 0) + throw std::invalid_argument("Invalid plaintext modulus"); + return t; +} + +Word read_exponent(PyObject* value, const Ring& ring) { + Word exponent = PyLong_AsUnsignedLongLong(value); + if (PyErr_Occurred()) throw PythonError{}; + if (!(exponent & 1) || exponent >= 2 * ring.n) + throw std::invalid_argument("Invalid Galois exponent"); + return exponent; +} + +PyObject* rotate_rows(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *key_object, *ciphertext_object, *exponent_object; + if (!PyArg_ParseTuple(args, "OOO", &key_object, &ciphertext_object, &exponent_object)) return nullptr; + auto key = get_capsule(key_object, key_name); + auto ciphertext = read_pair(ciphertext_object, *key->ring); + auto exponent = read_exponent(exponent_object, *key->ring); + Ciphertext result; + { WithoutGIL release; result = exponent == 1 ? ciphertext : rotate(ciphertext, exponent, *key); } + return write_components(result, *key->ring); + }); +} + +PyObject* hamming_tile(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *key_object, *query_object, *tile_object, *rotations_object, *mask_object, *t_object; + if (!PyArg_ParseTuple(args, "OOOOOO", &key_object, &query_object, &tile_object, + &rotations_object, &mask_object, &t_object)) return nullptr; + auto relin = get_capsule(key_object, key_name); + const auto& ring = *relin->ring; + Word t = read_t(t_object, ring); + auto query = read_pair(query_object, ring), tile = read_pair(tile_object, ring); + auto mask = read_poly(mask_object, ring); + if (!PyTuple_Check(rotations_object)) throw std::invalid_argument("Rotations must be a tuple"); + std::vector> rotations; + for (Py_ssize_t i = 0; i < PyTuple_GET_SIZE(rotations_object); ++i) { + auto pair = PyTuple_GET_ITEM(rotations_object, i); + if (!PyTuple_Check(pair) || PyTuple_GET_SIZE(pair) != 2) + throw std::invalid_argument("Invalid rotation key entry"); + Word exponent = read_exponent(PyTuple_GET_ITEM(pair, 0), ring); + auto key = get_capsule(PyTuple_GET_ITEM(pair, 1), key_name); + if (key->ring != relin->ring) throw std::invalid_argument("Rotation key belongs to another native ring"); + rotations.emplace_back(exponent, std::move(key)); + } + Ciphertext result; + { WithoutGIL release; result = distance_tile(std::move(query), tile, *relin, rotations, mask, t); } + return write_components(result, ring); + }); +} + +PyObject* compile_key(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *ring_object, *key_object; + if (!PyArg_ParseTuple(args, "OO", &ring_object, &key_object)) return nullptr; + auto ring = get_capsule(ring_object, ring_name); + if (!PyTuple_Check(key_object) || PyTuple_GET_SIZE(key_object) != static_cast(ring->digits)) + throw std::invalid_argument("Incorrect gadget digit count"); + std::vector> key; + for (std::size_t i = 0; i < ring->digits; ++i) + key.push_back(read_pair(PyTuple_GET_ITEM(key_object, i), *ring)); + KeyPtr prepared; + { WithoutGIL release; prepared = std::make_shared(ring, key); } + return capsule(std::move(prepared), key_name); + }); +} + +PyObject* apply_key(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *key_object, *poly_object; + if (!PyArg_ParseTuple(args, "OO", &key_object, &poly_object)) return nullptr; + auto key = get_capsule(key_object, key_name); + auto poly = read_poly(poly_object, *key->ring); + std::array result; + { WithoutGIL release; result = key->apply(poly); } + return write_components(result, *key->ring); + }); +} + +PyObject* ring_product(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *ring_object, *lhs_object, *rhs_object; + if (!PyArg_ParseTuple(args, "OOO", &ring_object, &lhs_object, &rhs_object)) return nullptr; + auto ring = get_capsule(ring_object, ring_name); + auto lhs = read_poly(lhs_object, *ring), rhs = read_poly(rhs_object, *ring); + Polynomial result; + { + WithoutGIL release; + result = ring->product(lhs, rhs); + for (auto& value : result) mpz_mod(value.get_mpz_t(), value.get_mpz_t(), ring->q.get_mpz_t()); + } + return write_poly(result, *ring); + }); +} + +PyObject* multiply(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *ring_object, *lhs_object, *rhs_object, *t_object; + if (!PyArg_ParseTuple(args, "OOOO", &ring_object, &lhs_object, &rhs_object, &t_object)) return nullptr; + auto ring = get_capsule(ring_object, ring_name); + Word t = read_t(t_object, *ring); + auto lhs = read_pair(lhs_object, *ring); + bool square = rhs_object == Py_None; + std::array rhs; + if (!square) rhs = read_pair(rhs_object, *ring); + std::array result; + { WithoutGIL release; result = ring->multiply(lhs, rhs, t, square); } + return write_components(result, *ring); + }); +} + +PyObject* ring_info(PyObject*, PyObject* argument) { + return checked([&]() -> PyObject* { + auto ring = get_capsule(argument, ring_name); + PyObject* primes = PyTuple_New(ring->transforms.size()); + if (!primes) return nullptr; + for (std::size_t i = 0; i < ring->transforms.size(); ++i) { + PyObject* value = PyLong_FromUnsignedLongLong(ring->transforms[i].modulus); + if (!value) { Py_DECREF(primes); return nullptr; } + PyTuple_SET_ITEM(primes, i, value); + } + return Py_BuildValue("{s:N,s:n,s:n,s:s,s:s}", "primes", primes, + "switch_prime_count", ring->switch_prime_count, + "transform_table_bytes", ring->bytes(), "gmp", gmp_version, + "compiler", __VERSION__); + }); +} + +PyObject* key_bytes(PyObject*, PyObject* argument) { + return checked([&]() -> PyObject* { + return PyLong_FromSize_t(get_capsule(argument, key_name)->bytes()); + }); +} + +PyMethodDef methods[] = { + {"create_ring", create_ring, METH_VARARGS, "Prepare exact CRT bases and negacyclic NTT tables."}, + {"compile_key", compile_key, METH_VARARGS, "Cache a gadget key in RNS/NTT form."}, + {"apply_key", apply_key, METH_VARARGS, "Evaluate an exact gadget dot product, reduced mod q."}, + {"ring_product", ring_product, METH_VARARGS, "Multiply two canonical polynomials mod q."}, + {"multiply", multiply, METH_VARARGS, "BFV tensor product and exact signed scale-and-round."}, + {"rotate_rows", rotate_rows, METH_VARARGS, "Galois automorphism and public-key switching."}, + {"hamming_tile", hamming_tile, METH_VARARGS, "Evaluate the native Hamming tile circuit."}, + {"ring_info", ring_info, METH_O, "Report public auxiliary-prime and table information."}, + {"key_bytes", key_bytes, METH_O, "Size of cached NTT coefficient arrays."}, + {nullptr, nullptr, 0, nullptr} +}; +PyModuleDef module = {PyModuleDef_HEAD_INIT, "_bfv_rns", "Native BFV RNS/NTT CPU kernels.", -1, + methods, nullptr, nullptr, nullptr, nullptr}; +} // namespace + +PyMODINIT_FUNC PyInit__bfv_rns() { + PyObject* result = PyModule_Create(&module); + if (result && PyModule_AddIntConstant(result, "ABI_VERSION", 1) < 0) { + Py_DECREF(result); return nullptr; + } + return result; +} diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h new file mode 100644 index 0000000..70731f8 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h @@ -0,0 +1,397 @@ +// Native XTrace BFV arithmetic. Independently implemented; no SEAL code is linked. +// Auxiliary RNS primes accelerate exact integer products without changing BFV's q. +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +namespace xtrace_bfv { +static_assert(sizeof(unsigned long) == 8 && sizeof(std::size_t) == 8, + "BFV RNS kernels require a 64-bit platform with 64-bit unsigned long"); +using Word = std::uint64_t; +using Wide = unsigned __int128; +using Polynomial = std::vector; +using Residues = std::vector>; +using Ciphertext = std::array; + +inline Word power_mod(Word value, Word exponent, Word modulus) { + Word result = 1; + while (exponent) { + if (exponent & 1) result = Wide(result) * value % modulus; + value = Wide(value) * value % modulus; + exponent >>= 1; + } + return result; +} + +struct Multiplier { + Word value, quotient; + Multiplier(Word value, Word modulus) + : value(value), quotient((Wide(value) << 64) / modulus) {} + + // Shoup reduction for a fixed multiplier: the quotient estimate is at + // most one below floor(input*value/p). The remainder is in [0,2p), even + // though both low-word products wrap modulo 2^64. Here p < 2^60. + Word multiply(Word input, Word modulus) const { + Word estimate = (Wide(input) * quotient) >> 64; + Word remainder = input * value - estimate * modulus; + return remainder >= modulus ? remainder - modulus : remainder; + } +}; + +class PrimeNTT { + std::size_t n_; + std::vector roots_, inverse_roots_, twists_, inverse_scales_; + +public: + const Word modulus; + PrimeNTT(std::size_t n, Word prime) : n_(n), modulus(prime) { + Word psi = 0; + for (Word candidate = 2; !psi; ++candidate) { + Word root = power_mod(candidate, (prime - 1) / (2 * n), prime); + if (power_mod(root, n, prime) == prime - 1) psi = root; + } + Word omega = Wide(psi) * psi % prime; + Word inverse_omega = power_mod(omega, prime - 2, prime); + Word inverse_psi = power_mod(psi, prime - 2, prime); + Word root = 1, inverse_root = 1, twist = 1; + Word scale = power_mod(n, prime - 2, prime); + roots_.reserve(n / 2); + inverse_roots_.reserve(n / 2); + twists_.reserve(n); + inverse_scales_.reserve(n); + for (std::size_t i = 0; i < n; ++i) { + if (i < n / 2) { + roots_.emplace_back(root, prime); + inverse_roots_.emplace_back(inverse_root, prime); + root = Wide(root) * omega % prime; + inverse_root = Wide(inverse_root) * inverse_omega % prime; + } + twists_.emplace_back(twist, prime); + inverse_scales_.emplace_back(scale, prime); + twist = Wide(twist) * psi % prime; + scale = Wide(scale) * inverse_psi % prime; + } + } + + // Negacyclic twist followed by a decimation-in-frequency transform. Output + // is bit-reversed; keys and query digits use the same order, so no explicit + // bit-reversal permutation is needed between forward and inverse NTTs. + void forward(std::vector& values) const { + for (std::size_t i = 0; i < n_; ++i) + values[i] = twists_[i].multiply(values[i], modulus); + for (std::size_t length = n_; length > 1; length >>= 1) { + std::size_t half = length / 2, stride = n_ / length; + for (std::size_t start = 0; start < n_; start += length) { + for (std::size_t j = 0; j < half; ++j) { + Word a = values[start + j], b = values[start + j + half]; + Word sum = a + b; + values[start + j] = sum >= modulus ? sum - modulus : sum; + values[start + j + half] = roots_[j * stride].multiply(a + modulus - b, modulus); + } + } + } + } + + // Complementary decimation-in-time inverse, fusing 1/N and the inverse + // negacyclic twist in the final coefficient pass. + void inverse(std::vector& values) const { + for (std::size_t length = 2; length <= n_; length <<= 1) { + std::size_t half = length / 2, stride = n_ / length; + for (std::size_t start = 0; start < n_; start += length) { + for (std::size_t j = 0; j < half; ++j) { + Word a = values[start + j]; + Word b = inverse_roots_[j * stride].multiply(values[start + j + half], modulus); + Word sum = a + b, difference = a + modulus - b; + values[start + j] = sum >= modulus ? sum - modulus : sum; + values[start + j + half] = difference >= modulus ? difference - modulus : difference; + } + } + } + for (std::size_t i = 0; i < n_; ++i) + values[i] = inverse_scales_[i].multiply(values[i], modulus); + } + + std::size_t bytes() const { return 3 * n_ * sizeof(Multiplier); } +}; + +struct CRTBase { + mpz_class product, half; + std::vector weights; +}; + +class Ring { + std::vector bases_; + +public: + const std::size_t n, coefficient_bytes, digits; + const unsigned digit_bits; + const mpz_class q; + std::vector transforms; + std::size_t switch_prime_count; + + Ring(std::size_t n, const mpz_class& q, unsigned bits) + : n(n), coefficient_bytes((mpz_sizeinbase(q.get_mpz_t(), 2) + 7) / 8), + digits(bits ? (mpz_sizeinbase(q.get_mpz_t(), 2) + bits - 1) / bits : 0), + digit_bits(bits), q(q) { + if (n < 8 || n > 32768 || (n & (n - 1)) || q < 3 || + mpz_sizeinbase(q.get_mpz_t(), 2) > 512 || bits < 1 || bits > mpz_sizeinbase(q.get_mpz_t(), 2)) + throw std::invalid_argument("Invalid native BFV ring parameters"); + // A coefficient of a gadget dot product has |c| <= N*L*(2^b-1)*(q-1). + // A BFV cross component is a sum of two products, bounded by 2N(q-1)^2. + // Auxiliary modulus M must exceed twice the bound for exact signed CRT. + mpz_class switch_bound = n * digits * ((mpz_class(1) << bits) - 1) * (q - 1); + mpz_class tensor_bound = 2 * n * (q - 1) * (q - 1); + mpz_class needed = 2 * std::max(switch_bound, tensor_bound); + mpz_class product = 1; + Word prime = (Word(1) << 60) - 2 * n + 1; + while (product <= needed) { + mpz_class candidate(prime); + while (!mpz_probab_prime_p(candidate.get_mpz_t(), 32)) { + if (prime <= (Word(1) << 59)) throw std::invalid_argument("No suitable 60-bit auxiliary prime"); + prime -= 2 * n; + candidate = prime; + } + transforms.emplace_back(n, prime); + product *= prime; + prime -= 2 * n; + } + product = 1; + for (std::size_t count = 1; count <= transforms.size(); ++count) { + product *= transforms[count - 1].modulus; + CRTBase base{product, product / 2, {}}; + for (std::size_t j = 0; j < count; ++j) { + Word p = transforms[j].modulus; + mpz_class partial = product / p; + Word residue = mpz_fdiv_ui(partial.get_mpz_t(), p); + base.weights.push_back(partial * power_mod(residue, p - 2, p)); + } + bases_.push_back(std::move(base)); + } + switch_prime_count = prime_count(switch_bound); + } + + std::size_t prime_count(const mpz_class& bound) const { + for (std::size_t count = 1; count <= bases_.size(); ++count) + if (bases_[count - 1].product > 2 * bound) return count; + throw std::invalid_argument("Insufficient auxiliary CRT modulus for exact reconstruction"); + } + + Residues encode(const Polynomial& poly, std::size_t count) const { + Residues result(count, std::vector(n)); + for (std::size_t j = 0; j < count; ++j) { + for (std::size_t i = 0; i < n; ++i) + result[j][i] = mpz_fdiv_ui(poly[i].get_mpz_t(), transforms[j].modulus); + transforms[j].forward(result[j]); + } + return result; + } + + Polynomial decode(Residues values) const { + std::size_t count = values.size(); + const auto& base = bases_.at(count - 1); + for (std::size_t j = 0; j < count; ++j) transforms[j].inverse(values[j]); + Polynomial result(n); + for (std::size_t i = 0; i < n; ++i) { + auto& value = result[i]; + for (std::size_t j = 0; j < count; ++j) + mpz_addmul_ui(value.get_mpz_t(), base.weights[j].get_mpz_t(), values[j][i]); + value %= base.product; + if (value > base.half) value -= base.product; + } + return result; + } + + Polynomial product(const Polynomial& lhs, const Polynomial& rhs) const { + mpz_class bound = n * *std::max_element(lhs.begin(), lhs.end()) * *std::max_element(rhs.begin(), rhs.end()); + std::size_t count = prime_count(bound); + auto a = encode(lhs, count), b = encode(rhs, count); + for (std::size_t j = 0; j < count; ++j) + for (std::size_t i = 0; i < n; ++i) + a[j][i] = Wide(a[j][i]) * b[j][i] % transforms[j].modulus; + return decode(std::move(a)); + } + + std::array multiply( + const std::array& lhs, const std::array& rhs, + Word t, bool square) const { + std::size_t count = prime_count(2 * n * (q - 1) * (q - 1)); + auto a0 = encode(lhs[0], count), a1 = encode(lhs[1], count); + Residues b0, b1; + if (!square) { b0 = encode(rhs[0], count); b1 = encode(rhs[1], count); } + std::array output; + for (auto& component : output) component.assign(count, std::vector(n)); + for (std::size_t j = 0; j < count; ++j) { + Word p = transforms[j].modulus; + for (std::size_t i = 0; i < n; ++i) { + Word x0 = a0[j][i], x1 = a1[j][i]; + Word y0 = square ? x0 : b0[j][i], y1 = square ? x1 : b1[j][i]; + output[0][j][i] = Wide(x0) * y0 % p; + output[1][j][i] = (Wide(x0) * y1 + Wide(x1) * y0) % p; + output[2][j][i] = Wide(x1) * y1 % p; + } + } + std::array result; + mpz_class denominator = 2 * q; + for (std::size_t k = 0; k < 3; ++k) { + result[k] = decode(std::move(output[k])); + for (auto& value : result[k]) { + // Preserve the reference's exact signed scale-and-round BEFORE + // reducing mod q: floor((2*t*integer_product + q) / (2*q)). + value = 2 * t * value + q; + mpz_fdiv_q(value.get_mpz_t(), value.get_mpz_t(), denominator.get_mpz_t()); + mpz_mod(value.get_mpz_t(), value.get_mpz_t(), q.get_mpz_t()); + } + } + return result; + } + + std::size_t bytes() const { + std::size_t result = 0; + for (const auto& plan : transforms) result += plan.bytes(); + return result; + } +}; + +class SwitchKey { + // digit -> component -> auxiliary prime -> bit-reversed NTT coefficient. + std::vector> key_; + +public: + const std::shared_ptr ring; + SwitchKey(std::shared_ptr ring, const std::vector>& key) + : ring(std::move(ring)) { + if (key.size() != this->ring->digits) throw std::invalid_argument("Incorrect gadget digit count"); + for (const auto& pair : key) + key_.push_back({this->ring->encode(pair[0], this->ring->switch_prime_count), + this->ring->encode(pair[1], this->ring->switch_prime_count)}); + } + + std::array apply(const Polynomial& poly) const { + const auto& r = *ring; + std::array output; + for (auto& component : output) component.assign(r.switch_prime_count, std::vector(r.n)); + // Decompose once, then reuse each integer digit across auxiliary primes. + std::vector decomposition; + std::vector> small_digits; +#if GMP_NUMB_BITS == 64 + if (r.digit_bits <= 60) { + small_digits.assign(r.digits, std::vector(r.n)); + Word mask = (Word(1) << r.digit_bits) - 1; + for (std::size_t digit = 0; digit < r.digits; ++digit) { + std::size_t bit = digit * r.digit_bits, limb = bit / 64, offset = bit % 64; + for (std::size_t i = 0; i < r.n; ++i) { + Word value = mpz_getlimbn(poly[i].get_mpz_t(), limb) >> offset; + if (offset) value |= mpz_getlimbn(poly[i].get_mpz_t(), limb + 1) << (64 - offset); + small_digits[digit][i] = value & mask; + } + } + } else +#endif + { + decomposition.assign(r.digits, Polynomial(r.n)); + for (std::size_t digit = 0; digit < r.digits; ++digit) + for (std::size_t i = 0; i < r.n; ++i) { + auto& value = decomposition[digit][i]; + mpz_fdiv_q_2exp(value.get_mpz_t(), poly[i].get_mpz_t(), digit * r.digit_bits); + mpz_fdiv_r_2exp(value.get_mpz_t(), value.get_mpz_t(), r.digit_bits); + } + } + for (std::size_t j = 0; j < r.switch_prime_count; ++j) { + Word p = r.transforms[j].modulus; + std::array, 2> accum{std::vector(r.n), std::vector(r.n)}; + std::vector digit_values(r.n); + for (std::size_t digit = 0; digit < r.digits; ++digit) { + for (std::size_t i = 0; i < r.n; ++i) { + if (small_digits.empty()) + digit_values[i] = mpz_fdiv_ui(decomposition[digit][i].get_mpz_t(), p); + else { + // Both the digit and p are at most 60 bits; p is close + // to 2^60, so one conditional subtraction suffices. + Word value = small_digits[digit][i]; + digit_values[i] = value >= p ? value - p : value; + } + } + r.transforms[j].forward(digit_values); + for (std::size_t k = 0; k < 2; ++k) + for (std::size_t i = 0; i < r.n; ++i) + accum[k][i] += Wide(digit_values[i]) * key_[digit][k][j][i]; + // Products are < 2^120; a batch of 256 fits in 128 bits. + // Longer gadgets (e.g. 512 one-bit digits) need intermediate reduction. + if ((digit + 1) % 256 == 0) + for (auto& component : accum) + for (auto& value : component) value %= p; + } + for (std::size_t k = 0; k < 2; ++k) + for (std::size_t i = 0; i < r.n; ++i) output[k][j][i] = accum[k][i] % p; + } + std::array result; + for (std::size_t k = 0; k < 2; ++k) { + result[k] = r.decode(std::move(output[k])); + for (auto& value : result[k]) mpz_mod(value.get_mpz_t(), value.get_mpz_t(), r.q.get_mpz_t()); + } + return result; + } + + std::size_t bytes() const { return ring->digits * 2 * ring->switch_prime_count * ring->n * sizeof(Word); } +}; + +inline void add_inplace(Ciphertext& lhs, const Ciphertext& rhs, const Ring& ring) { + for (std::size_t k = 0; k < 2; ++k) + for (std::size_t i = 0; i < ring.n; ++i) { + lhs[k][i] += rhs[k][i]; + if (lhs[k][i] >= ring.q) lhs[k][i] -= ring.q; + } +} + +inline Ciphertext rotate(const Ciphertext& ciphertext, Word exponent, const SwitchKey& key) { + const auto& r = *key.ring; + Ciphertext permuted{Polynomial(r.n), Polynomial(r.n)}; + for (std::size_t k = 0; k < 2; ++k) + for (std::size_t i = 0; i < r.n; ++i) { + std::size_t index = (i * exponent) % (2 * r.n); + auto& value = permuted[k][index % r.n]; + value = ciphertext[k][i]; + if (index >= r.n && value != 0) value = r.q - value; + } + auto switched = key.apply(permuted[1]); + for (std::size_t i = 0; i < r.n; ++i) { + switched[0][i] += permuted[0][i]; + if (switched[0][i] >= r.q) switched[0][i] -= r.q; + } + return switched; +} + +// Keep a complete Hamming tile inside the native module. This performs exactly +// the Python client's subtract/square/relinearize/rotate/add/mask circuit, while +// avoiding conversion back to Python mpz tuples after every intermediate step. +inline Ciphertext distance_tile( + Ciphertext query, const Ciphertext& tile, const SwitchKey& relin, + const std::vector>>& rotations, + const Polynomial& mask, Word t) { + const auto& r = *relin.ring; + for (std::size_t k = 0; k < 2; ++k) + for (std::size_t i = 0; i < r.n; ++i) { + query[k][i] -= tile[k][i]; + if (query[k][i] < 0) query[k][i] += r.q; + } + auto quadratic = r.multiply(query, query, t, true); + auto distance = relin.apply(quadratic[2]); + Ciphertext linear{std::move(quadratic[0]), std::move(quadratic[1])}; + add_inplace(distance, linear, r); + for (const auto& entry : rotations) { + auto shifted = entry.first == 1 ? distance : rotate(distance, entry.first, *entry.second); + add_inplace(distance, shifted, r); + } + for (auto& component : distance) { + component = r.product(component, mask); + for (auto& value : component) mpz_mod(value.get_mpz_t(), value.get_mpz_t(), r.q.get_mpz_t()); + } + return distance; +} +} // namespace xtrace_bfv diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py index 6e0841e..86824e6 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py @@ -6,12 +6,14 @@ """ from dataclasses import dataclass -from typing import Literal +import sys +from typing import Any, Literal import gmpy2 from gmpy2 import mpz from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _automorphism +from xtrace_sdk.x_vec.crypto.encryption.bfv_rns import BFVRNSArithmetic from xtrace_sdk.x_vec.utils.xtrace_types import ( BFVCiphertext, BFVPolynomial, @@ -19,7 +21,7 @@ BFVSwitchKey, ) -BFVServerBackend = Literal["optimized", "reference"] +BFVServerBackend = Literal["optimized", "reference", "rns"] @dataclass(frozen=True) @@ -73,11 +75,12 @@ class BFVEvaluator: """Reusable native BFV server arithmetic, bound to one public key. ``optimized`` lazily caches packed evaluation keys and fuses gadget products; - ``reference`` delegates to the original BFV routines. Both return identical + ``reference`` delegates to the original BFV routines. ``rns`` uses the + optional native C++ RNS/NTT kernels. All backends return identical ciphertexts. Reuse one evaluator across queries to amortize preparation. :param pk: Valid public key from key generation or public-key deserialization. - :param backend: ``optimized`` (default) or ``reference``. + :param backend: ``optimized`` (default), ``reference``, or ``rns``. The key dictionaries are snapshotted; immutable polynomial tuples are shared. Construct a new evaluator when changing keys. The cache is bounded by the @@ -85,12 +88,32 @@ class BFVEvaluator: """ def __init__(self, pk: BFVPublicKey, backend: BFVServerBackend = "optimized") -> None: - if backend not in ("optimized", "reference"): - raise ValueError("server_backend must be 'optimized' or 'reference'") + if backend not in ("optimized", "reference", "rns"): + raise ValueError("server_backend must be 'optimized', 'reference', or 'rns'") self.backend = backend self._pk: BFVPublicKey = {**pk, "galois_keys": dict(pk["galois_keys"])} # Exponent zero identifies relinearization; Galois exponents are odd. self._switch_keys: dict[int, _PackedSwitchKey] = {} + self._rns = BFVRNSArithmetic(self._pk) if backend == "rns" else None + + def cache_info(self) -> dict[str, Any]: + """Report prepared public-key payloads; excludes peak temporary allocations.""" + if self._rns is not None: + return self._rns.cache_info() + return { + "switch_keys": len(self._switch_keys), + "key_payload_bytes": sum( + sys.getsizeof(value) + for key in self._switch_keys.values() + for value in ( + key.low_mask, + key.bias_coefficient, + key.bias, + *(v for pair in key.pairs for v in pair), + ) + ), + "transform_table_bytes": 0, + } def _validate(self, ciphertext: BFVCiphertext) -> None: pk = self._pk @@ -122,6 +145,9 @@ def _pair(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> None: def _switch(self, poly: BFVPolynomial, exponent: int) -> tuple[BFVPolynomial, BFVPolynomial]: pk, params = self._pk, self._pk["params"] + if self._rns is not None: + key = pk["relin_key"] if exponent == 0 else pk["galois_keys"][exponent] + return self._rns.switch(poly, exponent, key) if exponent not in self._switch_keys: key = pk["relin_key"] if exponent == 0 else pk["galois_keys"][exponent] self._switch_keys[exponent] = _PackedSwitchKey.compile( @@ -159,6 +185,14 @@ def subtract(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> BFVCiphertext: def multiply_plain(self, ciphertext: BFVCiphertext, plaintext: BFVPolynomial) -> BFVCiphertext: """Apply an unscaled plaintext polynomial, e.g. a distance slot mask.""" + if self._rns is not None and ciphertext.modulus == self._pk["q"]: + self._validate(ciphertext) + plaintext = BFV._plaintext(plaintext, self._pk["params"]) + return BFVCiphertext( + tuple(self._rns.product(c, plaintext) for c in ciphertext.components), + ciphertext.modulus, + ciphertext.key_id, + ) return BFV.multiply_plain(ciphertext, plaintext, self._pk) def relinearize(self, ciphertext: BFVCiphertext) -> BFVCiphertext: @@ -185,7 +219,19 @@ def relinearize(self, ciphertext: BFVCiphertext) -> BFVCiphertext: ) def multiply(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> BFVCiphertext: - """Keep the reference exact tensor product and scale; accelerate relinearization.""" + """Exact integer tensor product, scale-and-round, then relinearization.""" + if self._rns is not None: + self._pair(lhs, rhs) + if lhs.modulus != self._pk["q"] or len(lhs.components) != 2: + raise ValueError("Multiplication requires two components at the original modulus") + result = BFVCiphertext( + self._rns.multiply( + lhs.components, rhs.components, self._pk["params"].plain_modulus + ), + lhs.modulus, + lhs.key_id, + ) + return self.relinearize(result) return self.relinearize(BFV.multiply(lhs, rhs, self._pk, relinearize=False)) def xor(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> BFVCiphertext: @@ -193,6 +239,33 @@ def xor(self, lhs: BFVCiphertext, rhs: BFVCiphertext) -> BFVCiphertext: difference = self.subtract(lhs, rhs) return self.multiply(difference, difference) + def hamming_tile( + self, query: BFVCiphertext, tile: BFVCiphertext, steps: list[int], mask: BFVPolynomial + ) -> BFVCiphertext: + """Square the difference, sum dimensions by rotations, then apply a slot mask.""" + if self._rns is not None: + self._pair(query, tile) + pk = self._pk + if query.modulus != pk["q"] or len(query.components) != 2: + raise ValueError("Multiplication requires two components at the original modulus") + if not pk["relin_key"]: + raise ValueError("Public key has no relinearization key") + n = pk["params"].poly_modulus_degree + exponents = [pow(3, step % (n // 2), 2 * n) for step in steps] + for step, exponent in zip(steps, exponents, strict=True): + if exponent != 1 and exponent not in pk["galois_keys"]: + raise ValueError(f"Public key has no rotation key for steps={step}") + mask = BFV._plaintext(mask, pk["params"]) + return BFVCiphertext( + self._rns.hamming_tile(query.components, tile.components, exponents, mask), + query.modulus, + query.key_id, + ) + distance = self.xor(query, tile) + for step in steps: + distance = self.add(distance, self.rotate_rows(distance, step)) + return self.multiply_plain(distance, mask) + def rotate_rows(self, ciphertext: BFVCiphertext, steps: int) -> BFVCiphertext: """Rotate both slot rows left (right for negative steps) using public keys.""" if self.backend == "reference": @@ -207,6 +280,12 @@ def rotate_rows(self, ciphertext: BFVCiphertext, steps: int) -> BFVCiphertext: return ciphertext if exponent not in pk["galois_keys"]: raise ValueError(f"Public key has no rotation key for steps={steps}") + if self._rns is not None: + return BFVCiphertext( + self._rns.rotate(ciphertext.components, exponent, pk["galois_keys"][exponent]), + q, + pk["key_id"], + ) c0, c1 = (_automorphism(c, exponent, q) for c in ciphertext.components) k0, k1 = self._switch(c1, exponent) return BFVCiphertext( diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py new file mode 100644 index 0000000..5609445 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py @@ -0,0 +1,111 @@ +"""Optional native RNS/NTT arithmetic with the existing BFV polynomial types. + +Auxiliary primes are temporary arithmetic workspaces. The BFV ciphertext modulus, +keys and wire representation remain the same as the Python/GMP implementation. +""" + +from typing import Any + +import gmpy2 +from gmpy2 import mpz + +from xtrace_sdk.x_vec.utils.xtrace_types import BFVPolynomial, BFVPublicKey, BFVSwitchKey + + +class BFVRNSArithmetic: + """Bind optional C++ kernels to one validated BFV public-key context.""" + + def __init__(self, pk: BFVPublicKey) -> None: + try: + from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns + except ImportError as error: + raise ImportError( + "BFV RNS CPU extension is unavailable. Build it with " + "make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON=/path/to/python; " + "or select server_backend='optimized' for the Python/GMP evaluator." + ) from error + if _bfv_rns.ABI_VERSION != 1: + raise ImportError("BFV RNS CPU extension has an incompatible ABI; rebuild it") + self._native = _bfv_rns + self._pk = pk + params = pk["params"] + self.n = params.poly_modulus_degree + self.width = (pk["q"].bit_length() + 7) // 8 + self._ring = _bfv_rns.create_ring(self.n, format(pk["q"], "x"), params.decomposition_bits) + self._keys: dict[int, object] = {} + + def _pack(self, poly: BFVPolynomial) -> bytes: + return gmpy2.pack(list(poly), 8 * self.width).to_bytes(self.n * self.width, "little") + + def _unpack(self, data: bytes) -> BFVPolynomial: + values = gmpy2.unpack(mpz.from_bytes(data, "little"), 8 * self.width) + values.extend([mpz(0)] * (self.n - len(values))) + return tuple(values) + + def switch( + self, poly: BFVPolynomial, exponent: int, key: BFVSwitchKey + ) -> tuple[BFVPolynomial, BFVPolynomial]: + """Prepare each public gadget key once, then reuse its NTT representation.""" + b, a = self._native.apply_key(self._prepare_key(exponent, key), self._pack(poly)) + return self._unpack(b), self._unpack(a) + + def _prepare_key(self, exponent: int, key: BFVSwitchKey) -> object: + if exponent not in self._keys: + self._keys[exponent] = self._native.compile_key( + self._ring, tuple((self._pack(b), self._pack(a)) for b, a in key) + ) + return self._keys[exponent] + + def rotate( + self, components: tuple[BFVPolynomial, ...], exponent: int, key: BFVSwitchKey + ) -> tuple[BFVPolynomial, ...]: + """Keep the automorphism and key-switch intermediates in the native module.""" + output = self._native.rotate_rows( + self._prepare_key(exponent, key), tuple(self._pack(c) for c in components), exponent + ) + return tuple(self._unpack(c) for c in output) + + def hamming_tile( + self, + query: tuple[BFVPolynomial, ...], + tile: tuple[BFVPolynomial, ...], + exponents: list[int], + mask: BFVPolynomial, + ) -> tuple[BFVPolynomial, ...]: + """Run the existing tile circuit without materializing Python intermediates.""" + relin = self._prepare_key(0, self._pk["relin_key"]) + rotations = tuple( + (g, relin if g == 1 else self._prepare_key(g, self._pk["galois_keys"][g])) + for g in exponents + ) + output = self._native.hamming_tile( + relin, + tuple(self._pack(c) for c in query), + tuple(self._pack(c) for c in tile), + rotations, + self._pack(mask), + self._pk["params"].plain_modulus, + ) + return tuple(self._unpack(c) for c in output) + + def product(self, lhs: BFVPolynomial, rhs: BFVPolynomial) -> BFVPolynomial: + """Exact negacyclic product, reduced modulo the original ciphertext modulus.""" + return self._unpack(self._native.ring_product(self._ring, self._pack(lhs), self._pack(rhs))) + + def multiply( + self, lhs: tuple[BFVPolynomial, ...], rhs: tuple[BFVPolynomial, ...], t: int + ) -> tuple[BFVPolynomial, ...]: + """Tensor products over the integers, then exact BFV scale-and-round.""" + output = self._native.multiply( + self._ring, + tuple(self._pack(poly) for poly in lhs), + None if lhs is rhs else tuple(self._pack(poly) for poly in rhs), + t, + ) + return tuple(self._unpack(poly) for poly in output) + + def cache_info(self) -> dict[str, Any]: + info = dict(self._native.ring_info(self._ring)) + info["switch_keys"] = len(self._keys) + info["key_payload_bytes"] = sum(self._native.key_bytes(key) for key in self._keys.values()) + return info diff --git a/tests/x_vec/native/test_bfv_rns.cpp b/tests/x_vec/native/test_bfv_rns.cpp new file mode 100644 index 0000000..40afc8d --- /dev/null +++ b/tests/x_vec/native/test_bfv_rns.cpp @@ -0,0 +1,85 @@ +// Standalone kernel oracles; also run under address/undefined-behavior sanitizers. +#include "rns_ntt.h" +#include +#include +#include +using namespace xtrace_bfv; + +Polynomial schoolbook(const Polynomial& a, const Polynomial& b) { + Polynomial result(a.size()); + for (std::size_t i = 0; i < a.size(); ++i) + for (std::size_t j = 0; j < b.size(); ++j) { + if (i + j < a.size()) result[i + j] += a[i] * b[j]; + else result[i + j - a.size()] -= a[i] * b[j]; + } + return result; +} + +int main() { + std::mt19937_64 random(1337); + gmp_randclass big_random(gmp_randinit_default); + big_random.seed(1337); + for (unsigned q_bits : {31, 100, 180, 512}) { + for (unsigned bits : {1, 15, 30, 59, 60, 61, 127}) { + if (bits > q_bits) continue; + mpz_class q = (mpz_class(1) << q_bits) - 1; + auto ring = std::make_shared(8, q, bits); + Polynomial a(8), b(8); + for (auto& value : a) value = big_random.get_z_range(q); + for (auto& value : b) value = big_random.get_z_range(q); + for (const auto& transform : ring->transforms) { + Word p = transform.modulus; + for (int i = 0; i < 300; ++i) { + Word multiplier = i ? random() % p : p - 1; + Word input = i ? random() % (2 * p) : 2 * p - 1; + assert(Multiplier(multiplier, p).multiply(input, p) == Wide(input) * multiplier % p); + } + std::vector values(8); + for (auto& value : values) value = random() % p; + auto original = values; + transform.forward(values); + transform.inverse(values); + assert(values == original); + } + for (int sample = 0; sample < 3; ++sample) { + if (sample == 1) { a.assign(8, q - 1); b.assign(8, q - 1); } + if (sample == 2) a.assign(8, 0); + assert(ring->product(a, b) == schoolbook(a, b)); + } + std::vector> key(ring->digits); + for (auto& pair : key) { + pair[0].assign(8, q - 1); + pair[1] = b; + } + a.assign(8, q - 1); + SwitchKey prepared(ring, key); + Ciphertext expected{Polynomial(8), Polynomial(8)}; + for (std::size_t digit = 0; digit < ring->digits; ++digit) { + Polynomial part(8); + for (std::size_t i = 0; i < 8; ++i) { + part[i] = a[i] >> (digit * bits); + mpz_fdiv_r_2exp(part[i].get_mpz_t(), part[i].get_mpz_t(), bits); + } + for (std::size_t k = 0; k < 2; ++k) { + auto product = schoolbook(part, key[digit][k]); + for (std::size_t i = 0; i < 8; ++i) expected[k][i] += product[i]; + } + } + for (auto& component : expected) + for (auto& value : component) mpz_mod(value.get_mpz_t(), value.get_mpz_t(), q.get_mpz_t()); + assert(prepared.apply(a) == expected); + } + } + // Exercise NTT lengths, both signs of X^N=-1, and large transform tables. + for (std::size_t n : {16, 64, 8192, 32768}) { + mpz_class q = (mpz_class(1) << 180) - 1; + Ring ring(n, q, 30); + Polynomial a(n), b(n); + a[n - 1] = q - 1; + b[1] = q - 1; + auto product = ring.product(a, b); + assert(product[0] == -(q - 1) * (q - 1)); + for (std::size_t i = 1; i < n; ++i) assert(product[i] == 0); + } + std::cout << "Native NTT, Shoup, signed CRT and gadget oracles passed\n"; +} diff --git a/tests/x_vec/test_bfv_client.py b/tests/x_vec/test_bfv_client.py index ca68f5c..c0d0bdb 100644 --- a/tests/x_vec/test_bfv_client.py +++ b/tests/x_vec/test_bfv_client.py @@ -66,9 +66,12 @@ def test_base_interface_and_individual_batch() -> None: @pytest.mark.parametrize("dimension", [1, 3, 9, 16]) -def test_packed_tile_row_and_response_boundaries(dimension: int) -> None: +@pytest.mark.parametrize("backend", ["optimized", "rns"]) +def test_packed_tile_row_and_response_boundaries(dimension: int, backend: str) -> None: + if backend == "rns": + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") client = small_client(dimension) - server = public_server(client) + server = public_server(client, backend) c = client.vectors_per_ciphertext reference = public_server(client, "reference") for count in sorted({0, 1, c - 1, c, c + 1, 31, 32, 33, 67}): @@ -110,7 +113,10 @@ def test_partial_tile_padding_is_zero_and_compaction_preserves_distances() -> No assert sum(v.bit_length() for v in compact) < sum(v.bit_length() for v in response[0]) / 2 -def test_public_only_evaluator_in_separate_process(tmp_path: Path) -> None: +@pytest.mark.parametrize("backend", ["optimized", "rns"]) +def test_public_only_evaluator_in_separate_process(tmp_path: Path, backend: str) -> None: + if backend == "rns": + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") client = small_client(9) vectors, query, expected = data(35, 9) public = json.loads(client.stringify_pk()) @@ -129,7 +135,7 @@ def test_public_only_evaluator_in_separate_process(tmp_path: Path) -> None: from pathlib import Path from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient p = json.loads(Path(sys.argv[1]).read_text()) -s = BFVClient(skip_key_gen=True) +s = BFVClient(skip_key_gen=True, server_backend=sys.argv[2]) s.load_config(p['config']) s.load_stringified_keys(p['pk']) assert s.keys is None @@ -140,7 +146,7 @@ def test_public_only_evaluator_in_separate_process(tmp_path: Path) -> None: print(json.dumps([[format(v, 'x') for v in row] for row in response])) """ run = subprocess.run( - [sys.executable, "-c", script, str(path)], + [sys.executable, "-c", script, str(path), backend], capture_output=True, text=True, check=True, diff --git a/tests/x_vec/test_bfv_evaluator.py b/tests/x_vec/test_bfv_evaluator.py index 30797d9..ef37e3e 100644 --- a/tests/x_vec/test_bfv_evaluator.py +++ b/tests/x_vec/test_bfv_evaluator.py @@ -1,6 +1,7 @@ """Exact coefficient comparisons and boundary oracles for optimized BFV evaluation.""" import random +import builtins from dataclasses import replace import numpy as np @@ -54,9 +55,12 @@ def test_fused_switch_against_schoolbook(q_bits: int, digit_bits: int) -> None: assert actual == tuple(tuple(value % q for value in row) for row in expected) -def test_operations_identical_to_reference(keys: BFVKeyPair) -> None: +@pytest.mark.parametrize("backend", ["optimized", "rns"]) +def test_operations_identical_to_reference(keys: BFVKeyPair, backend: str) -> None: + if backend == "rns": + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") pk, rng = keys["pk"], random.Random(20) - evaluator = BFVEvaluator(pk) + evaluator = BFVEvaluator(pk, backend) for _ in range(3): pa, pb = ( BFV.batch_encode([rng.randrange(97) for _ in range(16)], pk["params"]) for _ in range(2) @@ -78,9 +82,14 @@ def test_operations_identical_to_reference(keys: BFVKeyPair) -> None: assert evaluator.subtract(compact, compact) == BFV.subtract(compact, compact, pk) -def test_reject_malformed_ciphertexts_and_keep_integral_support(keys: BFVKeyPair) -> None: +@pytest.mark.parametrize("backend", ["optimized", "rns"]) +def test_reject_malformed_ciphertexts_and_keep_integral_support( + keys: BFVKeyPair, backend: str +) -> None: + if backend == "rns": + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") pk = keys["pk"] - evaluator = BFVEvaluator(pk) + evaluator = BFVEvaluator(pk, backend) ct = BFV.encrypt(BFV.batch_encode([1], pk["params"]), pk) malformed = [ replace(ct, key_id="wrong"), @@ -102,6 +111,8 @@ def test_reject_malformed_ciphertexts_and_keep_integral_support(keys: BFVKeyPair evaluator.rotate_rows(bad, 1) with pytest.raises(ValueError): evaluator.relinearize(bad) + with pytest.raises(ValueError): + evaluator.hamming_tile(ct, bad, [1], BFV.batch_encode([1], pk["params"])) for value in (0, False, np.int64(0)): candidate = replace(ct, components=((value, *ct.components[0][1:]), ct.components[1])) assert evaluator.add(ct, candidate) == BFV.add(ct, candidate, pk) @@ -143,3 +154,21 @@ def test_missing_keys_and_snapshot_cache(keys: BFVKeyPair) -> None: assert not reference._switch_keys with pytest.raises(ValueError, match="server_backend"): BFVEvaluator(keys["pk"], "invalid") + + +def test_unavailable_native_extension_keeps_gmp_fallback( + keys: BFVKeyPair, monkeypatch: pytest.MonkeyPatch +) -> None: + original_import = builtins.__import__ + + def without_native(name: str, *args, **kwargs): + if name == "xtrace_sdk.x_vec.crypto.bfv_cpu_ext": + raise ImportError("Native module intentionally unavailable") + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", without_native) + with pytest.raises(ImportError, match="Build it with"): + BFVEvaluator(keys["pk"], "rns") + pk = keys["pk"] + ct = BFV.encrypt(BFV.batch_encode([1], pk["params"]), pk) + assert BFVEvaluator(pk).multiply(ct, ct) == BFV.multiply(ct, ct, pk) diff --git a/tests/x_vec/test_bfv_rns.py b/tests/x_vec/test_bfv_rns.py new file mode 100644 index 0000000..1ff800f --- /dev/null +++ b/tests/x_vec/test_bfv_rns.py @@ -0,0 +1,193 @@ +"""Optional native RNS/NTT kernels: independent oracles and boundary validation.""" + +import gc +import json +import math +import random +from concurrent.futures import ThreadPoolExecutor + +import gmpy2 +import pytest +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _poly_product +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVEvaluator +from xtrace_sdk.x_vec.crypto.encryption.bfv_rns import BFVRNSArithmetic +from xtrace_sdk.x_vec.utils.xtrace_types import BFVCiphertext, BFVKeyPair + +native = pytest.importorskip( + "xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns", reason="Build the optional BFV CPU extension" +) + + +def schoolbook(a: tuple, b: tuple) -> tuple: + result = [mpz(0)] * len(a) + for i, x in enumerate(a): + for j, y in enumerate(b): + result[(i + j) % len(a)] += x * y * (1 if i + j < len(a) else -1) + return tuple(result) + + +@pytest.mark.parametrize( + "q_bits,bits", + [ + (31, 1), + (100, 15), + (180, 30), + (180, 59), + (180, 60), + (180, 61), + (512, 127), + (512, 1), + (100, 100), + ], +) +def test_rns_switch_and_products_against_integer_oracle(q_bits: int, bits: int) -> None: + keys = BFV.key_gen(8, 17, q_bits, bits) + pk, rng = keys["pk"], random.Random(bits) + q = pk["q"] + arithmetic = BFVRNSArithmetic(pk) + maximum = (q - 1,) * 8 + random_poly = tuple(mpz(rng.randrange(q)) for _ in range(8)) + inputs = [(mpz(0),) * 8, maximum, (mpz(0),) * 7 + (q - 1,), random_poly] + # Maximum gadget keys stress the sum-of-products CRT bound. This oracle + # uses signed schoolbook convolution, not either implementation's kernels. + digit_count = len(pk["relin_key"]) + key = tuple((maximum, random_poly) for _ in range(digit_count)) + for poly in inputs: + expected = [[mpz(0)] * 8 for _ in range(2)] + for digit, pair in enumerate(key): + part = tuple((c >> (digit * bits)) & ((1 << bits) - 1) for c in poly) + for k in range(2): + for i, value in enumerate(schoolbook(part, pair[k])): + expected[k][i] += value + assert arithmetic.switch(poly, 0, key) == tuple( + tuple(c % q for c in row) for row in expected + ) + for other in (maximum, random_poly): + assert arithmetic.product(poly, other) == tuple(c % q for c in schoolbook(poly, other)) + info = arithmetic.cache_info() + primes = info["primes"] + assert len(primes) == len(set(primes)) + assert all(2**59 < p < 2**60 and p % 16 == 1 and gmpy2.is_prime(p) for p in primes) + modulus = math.prod(primes[: info["switch_prime_count"]]) + assert modulus > 2 * 8 * digit_count * ((1 << bits) - 1) * (q - 1) + + +@pytest.fixture(scope="module") +def keys() -> BFVKeyPair: + return BFV.key_gen(16, 97, 100, 15, rotation_steps=[1, -1, 2, 4]) + + +def test_exact_signed_tensor_scaling_and_multiple_levels(keys: BFVKeyPair) -> None: + pk, rng = keys["pk"], random.Random(20) + arithmetic = BFVRNSArithmetic(pk) + evaluator = BFVEvaluator(pk, "rns") + maximum = (pk["q"] - 1,) * 16 + for components in ( + (maximum, maximum), + ((mpz(0),) * 16, maximum), + tuple(tuple(mpz(rng.randrange(pk["q"])) for _ in range(16)) for _ in range(2)), + ): + ct = BFVCiphertext(components, pk["q"], pk["key_id"]) + assert ( + arithmetic.multiply(components, components, 97) + == BFV.multiply(ct, ct, pk, relinearize=False).components + ) + a, b = list(range(16)), [96 - i for i in range(16)] + ca, cb = (BFV.encrypt(BFV.batch_encode(v, pk["params"]), pk) for v in (a, b)) + product = evaluator.multiply(ca, cb) + assert product == BFV.multiply(ca, cb, pk) + product = evaluator.multiply(product, ca) + assert BFV.batch_decode(BFV.decrypt(product, keys), pk["params"]) == [ + x * y * x % 97 for x, y in zip(a, b, strict=True) + ] + + +def test_native_buffers_capsules_and_parameter_bounds(keys: BFVKeyPair) -> None: + pk = keys["pk"] + arithmetic = BFVRNSArithmetic(pk) + ring, poly = arithmetic._ring, arithmetic._pack(pk["a"]) + for wrong in (poly[:-1], poly + b"\0", arithmetic._pack((pk["q"],) * 16)): + with pytest.raises(ValueError): + native.ring_product(ring, wrong, poly) + with pytest.raises(TypeError): + native.ring_product(ring, [0] * len(poly), poly) + with pytest.raises(ValueError): + native.ring_product(None, poly, poly) + for n, q, bits in ( + (7, "11", 1), + (2**32 + 8, "11", 1), + (8, "-1", 1), + (8, "bad!", 1), + (8, "11", 0), + (8, "11", 513), + (8, "11", 2**32 + 1), + ): + with pytest.raises((ValueError, OverflowError)): + native.create_ring(n, q, bits) + for t in (-1, -(1 << 64) + 17, 1 << 64, 1 << 63, 0, 1): + with pytest.raises((ValueError, OverflowError)): + native.multiply(ring, (poly, poly), None, t) + with pytest.raises(ValueError): + native.multiply(ring, (poly,), None, 97) + with pytest.raises(ValueError): + native.compile_key(ring, ()) + key = arithmetic._prepare_key(0, pk["relin_key"]) + for exponent in (0, 2, 32, -1, 1 << 64): + with pytest.raises((ValueError, OverflowError)): + native.rotate_rows(key, (poly, poly), exponent) + other = BFVRNSArithmetic(pk) + other_key = other._prepare_key(0, pk["relin_key"]) + with pytest.raises(ValueError, match="another native ring"): + native.hamming_tile(key, (poly, poly), (poly, poly), ((3, other_key),), poly, 97) + # The key owns a shared reference to its native plan after Python owners die. + expected = native.apply_key(key, poly) + del arithmetic, ring + gc.collect() + assert native.apply_key(key, poly) == expected + + +def test_public_server_cache_reload_and_concurrent_searches() -> None: + client = BFVClient(9, 32, 193, 120, 20, response_modulus_bits=40) + server = BFVClient(skip_key_gen=True, server_backend="rns") + server.load_config(json.loads(client.stringify_config())) + server.load_stringified_keys(client.stringify_pk()) + assert server.keys is None + query = client.encrypt_vec_one([0, 1, 0] * 3) + vectors = [[0, 1, 0] * 3, [1, 0, 1] * 3, [0] * 9] * 13 + index = client.encrypt_vec_packed(vectors) + with ThreadPoolExecutor(max_workers=2) as pool: + results = list( + pool.map( + lambda _: server.encode_hamming_server_packed(query, index, len(vectors)), range(6) + ) + ) + assert all(result == results[0] for result in results) + assert client.decode_hamming_client_packed(results[0], len(vectors)) == [0, 9, 3] * 13 + info = server._evaluator().cache_info() + assert info["switch_keys"] <= len(client._pk()["galois_keys"]) + 1 + assert info["key_payload_bytes"] > 0 + server.load_stringified_keys(client.stringify_pk()) + assert server._evaluator().cache_info()["switch_keys"] == 0 + assert server.encode_hamming_server_packed(query, index, len(vectors)) == results[0] + + +def test_default_parameters_match_gmp() -> None: + client = BFVClient() + rng = random.Random(1337) + query = [rng.getrandbits(1) for _ in range(512)] + vectors = [[rng.getrandbits(1) for _ in range(512)] for _ in range(33)] + vectors[0] = query[:] + vectors[1] = [1 - bit for bit in query] + encrypted_query, index = client.encrypt_vec_one(query), client.encrypt_vec_packed(vectors) + original = client.encode_hamming_server_packed(encrypted_query, index, 33, compact=False) + client.server_backend = "rns" + assert ( + client.encode_hamming_server_packed(encrypted_query, index, 33, compact=False) == original + ) + response = client.encode_hamming_server_packed(encrypted_query, index, 33) + assert client.decode_hamming_client_packed(response, 33) == [ + sum(x != y for x, y in zip(v, query, strict=True)) for v in vectors + ] From 28170049a93a15b1a06a42f5ae89d74e80f50be2 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Thu, 10 Sep 2026 21:40:48 -0400 Subject: [PATCH 05/12] Add complete native BFV server evaluation and phase profiling --- .github/workflows/ci.yml | 12 +- CHANGELOG.md | 2 + benchmarks/bfv_client_matrix.py | 2 +- benchmarks/bfv_server.py | 34 +- .../results/native_bfv_cpp_server_1024.json | 388 ++++++++++++++++++ .../results/native_bfv_cpp_server_32.json | 388 ++++++++++++++++++ docs/api_rst/xtrace_sdk.crypto.encryption.rst | 8 + docs/index.rst | 1 + docs/research/native-bfv-ntt.md | 4 + docs/research/native-bfv-server.md | 250 +++++++++++ docs/research/native-bfv.md | 5 + src/xtrace_sdk/x_vec/crypto/bfv_client.py | 30 +- .../x_vec/crypto/bfv_cpu_ext/Makefile | 2 +- .../x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi | 16 +- .../x_vec/crypto/bfv_cpu_ext/bfv_server.h | 154 +++++++ .../x_vec/crypto/bfv_cpu_ext/bindings.cpp | 143 ++++++- .../x_vec/crypto/bfv_cpu_ext/packed_wire.h | 61 +++ .../x_vec/crypto/bfv_cpu_ext/profile.h | 62 +++ .../x_vec/crypto/bfv_cpu_ext/rns_ntt.h | 192 ++++++++- .../x_vec/crypto/encryption/bfv_evaluator.py | 17 +- .../x_vec/crypto/encryption/bfv_native.py | 125 ++++++ .../x_vec/crypto/encryption/bfv_rns.py | 8 +- tests/x_vec/native/run_binding_sanitizers.py | 25 ++ tests/x_vec/native/test_bfv_rns.cpp | 41 +- tests/x_vec/test_bfv_client.py | 8 +- tests/x_vec/test_bfv_evaluator.py | 8 +- tests/x_vec/test_bfv_native.py | 166 ++++++++ tests/x_vec/test_bfv_rns.py | 5 +- 28 files changed, 2096 insertions(+), 61 deletions(-) create mode 100644 benchmarks/results/native_bfv_cpp_server_1024.json create mode 100644 benchmarks/results/native_bfv_cpp_server_32.json create mode 100644 docs/research/native-bfv-server.md create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h create mode 100644 src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py create mode 100644 tests/x_vec/native/run_binding_sanitizers.py create mode 100644 tests/x_vec/test_bfv_native.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a54fb49..9d2ebea 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -67,14 +67,22 @@ jobs: sudo apt-get update sudo apt-get install -y libgmp-dev make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" - uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 1' - uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_client.py + uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 2' + uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_client.py - name: BFV native arithmetic sanitizers run: | g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext tests/x_vec/native/test_bfv_rns.cpp -lgmpxx -lgmp -o /tmp/test_bfv_rns /tmp/test_bfv_rns + - name: BFV Python binding sanitizers + run: | + bfv_py_include=$(.venv/bin/python -c 'import sysconfig; print(sysconfig.get_paths()["include"])') + g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -fPIC -shared -I"$bfv_py_include" src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp -lgmpxx -lgmp -o /tmp/bfv_sanitized.so + bfv_asan_runtime=$(g++ -print-file-name=libasan.so) + bfv_cpp_runtime=$(g++ -print-file-name=libstdc++.so) + ASAN_OPTIONS=detect_leaks=0 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 LD_PRELOAD="$bfv_asan_runtime:$bfv_cpp_runtime" .venv/bin/python tests/x_vec/native/run_binding_sanitizers.py /tmp/bfv_sanitized.so + # ── Integration tests (require XTrace API credentials) ────────── # Triggers: # - PRs: only when explicitly labeled `run-integration` (avoids quota diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ec3051..29c0184 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,8 @@ All notable changes to this project will be documented in this file. ### Added +- Complete C++ BFV server evaluation through `server_backend="native"`: direct packed ciphertext buffers, prepared masks, native result merging and compaction, lazy transforms, Barrett reduction and certified CRT conversion modulo q with exact fallback. Includes opt-in native phase profiling; existing BFV keys, wire formats and backends remain compatible. Rebuild the optional extension for ABI version 2. + - Optional native BFV `server_backend="rns"`, with independently implemented C++ RNS/NTT polynomial arithmetic, cached transformed evaluation keys, exact GMP reconstruction, and a native Hamming tile circuit. Preserves native BFV keys, ciphertext outputs and wire format. Includes native-boundary tests, sanitizer oracles, optional SEAL benchmarks and correctly tagged binary wheels. The previous GMP and reference backends remain available. - Native GMP BFV primitives in `crypto/encryption/bfv.py` and a `BFVClient` in `crypto/bfv_client.py`, with CRT batching, ciphertext multiplication/relinearization, public-key Hamming evaluation, packed distance responses, and terminal modulus compaction. Includes CPU benchmarks and tests against plaintext algebra and an optional SEAL oracle. This is an experimental local implementation; the existing Paillier clients and SEAL research example are preserved. - Offline BFV packed Hamming experiment under `experiments/bfv`, with an isolated public-key server evaluator, correctness tests, a comparison against an exported Paillier Git revision, and a research report. The production Paillier path and dependencies are unchanged. diff --git a/benchmarks/bfv_client_matrix.py b/benchmarks/bfv_client_matrix.py index 09d468f..9805683 100644 --- a/benchmarks/bfv_client_matrix.py +++ b/benchmarks/bfv_client_matrix.py @@ -237,7 +237,7 @@ def main() -> None: parser.add_argument("--response-modulus-bits", type=int, default=50) parser.add_argument("--no-compact", action="store_true") parser.add_argument( - "--server-backend", choices=("optimized", "reference", "rns"), default="optimized" + "--server-backend", choices=("optimized", "reference", "rns", "native"), default="optimized" ) parser.add_argument("--seed", type=int, default=1337) parser.add_argument( diff --git a/benchmarks/bfv_server.py b/benchmarks/bfv_server.py index 693960b..de3bb77 100644 --- a/benchmarks/bfv_server.py +++ b/benchmarks/bfv_server.py @@ -50,7 +50,7 @@ def main() -> None: parser.add_argument( "--backends", default="reference,optimized", - help="Comma-separated native backends: reference,optimized,rns", + help="Comma-separated native backends: reference,optimized,rns,native", ) parser.add_argument( "--seal", @@ -62,6 +62,11 @@ def main() -> None: type=Path, help="Run one extra warm search per backend under cProfile, outside reported timings", ) + parser.add_argument( + "--native-profile", + action="store_true", + help="Collect exclusive C++ phase timings in additional warm searches, outside reported timings", + ) args = parser.parse_args() if args.num_vectors < 1 or args.repeats < 1 or args.embed_len < 1: parser.error("num-vectors, embed-len and repeats must be positive") @@ -69,11 +74,13 @@ def main() -> None: if ( not names or len(set(names)) != len(names) - or any(b not in ("reference", "optimized", "rns") for b in names) + or any(b not in ("reference", "optimized", "rns", "native") for b in names) ): parser.error("backends must be distinct native backend names") if args.seal and (args.poly_modulus_degree != 8192 or args.plain_modulus != 65537): parser.error("the SEAL comparison requires N=8192 and t=65537") + if args.native_profile and not any(b in ("rns", "native") for b in names): + parser.error("native-profile requires the rns or native backend") backends = tuple(cast(BFVServerBackend, name) for name in names) vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) print("Generating one key set and encrypting the shared input", file=sys.stderr, flush=True) @@ -197,6 +204,16 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: profiler.dump_stats(str(args.profile_dir / f"{backend}.prof")) assert response == reference_response + native_phases = {} + if args.native_profile: + from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns + + for backend in backends: + if backend in ("rns", "native"): + response, phases = _bfv_rns.profile_call(partial(evaluate, backend)) + assert response == reference_response + native_phases[backend] = phases + summary = { backend: { "first_search_s": values[0], @@ -212,12 +229,13 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py", REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py", REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py", + REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py", *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.cpp"), *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.h"), REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile", REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", ] - if "rns" in backends: + if any(b in ("rns", "native") for b in backends): source_paths.extend((REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.so")) if args.seal: source_paths.append(REPO_ROOT / "experiments/bfv/packed_hamming.py") @@ -246,7 +264,7 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: ), }, "command": sys.argv, - "measurement_notes": "One CPU process, identical public keys and encrypted inputs for both backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", "config": config, "seed": args.seed, "vectors": len(vectors), @@ -271,6 +289,8 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: for backend, values in timings.items() }, "seal_comparison": seal_info, + "native_phases": native_phases, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", "sizes": { "encrypted_index_bytes": len(index_wire), "public_keys_bytes": key_bytes, @@ -279,7 +299,11 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: "query_plus_response_bytes": len(query_wire) + len(response_wire), }, "caches": { - backend: server._evaluator().cache_info() for backend, server in servers.items() + backend: { + **server._evaluator().cache_info(), + "server_plan_bytes": server._native().cache_bytes() if backend == "native" else 0, + } + for backend, server in servers.items() }, "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", "minimum_response_noise_budget_bits": min( diff --git a/benchmarks/results/native_bfv_cpp_server_1024.json b/benchmarks/results/native_bfv_cpp_server_1024.json new file mode 100644 index 0000000..fe9d1b3 --- /dev/null +++ b/benchmarks/results/native_bfv_cpp_server_1024.json @@ -0,0 +1,388 @@ +{ + "environment": { + "utc": "2026-09-11T01:33:24.194247+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "79ad443f2cc4606af8f142af075ecdcf8e62d8a2", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "1024", + "--backends", + "rns,native", + "--seal", + "--repeats", + "4", + "--native-profile", + "--json-out", + "benchmarks/results/native_bfv_cpp_server_1024.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "response_modulus_bits": 50, + "device": "cpu" + }, + "seed": 1337, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 4.330939290113747, + "index_encrypt_s": 6.664838324999437, + "query_encrypt_s": 0.10223700385540724, + "public_key_export_s": 0.8034978481009603, + "rns_public_key_import_s": 0.8361312141641974, + "native_public_key_import_s": 0.8487555379979312 + }, + "runs": [ + { + "backend": "rns", + "repeat": 0, + "server_compute_s": 8.21241048607044 + }, + { + "backend": "native", + "repeat": 0, + "server_compute_s": 5.681734326994047 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 2.366231924155727 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 2.358917977893725 + }, + { + "backend": "native", + "repeat": 1, + "server_compute_s": 5.342603240860626 + }, + { + "backend": "rns", + "repeat": 1, + "server_compute_s": 7.938400606857613 + }, + { + "backend": "rns", + "repeat": 2, + "server_compute_s": 7.9265675991773605 + }, + { + "backend": "native", + "repeat": 2, + "server_compute_s": 5.330243221018463 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 2.363118221051991 + }, + { + "backend": "seal", + "repeat": 3, + "server_compute_s": 2.359207842964679 + }, + { + "backend": "native", + "repeat": 3, + "server_compute_s": 5.342728961026296 + }, + { + "backend": "rns", + "repeat": 3, + "server_compute_s": 7.920907473890111 + } + ], + "summary": { + "rns": { + "first_search_s": 8.21241048607044, + "warm_searches_s": [ + 7.938400606857613, + 7.9265675991773605, + 7.920907473890111 + ], + "warm_median_s": 7.9265675991773605 + }, + "native": { + "first_search_s": 5.681734326994047, + "warm_searches_s": [ + 5.342603240860626, + 5.330243221018463, + 5.342728961026296 + ], + "warm_median_s": 5.342603240860626 + }, + "seal": { + "first_search_s": 2.366231924155727, + "warm_searches_s": [ + 2.358917977893725, + 2.363118221051991, + 2.359207842964679 + ], + "warm_median_s": 2.359207842964679 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "rns": { + "first": 1.0, + "warm_median": 1.0 + }, + "native": { + "first": 1.4454055774929662, + "warm_median": 1.4836526767614677 + }, + "seal": { + "first": 3.4706701410938963, + "warm_median": 3.3598428484437832 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 27678953, + "query_bytes": 432588, + "public_key_bytes": 39586623, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 105880, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "native_phases": { + "rns": { + "forward_ntt": { + "seconds": 1.561838909, + "calls": 18792 + }, + "inverse_ntt": { + "seconds": 0.759356965, + "calls": 7480 + }, + "to_residues": { + "seconds": 0.144149786, + "calls": 384 + }, + "crt_reconstruct": { + "seconds": 1.3842246550000001, + "calls": 1726 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.05560698100000001, + "calls": 703 + }, + "pointwise": { + "seconds": 0.421228254, + "calls": 895 + }, + "scale_round": { + "seconds": 0.159821076, + "calls": 192 + }, + "mod_q": { + "seconds": 0.485560737, + "calls": 1534 + }, + "buffers": { + "seconds": 0.026597349000000003, + "calls": 1151 + }, + "automorphism": { + "seconds": 0.35820702000000004, + "calls": 639 + }, + "add_sub": { + "seconds": 0.19624586500000002, + "calls": 704 + }, + "wire_import": { + "seconds": 0.0, + "calls": 0 + }, + "wire_export": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 2.388239892, + "calls": 1 + } + }, + "native": { + "forward_ntt": { + "seconds": 1.564787522, + "calls": 18280 + }, + "inverse_ntt": { + "seconds": 0.5568240910000001, + "calls": 7480 + }, + "to_residues": { + "seconds": 0.12858792700000002, + "calls": 256 + }, + "crt_reconstruct": { + "seconds": 0.865034472, + "calls": 1726 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.053277151, + "calls": 703 + }, + "pointwise": { + "seconds": 0.407613388, + "calls": 895 + }, + "scale_round": { + "seconds": 0.16121975600000002, + "calls": 193 + }, + "mod_q": { + "seconds": 0.49114541400000006, + "calls": 1534 + }, + "buffers": { + "seconds": 0.024296814, + "calls": 1023 + }, + "automorphism": { + "seconds": 0.35786369900000004, + "calls": 639 + }, + "add_sub": { + "seconds": 0.211706636, + "calls": 767 + }, + "wire_import": { + "seconds": 0.028895339000000003, + "calls": 130 + }, + "wire_export": { + "seconds": 0.00016453300000000002, + "calls": 2 + }, + "other": { + "seconds": 0.502915741, + "calls": 1 + } + } + }, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", + "sizes": { + "encrypted_index_bytes": 23598495, + "public_keys_bytes": 85603219, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "caches": { + "rns": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": false, + "switch_keys": 16, + "key_payload_bytes": 50331648, + "server_plan_bytes": 0 + }, + "native": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "switch_keys": 18, + "key_payload_bytes": 56623104, + "server_plan_bytes": 655360 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "6e8187062aef810a0cc25c4e3689a99ba79ee29ea5048fe6aa297daf6273dd40", + "benchmarks/bfv_client_matrix.py": "925f50c0e21c591d293c308e74dd5b5ab9ee9c6016543c956dbfce8b6ecec502", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "67ee7ebcce11b4b4bf55297a32811904a285c7c3f4edd884e61ef85af88f2636", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "6f03ac75f42218170da2bd785605ee394940a202afdd5471545a8ae3fdc8c3d8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "88c944adbf6a717cbf32e0a62d7e7d9f648f5db0d92e954c8a2e1307eeb45d81", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "96ce615ee4b187c45d3118c1a17df679d5ad9cbe2f233783e2a15ed43bd1aeff", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "176a4d01c38e0e20aa9e6ed2dd78bc5a88f50d5262b5a9434307d254befeab50", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "3931b73d1622443476cef5f9de2af03819364dd3cd54c5efe5e2ed49acd136f7", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "e019dad8d819626f051a31519557b26ff65648d0327e28ddcd943d2344f57b8a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "88ec7b5295f2916f45a531d695e6c18f91b3b8dddcae0ada8c0d4527a56ed4f8", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "26221a0fd074abd1251c03e5b54a7db3565e024ec942be94da10d6a1606e0233", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "ba256b062d975310c3de166cd5242c1cea5254250a769b942b0a49f5a7aed434", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "7a10c977ee648eb1975286b313f009b3508183efe9e942cd710ae6ad5ae279a3", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "c86ff6e20fe63db5599baaa086f87dbb3f33d956fa175388e054b79d765c8e0c", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/benchmarks/results/native_bfv_cpp_server_32.json b/benchmarks/results/native_bfv_cpp_server_32.json new file mode 100644 index 0000000..d2847f5 --- /dev/null +++ b/benchmarks/results/native_bfv_cpp_server_32.json @@ -0,0 +1,388 @@ +{ + "environment": { + "utc": "2026-09-11T01:31:51.513445+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "79ad443f2cc4606af8f142af075ecdcf8e62d8a2", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "32", + "--backends", + "rns,native", + "--seal", + "--repeats", + "4", + "--native-profile", + "--json-out", + "benchmarks/results/native_bfv_cpp_server_32.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "response_modulus_bits": 50, + "device": "cpu" + }, + "seed": 1337, + "vectors": 32, + "setup_timings": { + "key_generation_s": 4.342289729043841, + "index_encrypt_s": 0.21233072807081044, + "query_encrypt_s": 0.10064343316480517, + "public_key_export_s": 0.7829922570381314, + "rns_public_key_import_s": 0.856646987143904, + "native_public_key_import_s": 0.851454121991992 + }, + "runs": [ + { + "backend": "rns", + "repeat": 0, + "server_compute_s": 0.4936013799160719 + }, + { + "backend": "native", + "repeat": 0, + "server_compute_s": 0.5134266279637814 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 0.0763197538908571 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 0.07405148795805871 + }, + { + "backend": "native", + "repeat": 1, + "server_compute_s": 0.16401935601606965 + }, + { + "backend": "rns", + "repeat": 1, + "server_compute_s": 0.26051225513219833 + }, + { + "backend": "rns", + "repeat": 2, + "server_compute_s": 0.2602035889867693 + }, + { + "backend": "native", + "repeat": 2, + "server_compute_s": 0.1639279848895967 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 0.0742087010294199 + }, + { + "backend": "seal", + "repeat": 3, + "server_compute_s": 0.07412873883731663 + }, + { + "backend": "native", + "repeat": 3, + "server_compute_s": 0.1639478448778391 + }, + { + "backend": "rns", + "repeat": 3, + "server_compute_s": 0.2600402070675045 + } + ], + "summary": { + "rns": { + "first_search_s": 0.4936013799160719, + "warm_searches_s": [ + 0.26051225513219833, + 0.2602035889867693, + 0.2600402070675045 + ], + "warm_median_s": 0.2602035889867693 + }, + "native": { + "first_search_s": 0.5134266279637814, + "warm_searches_s": [ + 0.16401935601606965, + 0.1639279848895967, + 0.1639478448778391 + ], + "warm_median_s": 0.1639478448778391 + }, + "seal": { + "first_search_s": 0.0763197538908571, + "warm_searches_s": [ + 0.07405148795805871, + 0.0742087010294199, + 0.07412873883731663 + ], + "warm_median_s": 0.07412873883731663 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "rns": { + "first": 1.0, + "warm_median": 1.0 + }, + "native": { + "first": 0.9613864046624633, + "warm_median": 1.5871119817442698 + }, + "seal": { + "first": 6.467544177644472, + "warm_median": 3.5101580448820755 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 865239, + "query_bytes": 432560, + "public_key_bytes": 39586592, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 103312, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "native_phases": { + "rns": { + "forward_ntt": { + "seconds": 0.046839558, + "calls": 564 + }, + "inverse_ntt": { + "seconds": 0.022890933000000002, + "calls": 226 + }, + "to_residues": { + "seconds": 0.004476374, + "calls": 12 + }, + "crt_reconstruct": { + "seconds": 0.041671711, + "calls": 52 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.0015517270000000001, + "calls": 21 + }, + "pointwise": { + "seconds": 0.012541273, + "calls": 27 + }, + "scale_round": { + "seconds": 0.004987500000000001, + "calls": 6 + }, + "mod_q": { + "seconds": 0.014520551000000001, + "calls": 46 + }, + "buffers": { + "seconds": 0.000788527, + "calls": 35 + }, + "automorphism": { + "seconds": 0.010529403000000001, + "calls": 19 + }, + "add_sub": { + "seconds": 0.006080952000000001, + "calls": 22 + }, + "wire_import": { + "seconds": 0.0, + "calls": 0 + }, + "wire_export": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.09332303, + "calls": 1 + } + }, + "native": { + "forward_ntt": { + "seconds": 0.047080215, + "calls": 548 + }, + "inverse_ntt": { + "seconds": 0.016968431000000003, + "calls": 226 + }, + "to_residues": { + "seconds": 0.004014191, + "calls": 8 + }, + "crt_reconstruct": { + "seconds": 0.026213566, + "calls": 52 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.0016716600000000002, + "calls": 21 + }, + "pointwise": { + "seconds": 0.012280482, + "calls": 27 + }, + "scale_round": { + "seconds": 0.006211254, + "calls": 7 + }, + "mod_q": { + "seconds": 0.014689607, + "calls": 46 + }, + "buffers": { + "seconds": 0.00095556, + "calls": 31 + }, + "automorphism": { + "seconds": 0.010587844, + "calls": 19 + }, + "add_sub": { + "seconds": 0.006361085000000001, + "calls": 23 + }, + "wire_import": { + "seconds": 0.0013337070000000002, + "calls": 6 + }, + "wire_export": { + "seconds": 0.000158364, + "calls": 2 + }, + "other": { + "seconds": 0.015787336000000003, + "calls": 1 + } + } + }, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", + "sizes": { + "encrypted_index_bytes": 737480, + "public_keys_bytes": 85603223, + "query_bytes": 368754, + "response_bytes": 102494, + "query_plus_response_bytes": 471248 + }, + "caches": { + "rns": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": false, + "switch_keys": 11, + "key_payload_bytes": 34603008, + "server_plan_bytes": 0 + }, + "native": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "switch_keys": 18, + "key_payload_bytes": 56623104, + "server_plan_bytes": 655360 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 31, + "distance": 0 + }, + { + "index": 19, + "distance": 241 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "6e8187062aef810a0cc25c4e3689a99ba79ee29ea5048fe6aa297daf6273dd40", + "benchmarks/bfv_client_matrix.py": "925f50c0e21c591d293c308e74dd5b5ab9ee9c6016543c956dbfce8b6ecec502", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "67ee7ebcce11b4b4bf55297a32811904a285c7c3f4edd884e61ef85af88f2636", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "6f03ac75f42218170da2bd785605ee394940a202afdd5471545a8ae3fdc8c3d8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "88c944adbf6a717cbf32e0a62d7e7d9f648f5db0d92e954c8a2e1307eeb45d81", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "96ce615ee4b187c45d3118c1a17df679d5ad9cbe2f233783e2a15ed43bd1aeff", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "176a4d01c38e0e20aa9e6ed2dd78bc5a88f50d5262b5a9434307d254befeab50", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "3931b73d1622443476cef5f9de2af03819364dd3cd54c5efe5e2ed49acd136f7", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "e019dad8d819626f051a31519557b26ff65648d0327e28ddcd943d2344f57b8a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "88ec7b5295f2916f45a531d695e6c18f91b3b8dddcae0ada8c0d4527a56ed4f8", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "26221a0fd074abd1251c03e5b54a7db3565e024ec942be94da10d6a1606e0233", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "ba256b062d975310c3de166cd5242c1cea5254250a769b942b0a49f5a7aed434", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "7a10c977ee648eb1975286b313f009b3508183efe9e942cd710ae6ad5ae279a3", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "c86ff6e20fe63db5599baaa086f87dbb3f33d956fa175388e054b79d765c8e0c", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/docs/api_rst/xtrace_sdk.crypto.encryption.rst b/docs/api_rst/xtrace_sdk.crypto.encryption.rst index acb1be0..465b3b8 100644 --- a/docs/api_rst/xtrace_sdk.crypto.encryption.rst +++ b/docs/api_rst/xtrace_sdk.crypto.encryption.rst @@ -28,6 +28,14 @@ xtrace\_sdk.crypto.encryption.bfv\_rns module :show-inheritance: :undoc-members: +xtrace\_sdk.crypto.encryption.bfv\_native module +---------------------------------------------- + +.. automodule:: xtrace_sdk.x_vec.crypto.encryption.bfv_native + :members: + :show-inheritance: + :undoc-members: + xtrace\_sdk.crypto.encryption.aes module ---------------------------------------- diff --git a/docs/index.rst b/docs/index.rst index 06b3abc..dee7bcd 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -83,6 +83,7 @@ Choose your path: research/native-bfv research/native-bfv-ntt + research/native-bfv-server research/bfv-packed-hamming .. toctree:: diff --git a/docs/research/native-bfv-ntt.md b/docs/research/native-bfv-ntt.md index 443c51c..b22244e 100644 --- a/docs/research/native-bfv-ntt.md +++ b/docs/research/native-bfv-ntt.md @@ -6,6 +6,10 @@ modulus, error distribution, packing layout and wire format. The default `"optimized"` GMP evaluator and the original `"reference"` evaluator remain available. The earlier SEAL experiment is preserved for comparison. +The subsequent [`native` server backend](native-bfv-server.md) runs complete +searches inside C++, adds native phase profiling, and further optimizes this +arithmetic while preserving the same keys and ciphertexts. + ## What we learned from SEAL The source review used the pinned **SEAL v4.1.2** tag, rather than an unversioned diff --git a/docs/research/native-bfv-server.md b/docs/research/native-bfv-server.md new file mode 100644 index 0000000..2ac2aeb --- /dev/null +++ b/docs/research/native-bfv-server.md @@ -0,0 +1,250 @@ +# Complete C++ BFV server evaluation + +`BFVClient(server_backend="native")` evaluates a complete encrypted Hamming +search inside our own C++ implementation. Python passes packed ciphertext +buffers in and receives packed result buffers out. Masks, polynomial operations, +rotations, the result merge tree, and terminal modulus compaction run in C++. +Key generation, encryption and decryption continue to use `encryption/bfv.py`. + +This is an optional server backend with the same parameters, keys, packing and +wire format as the earlier native BFV implementation. The `optimized`, +`reference` and `rns` backends remain available. It uses GMP and the C++ standard +library, and does not compile or link SEAL. Like the existing scheme, this is +experimental leveled BFV without a security audit or bootstrapping. + +## Build and use + +Rebuild the extension from the repository root. The new interface is ABI version +2; an older binary raises a rebuild error. + +```bash +make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" +``` + +```python +import json +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + +client = BFVClient(embed_len=512) +server = BFVClient(skip_key_gen=True, server_backend="native") +server.load_config(json.loads(client.stringify_config())) +server.load_stringified_keys(client.stringify_pk()) +assert server.keys is None + +query = [0, 1] * 256 +vectors = [query, [1 - bit for bit in query]] +encrypted_query = client.encrypt_vec_one(query) +encrypted_index = client.encrypt_vec_packed(vectors) +response = server.encode_hamming_server_packed(encrypted_query, encrypted_index, 2) +assert client.decode_hamming_client_packed(response, 2) == [0, 512] +``` + +The existing individual `encode_hamming_server` interface also uses the complete +native evaluator. Reuse a server across queries to reuse prepared public keys +and the full-tile mask. Loading keys or changing the configured layout resets +the native plan. `server_backend` remains a runtime option; the default is +`optimized`. This does not add BFV support to the production HTTP endpoints. + +## What profiling established + +The first instrumented full tile at N=8192 and dimension 512 spent about 35% of +native time in forward/inverse transforms and 30% in CRT reconstruction and +reduction modulo q. These measurements motivated both arithmetic changes and +moving the rest of the search into C++. The original Python profile could not +separate these costs inside a native call. + +The benchmark now supports `--native-profile`. After its timed searches, it runs +one additional warm search for each compiled backend and records exclusive +native phase durations and call counts. Nested timings are subtracted from their +parent category. Timers read the clock only during an opt-in profiling session. +Sessions are per-thread and restored when a call raises an exception. + +`other` includes Python work, binding overhead and native work outside named +scopes. `buffers` covers explicitly instrumented buffer setup. GMP allocations +remain included in the arithmetic operations that perform them; the profile +does not claim to isolate every allocator call. Use normal benchmark samples, +not instrumented runs, for speed comparisons. + +```bash +.venv/bin/python benchmarks/bfv_server.py --num-vectors 1024 \ + --backends rns,native --seal --repeats 4 --native-profile \ + --json-out /tmp/bfv-native-server-1024.json +``` + +Omit `--seal` when TenSEAL is unavailable. Native backends share exactly the +same encrypted input and must return identical ciphertexts. SEAL uses the same +plaintext vectors with its own parameters and format. The comparison is a local +server-call measurement, not an equal-security or identical-kernel comparison. +Native search times include ciphertext framing/buffer conversion, arithmetic +and compaction, but exclude outer MessagePack. SEAL includes its envelope and +binding temporary-file I/O. Key import, encryption, client decryption and network +latency are excluded from both. + +## Recorded results + +Measured on the same AMD Ryzen 7 5800X CPU with Python 3.12.3, GMP 6.3.0, +GCC 13.3.0 (`-O3`) and TenSEAL 0.3.16. Each vector has 512 bits. Both native +backends use N=8192, t=65537, the original 180-bit q, six 30-bit gadget digits, +and 50-bit response compaction. Each backend has one first search and three +subsequent searches; the warm column reports their median. + +| Vectors | Backend | First search (s) | Warm median (s) | +| ---: | --- | ---: | ---: | +| 32 | Previous `rns` | 0.4936 | 0.2602 | +| 32 | Complete C++ `native` | 0.5134 | 0.1639 | +| 32 | SEAL prototype | 0.0763 | 0.0741 | +| 1,024 | Previous `rns` | 8.2124 | 7.9266 | +| 1,024 | Complete C++ `native` | 5.6817 | 5.3426 | +| 1,024 | SEAL prototype | 2.3662 | 2.3592 | + +For 1,024 vectors, the complete C++ backend is **1.48 times faster** than the +previous RNS backend, reducing server time by **32.6%**. SEAL remains **2.26 +times faster** in this practical comparison. Native ciphertexts matched exactly +on every run. The 102,496-byte response, 471,250-byte query plus response and +26-bit minimum response noise budget are the same for both native backends. + +The additional whole-search profile explains where time changed: + +| Exclusive phase, 1,024 vectors | Previous `rns` (s) | Complete C++ `native` (s) | +| --- | ---: | ---: | +| Forward transforms | 1.562 | 1.565 | +| Inverse transforms | 0.759 | 0.557 | +| CRT reconstruction/conversion | 1.384 | 0.865 | +| Reduction modulo q | 0.486 | 0.491 | +| Pointwise work | 0.421 | 0.408 | +| Other: Python, binding and unscoped native work | 2.388 | 0.503 | + +The remaining phases are recorded in the JSON files. These are exclusive +instrumented durations, not additional speed benchmark samples. The biggest +reductions are in Python/binding work and CRT conversion; inverse transforms +also improve. Forward-transform and pointwise times are essentially unchanged +in this profile, so it does not establish an individual speedup for every new +primitive. The certified CRT fallback count is zero on these searches; the +boundary tests exercise it deliberately. + +The native plan prepares all 18 keys needed by this layout, while `rns` prepares +keys as operations request them (16 for 1,024 vectors, 11 for 32). This explains +the slightly higher first-use cost for the small workload. The native cache has +56,623,104 bytes of key coefficient arrays, 2,752,512 bytes of transform tables +and 655,360 bytes of prepared plaintext tables/mask. These are additional to the +loaded public key, exclude containers/CRT constants/temporary allocations, and +are not peak process memory. + +Raw samples, native profiles, parameters and source/binary hashes are in +`benchmarks/results/native_bfv_cpp_server_32.json` and +`benchmarks/results/native_bfv_cpp_server_1024.json`. The earlier benchmark +files are retained. + +## Changes inside the native server + +- `bfv_server.h` owns the public evaluation-key handles, prepared plaintext NTT + tables and transformed full-tile mask. A partial mask is local to a search. + The same mask transforms are reused for both ciphertext components. +- `packed_wire.h` imports/exports the original bit-packed polynomial format + directly in C++. It checks lengths and coefficient bounds. Python validates + the small framing header, without unpacking N Python integers per polynomial. +- A streaming binary merge tree retains O(log(padded dimension)) intermediate + ciphertexts. It parses each indexed tile once and keeps results in C++ through + compaction. Native calls release the GIL; plans are immutable and working + arrays belong to each call. +- The `native` arithmetic mode uses lazy NTT butterflies and reciprocal-based + Barrett reduction. The `rns` mode retains the earlier canonical butterflies + and general remainder operations for comparison. +- Key switching and prepared-mask multiplication compute their final result + modulo q using certified CRT conversion. Ciphertext multiplication still + reconstructs its full signed tensor product before BFV scale-and-round. + +These build on the [earlier SEAL source review](native-bfv-ntt.md). The +[SEAL evaluator](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/evaluator.cpp) +and [NTT routines](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/util/ntt.cpp) +use specialized modular arithmetic and lazy transforms. Our implementations and +the CRT bounds below are independent. + +## Exact CRT conversion modulo q + +The auxiliary primes p_j still provide enough range to identify the signed +integer convolution, as described in [the original bounds](native-bfv-ntt.md). +Let M be their product, M_j = M/p_j, and r_j an output residue after inverse NTT. +Define: + +```text +y_j = r_j * inverse(M_j mod p_j) mod p_j +theta = sum(y_j / p_j) +k = floor(theta + 1/2) +signed result = sum(y_j * M_j) - k*M +``` + +We need only the last expression modulo q for key switching and mask products. +Precomputing `M_j mod q` and `M mod q` lets us form that residue without first +constructing the full integer modulo M. GMP still holds the final coefficients. + +The value k must be exact. With entirely integer arithmetic, compute: + +```text +F = sum(floor(2^64 * y_j / p_j)) +F/2^64 <= theta < (F + number_of_primes)/2^64 +``` + +If both ends of this interval round to the same integer, k is certified. If +they straddle a rounding boundary, use the original exact GMP CRT reconstruction +for that coefficient. No floating-point approximation or probabilistic +correctness assumption is used. Tests explicitly place integers on either side +of M/2 and verify that this fallback runs and agrees with the integer oracle. + +Ciphertext tensor products require the unreduced integer before multiplication +by t/q. They keep the original exact reconstruction and signed rounding rule; +reducing them modulo q early would be incorrect. + +## Native arithmetic bounds + +The fast forward transform keeps internal values below 2p and canonicalizes +its final output. The fast inverse keeps internal values below 4p and produces +canonical coefficients after inverse scaling. With p<2^60, these additions and +Shoup products fit the existing word bounds. Canonical forward outputs preserve +the key-switch accumulator proof: products are below 2^120, and batches of 256 +fit unsigned 128-bit accumulators. + +Barrett reduction stores `floor(2^128/p)`. The estimated quotient is at most one +below the true quotient, so a single correction gives the exact remainder for +any unsigned 128-bit input. The certified CRT fraction calculation specializes +this reciprocal calculation to `floor(y*2^64/p)`, with y None: self.vectors_per_ciphertext = 2 * self.lanes_per_row self.distances_per_ciphertext = n self._mask_cache: dict[int, BFVPolynomial] = {} + self._native_server: BFVNativeServer | None = None def _rotation_steps(self) -> list[int]: return [ @@ -134,6 +136,21 @@ def _evaluator(self) -> BFVEvaluator: def has_gpu() -> bool: return False + def _native(self) -> BFVNativeServer: + arithmetic = self._evaluator()._rns + if arithmetic is None: + raise ValueError("The native server requires a compiled BFV backend") + if ( + self._native_server is None + or self._native_server.arithmetic is not arithmetic + or self._native_server.padded_embed_len != self.padded_embed_len + or self._native_server.response_modulus_bits != self.response_modulus_bits + ): + self._native_server = BFVNativeServer( + arithmetic, self.padded_embed_len, self.response_modulus_bits + ) + return self._native_server + def stringify_pk(self) -> str: """Public encryption/evaluation keys; safe to separate from the secret key.""" return BFV.serialize_public_key(self._pk()) @@ -185,6 +202,7 @@ def load_stringified_keys(self, pk: str, sk: str | None = None) -> None: self.keys = keys self._server_evaluator = None self._evaluator_public_key = None + self._native_server = None def _validate_vector(self, embd: Sequence[int]) -> None: if len(embd) != self.embed_len: @@ -262,6 +280,8 @@ def encode_hamming_server( compact: bool = True, ) -> EncryptedVector: """Compute an encrypted distance using only public evaluation keys.""" + if self.server_backend == "native": + return self._native().search(ct1, [ct2], 1, compact=compact)[0] pk = self._pk() result = self._distance_tile( BFV.ciphertext_from_ints(ct1, pk), BFV.ciphertext_from_ints(ct2, pk), 1 @@ -287,6 +307,8 @@ def encode_hamming_server_packed( pk = self._pk() capacity = self.vectors_per_ciphertext self._validate_count(vector_count, len(index), capacity) + if self.server_backend == "native": + return self._native().search(query, index, vector_count, compact=compact) query_ct = BFV.ciphertext_from_ints(query, pk) evaluator = self._evaluator() result = [] diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile index 9657a09..1d66de0 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile @@ -15,7 +15,7 @@ LDLIBS ?= -lgmpxx -lgmp .PHONY: all clean all: $(TARGET) -$(TARGET): bindings.cpp rns_ntt.h Makefile +$(TARGET): bindings.cpp rns_ntt.h profile.h bfv_server.h packed_wire.h Makefile $(CXX) $(CPPFLAGS) $(CXXFLAGS) -std=c++17 -Wall -Wextra -fPIC -shared \ -I$(PY_INC) bindings.cpp $(LDFLAGS) $(LDLIBS) -o $@ diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi index 5d9bacd..e65625d 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi @@ -2,7 +2,7 @@ from typing import Any ABI_VERSION: int -def create_ring(n: int, q_hex: str, digit_bits: int) -> object: ... +def create_ring(n: int, q_hex: str, digit_bits: int, fast: bool = False) -> object: ... def compile_key(ring: object, key: tuple[tuple[bytes, bytes], ...]) -> object: ... def apply_key(key: object, poly: bytes) -> tuple[bytes, bytes]: ... def ring_product(ring: object, lhs: bytes, rhs: bytes) -> bytes: ... @@ -22,3 +22,17 @@ def hamming_tile( ) -> tuple[bytes, bytes]: ... def ring_info(ring: object) -> dict[str, Any]: ... def key_bytes(key: object) -> int: ... +def profile_hamming_tile(*args: Any) -> tuple[tuple[bytes, bytes], dict[str, Any]]: ... +def create_server( + relin: object, keys: tuple[tuple[int, object], ...], padded: int, t: int, target_hex: str +) -> object: ... +def packed_search( + server: object, + query: tuple[bytes, bytes], + index: tuple[tuple[bytes, bytes], ...], + count: int, + compact: bool, +) -> list[tuple[bytes, bytes]]: ... +def profile_packed_search(*args: Any) -> tuple[list[tuple[bytes, bytes]], dict[str, Any]]: ... +def server_bytes(server: object) -> int: ... +def profile_call(callback: Any) -> tuple[Any, dict[str, Any]]: ... diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h new file mode 100644 index 0000000..f5e358d --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h @@ -0,0 +1,154 @@ +// Complete public-key BFV Hamming evaluation, independent of Python objects. +// Ciphertexts cross the binding once, retain native coefficients through the +// circuit and merge tree, and are serialized only after terminal compaction. +#pragma once +#include "rns_ntt.h" +#include +#include + +namespace xtrace_bfv { +using KeyHandle = std::shared_ptr; + +class PreparedPlain { + const Ring& ring_; + Residues value_; +public: + PreparedPlain(const Ring& ring, const Polynomial& value) : ring_(ring) { + mpz_class bound = ring.n * (ring.q - 1) * *std::max_element(value.begin(), value.end()); + value_ = ring.encode(value, ring.prime_count(bound)); + } + Polynomial apply(const Polynomial& value) const { + auto transformed = ring_.encode(value, value_.size()); + { ProfileScope timing(Phase::pointwise); + for (std::size_t j = 0; j < value_.size(); ++j) + for (std::size_t i = 0; i < ring_.n; ++i) + transformed[j][i] = ring_.transforms[j].remainder(Wide(transformed[j][i]) * value_[j][i]); } + return ring_.decode_mod_q(std::move(transformed)); + } + std::size_t bytes() const { return value_.size() * ring_.n * sizeof(Word); } +}; + +class HammingServer { + KeyHandle relin_; + std::map keys_; + PrimeNTT plaintext_; + std::vector left_, right_; + std::unique_ptr full_mask_; + + Word exponent(std::size_t step) const { return power_mod(3, step % (ring->n / 2), 2 * ring->n); } + const SwitchKey& key(Word g) const { + auto found = keys_.find(g); + if (found == keys_.end()) throw std::invalid_argument("Missing native server rotation key"); + return *found->second; + } + Polynomial mask(std::size_t count) const { + std::vector evaluations(ring->n); + unsigned log_n = 0; + for (std::size_t v = ring->n; v > 1; v >>= 1) ++log_n; + Word g = 1; + for (std::size_t lane = 0; lane < lanes; ++lane) { + for (std::size_t row = 0; row < 2; ++row) { + if (row * lanes + lane >= count) continue; + std::size_t position = ((row ? 2 * ring->n - g : g) - 1) / 2; + std::size_t reversed = 0; + for (unsigned bit = 0; bit < log_n; ++bit) { reversed = 2 * reversed + (position & 1); position >>= 1; } + evaluations[reversed] = 1; + } + g = g * 3 % (2 * ring->n); + } + plaintext_.inverse(evaluations); + Polynomial result(ring->n); + for (std::size_t i = 0; i < ring->n; ++i) result[i] = evaluations[i]; + return result; + } + +public: + const std::shared_ptr ring; + const std::size_t padded, lanes, capacity; + const Word t; + const mpz_class target; + + HammingServer(KeyHandle relin, std::map keys, + std::size_t padded, Word t, const mpz_class& target) + : relin_(std::move(relin)), keys_(std::move(keys)), plaintext_(relin_->ring->n, t, relin_->ring->fast), + ring(relin_->ring), padded(padded), lanes(ring->n / (2 * padded)), + capacity(2 * lanes), t(t), target(target) { + // The binding checks algebraic parameters before constructing the NTT. + for (std::size_t size = 1; size < padded; size *= 2) { + left_.push_back(exponent(lanes * size)); + right_.push_back(exponent(ring->n / 2 - lanes * size)); + key(left_.back()); key(right_.back()); + } + full_mask_ = std::make_unique(*ring, mask(capacity)); + } + + Ciphertext tile(const Ciphertext& query, const Ciphertext& indexed, const PreparedPlain& selected) const { + Ciphertext difference = query; + { ProfileScope timing(Phase::add_sub); + for (std::size_t k = 0; k < 2; ++k) + for (std::size_t i = 0; i < ring->n; ++i) { + difference[k][i] -= indexed[k][i]; + if (difference[k][i] < 0) difference[k][i] += ring->q; + } } + auto product = ring->multiply(difference, difference, t, true); + auto distance = relin_->apply(product[2]); + add_inplace(distance, {std::move(product[0]), std::move(product[1])}, *ring); + for (auto g : left_) { + auto rotated = rotate(distance, g, key(g)); + add_inplace(distance, rotated, *ring); + } + for (auto& component : distance) component = selected.apply(component); + return distance; + } + + void compact(Ciphertext& value) const { + ProfileScope timing(Phase::scale_round); + mpz_class denominator = 2 * ring->q; + for (auto& component : value) + for (auto& c : component) { + c = 2 * target * c + ring->q; + mpz_fdiv_q(c.get_mpz_t(), c.get_mpz_t(), denominator.get_mpz_t()); + mpz_mod(c.get_mpz_t(), c.get_mpz_t(), target.get_mpz_t()); + } + } + + // read_tile/emit allow the binding to stream packed bytes with the GIL + // released. At most O(log(padded)) intermediate ciphertexts are retained. + template + void search(const Ciphertext& query, std::size_t count, ReadTile read_tile, Emit emit, bool compact_result) const { + const auto tiles = (count + capacity - 1) / capacity; + std::unique_ptr partial; + if (count % capacity) partial = std::make_unique(*ring, mask(count % capacity)); + for (std::size_t start = 0; start < tiles; start += padded) { + std::vector> stack; + for (std::size_t at = start; at < std::min(start + padded, tiles); ++at) { + const auto& selected = at + 1 == tiles && partial ? *partial : *full_mask_; + auto result = tile(query, read_tile(at), selected); + std::size_t size = 1, level = 0; + while (!stack.empty() && stack.back().second == size) { + auto rotated = rotate(result, right_.at(level), key(right_[level])); + result = std::move(stack.back().first); + stack.pop_back(); + add_inplace(result, rotated, *ring); + size *= 2; ++level; + } + stack.emplace_back(std::move(result), size); + } + auto result = std::move(stack.back().first); + stack.pop_back(); + while (!stack.empty()) { + auto size = stack.back().second; + std::size_t level = 0; + while ((std::size_t(1) << level) < size) ++level; + auto rotated = rotate(result, right_.at(level), key(right_[level])); + result = std::move(stack.back().first); + stack.pop_back(); + add_inplace(result, rotated, *ring); + } + if (compact_result) compact(result); + emit(result); + } + } + std::size_t bytes() const { return plaintext_.bytes() + full_mask_->bytes(); } +}; +} // namespace xtrace_bfv diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp index 3debc1a..1a5efcb 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp @@ -2,14 +2,18 @@ #define PY_SSIZE_T_CLEAN #include #include "rns_ntt.h" +#include "bfv_server.h" +#include "packed_wire.h" #include using namespace xtrace_bfv; namespace { using RingPtr = std::shared_ptr; using KeyPtr = std::shared_ptr; +using ServerPtr = std::shared_ptr; constexpr const char* ring_name = "xtrace.bfv.rns.ring.v1"; constexpr const char* key_name = "xtrace.bfv.rns.key.v1"; +constexpr const char* server_name = "xtrace.bfv.server.v1"; struct PythonError {}; struct WithoutGIL { @@ -103,14 +107,15 @@ PyObject* create_ring(PyObject*, PyObject* args) { return checked([&]() -> PyObject* { PyObject *n_object, *bits_object; const char* q_hex; - if (!PyArg_ParseTuple(args, "OsO", &n_object, &q_hex, &bits_object)) return nullptr; + int fast = 0; + if (!PyArg_ParseTuple(args, "OsO|p", &n_object, &q_hex, &bits_object, &fast)) return nullptr; auto n = PyLong_AsUnsignedLongLong(n_object), bits = PyLong_AsUnsignedLongLong(bits_object); if (PyErr_Occurred()) return nullptr; if (n > 32768 || bits > 512) throw std::invalid_argument("Invalid native BFV ring parameters"); mpz_class q; if (q.set_str(q_hex, 16) != 0) throw std::invalid_argument("Invalid ciphertext modulus"); RingPtr ring; - { WithoutGIL release; ring = std::make_shared(n, q, bits); } + { WithoutGIL release; ring = std::make_shared(n, q, bits, fast); } return capsule(std::move(ring), ring_name); }); } @@ -241,10 +246,10 @@ PyObject* ring_info(PyObject*, PyObject* argument) { if (!value) { Py_DECREF(primes); return nullptr; } PyTuple_SET_ITEM(primes, i, value); } - return Py_BuildValue("{s:N,s:n,s:n,s:s,s:s}", "primes", primes, + return Py_BuildValue("{s:N,s:n,s:n,s:s,s:s,s:O}", "primes", primes, "switch_prime_count", ring->switch_prime_count, "transform_table_bytes", ring->bytes(), "gmp", gmp_version, - "compiler", __VERSION__); + "compiler", __VERSION__, "fast_arithmetic", ring->fast ? Py_True : Py_False); }); } @@ -254,6 +259,128 @@ PyObject* key_bytes(PyObject*, PyObject* argument) { }); } +PyObject* create_server(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *relin_object, *keys_object, *padded_object, *t_object; + const char* target_hex; + if (!PyArg_ParseTuple(args, "OOOOs", &relin_object, &keys_object, &padded_object, &t_object, &target_hex)) return nullptr; + auto relin = get_capsule(relin_object, key_name); + const auto& ring = *relin->ring; + auto padded = PyLong_AsUnsignedLongLong(padded_object); + if (PyErr_Occurred()) return nullptr; + Word t = read_t(t_object, ring); + mpz_class plain(t), target; + if (!padded || padded > ring.n / 2 || (padded & (padded - 1)) || + t % (2 * ring.n) != 1 || !mpz_probab_prime_p(plain.get_mpz_t(), 32)) + throw std::invalid_argument("Invalid native server layout or plaintext modulus"); + if (target.set_str(target_hex, 16) != 0 || target <= plain || target > ring.q) + throw std::invalid_argument("Invalid native response modulus"); + if (!PyTuple_Check(keys_object)) throw std::invalid_argument("Rotation keys must be a tuple"); + std::map keys; + for (Py_ssize_t i = 0; i < PyTuple_GET_SIZE(keys_object); ++i) { + auto entry = PyTuple_GET_ITEM(keys_object, i); + if (!PyTuple_Check(entry) || PyTuple_GET_SIZE(entry) != 2) throw std::invalid_argument("Invalid rotation key entry"); + Word g = read_exponent(PyTuple_GET_ITEM(entry, 0), ring); + auto key = get_capsule(PyTuple_GET_ITEM(entry, 1), key_name); + if (key->ring != relin->ring) throw std::invalid_argument("Rotation key belongs to another native ring"); + if (!keys.emplace(g, std::move(key)).second) throw std::invalid_argument("Duplicate rotation key"); + } + ServerPtr server; + { WithoutGIL release; server = std::make_shared(relin, std::move(keys), padded, t, target); } + return capsule(std::move(server), server_name); + }); +} + +using PackedPair = std::array; +PackedPair packed_pair(PyObject* object, const Ring& ring) { + if (!PyTuple_Check(object) || PyTuple_GET_SIZE(object) != 2) throw std::invalid_argument("Expected two packed components"); + PackedPair result; + auto bytes = (ring.n * mpz_sizeinbase(ring.q.get_mpz_t(), 2) + 7) / 8; + for (std::size_t i = 0; i < 2; ++i) { + auto value = PyTuple_GET_ITEM(object, i); + if (!PyBytes_Check(value) || PyBytes_GET_SIZE(value) != static_cast(bytes)) + throw std::invalid_argument("Incorrect packed polynomial bytes"); + result[i] = {PyBytes_AS_STRING(value), bytes}; + } + return result; +} + +PyObject* packed_search(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *server_object, *query_object, *index_object, *count_object; + int compact; + if (!PyArg_ParseTuple(args, "OOOOp", &server_object, &query_object, &index_object, &count_object, &compact)) return nullptr; + auto server = get_capsule(server_object, server_name); + auto count = PyLong_AsUnsignedLongLong(count_object); + if (PyErr_Occurred()) return nullptr; + if (!PyTuple_Check(index_object) || count > SIZE_MAX - server->capacity || + (count + server->capacity - 1) / server->capacity != static_cast(PyTuple_GET_SIZE(index_object))) + throw std::invalid_argument("vector_count does not match the packed ciphertext count"); + auto query_bytes = packed_pair(query_object, *server->ring); + std::vector index; + for (Py_ssize_t i = 0; i < PyTuple_GET_SIZE(index_object); ++i) + index.push_back(packed_pair(PyTuple_GET_ITEM(index_object, i), *server->ring)); + std::vector> output; + { + WithoutGIL release; + const auto& ring = *server->ring; + Ciphertext query{import_packed(query_bytes[0], ring), import_packed(query_bytes[1], ring)}; + server->search(query, count, [&](std::size_t at) -> Ciphertext { + return {import_packed(index[at][0], ring), import_packed(index[at][1], ring)}; + }, [&](const Ciphertext& value) { + const auto& q = compact ? server->target : ring.q; + output.push_back({export_packed(value[0], q), export_packed(value[1], q)}); + }, compact); + } + PyObject* result = PyList_New(output.size()); + if (!result) return nullptr; + for (std::size_t i = 0; i < output.size(); ++i) { + auto& v = output[i]; + PyObject* pair = Py_BuildValue("(y#y#)", v[0].data(), static_cast(v[0].size()), + v[1].data(), static_cast(v[1].size())); + if (!pair) { Py_DECREF(result); return nullptr; } + PyList_SET_ITEM(result, i, pair); + } + return result; + }); +} + +PyObject* server_bytes(PyObject*, PyObject* argument) { + return checked([&]() -> PyObject* { + return PyLong_FromSize_t(get_capsule(argument, server_name)->bytes()); + }); +} + +template +PyObject* profiled(Function function, PyObject* args) { + return checked([&]() -> PyObject* { + NativeProfile profile; + PyObject* result; + { ProfileSession session(profile); ProfileScope total(Phase::other); + result = function(nullptr, args); } + if (!result) return nullptr; + PyObject* timings = PyDict_New(); + if (!timings) { Py_DECREF(result); return nullptr; } + for (std::size_t i = 0; i < profile.ns.size(); ++i) { + PyObject* entry = Py_BuildValue("{s:d,s:K}", "seconds", profile.ns[i] * 1e-9, + "calls", static_cast(profile.calls[i])); + if (!entry || PyDict_SetItemString(timings, phase_names[i], entry) < 0) { + Py_XDECREF(entry); Py_DECREF(timings); Py_DECREF(result); return nullptr; + } + Py_DECREF(entry); + } + PyObject* output = PyTuple_Pack(2, result, timings); + Py_DECREF(result); Py_DECREF(timings); + return output; + }); +} + +PyObject* profile_hamming_tile(PyObject*, PyObject* args) { return profiled(hamming_tile, args); } +PyObject* profile_packed_search(PyObject*, PyObject* args) { return profiled(packed_search, args); } +PyObject* profile_call(PyObject*, PyObject* callback) { + return profiled([](PyObject*, PyObject* value) { return PyObject_CallNoArgs(value); }, callback); +} + PyMethodDef methods[] = { {"create_ring", create_ring, METH_VARARGS, "Prepare exact CRT bases and negacyclic NTT tables."}, {"compile_key", compile_key, METH_VARARGS, "Cache a gadget key in RNS/NTT form."}, @@ -262,6 +389,12 @@ PyMethodDef methods[] = { {"multiply", multiply, METH_VARARGS, "BFV tensor product and exact signed scale-and-round."}, {"rotate_rows", rotate_rows, METH_VARARGS, "Galois automorphism and public-key switching."}, {"hamming_tile", hamming_tile, METH_VARARGS, "Evaluate the native Hamming tile circuit."}, + {"profile_hamming_tile", profile_hamming_tile, METH_VARARGS, "Return a tile and exclusive per-phase native timings."}, + {"create_server", create_server, METH_VARARGS, "Prepare an immutable public-key native Hamming server."}, + {"packed_search", packed_search, METH_VARARGS, "Evaluate a complete packed search using native wire buffers."}, + {"profile_packed_search", profile_packed_search, METH_VARARGS, "Return a search and exclusive per-phase native timings."}, + {"profile_call", profile_call, METH_O, "Profile native phases of a synchronous callable; other includes Python work."}, + {"server_bytes", server_bytes, METH_O, "Payload bytes in prepared plaintext tables and mask."}, {"ring_info", ring_info, METH_O, "Report public auxiliary-prime and table information."}, {"key_bytes", key_bytes, METH_O, "Size of cached NTT coefficient arrays."}, {nullptr, nullptr, 0, nullptr} @@ -272,7 +405,7 @@ PyModuleDef module = {PyModuleDef_HEAD_INIT, "_bfv_rns", "Native BFV RNS/NTT CPU PyMODINIT_FUNC PyInit__bfv_rns() { PyObject* result = PyModule_Create(&module); - if (result && PyModule_AddIntConstant(result, "ABI_VERSION", 1) < 0) { + if (result && PyModule_AddIntConstant(result, "ABI_VERSION", 2) < 0) { Py_DECREF(result); return nullptr; } return result; diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h new file mode 100644 index 0000000..65b7fa6 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h @@ -0,0 +1,61 @@ +// The existing wire packs each coefficient into q.bit_length() bits, without +// padding individual coefficients to byte boundaries. No Python mpz tuples are +// needed to import or export these buffers. +#pragma once +#include "rns_ntt.h" +#include +#include +#include + +namespace xtrace_bfv { +inline Word load_word(std::string_view data, std::size_t offset) { + if (offset >= data.size()) return 0; + Word result = 0; + std::memcpy(&result, data.data() + offset, std::min(8, data.size() - offset)); +#if __BYTE_ORDER__ != __ORDER_LITTLE_ENDIAN__ + result = __builtin_bswap64(result); +#endif + return result; +} + +inline Polynomial import_packed(std::string_view data, const Ring& ring) { + ProfileScope timing(Phase::wire_import); + const auto bits = mpz_sizeinbase(ring.q.get_mpz_t(), 2); + if (data.size() != (ring.n * bits + 7) / 8) throw std::invalid_argument("Incorrect packed polynomial byte length"); + const auto words = (bits + 63) / 64; + Polynomial result(ring.n); + for (std::size_t i = 0; i < ring.n; ++i) { + std::array limbs{}; + auto offset = (i * bits / 64) * 8, shift = i * bits % 64; + for (std::size_t j = 0; j < words; ++j) { + limbs[j] = load_word(data, offset + 8 * j) >> shift; + if (shift) limbs[j] |= load_word(data, offset + 8 * j + 8) << (64 - shift); + } + if (bits % 64) limbs[words - 1] &= (Word(1) << (bits % 64)) - 1; + mpz_import(result[i].get_mpz_t(), words, -1, sizeof(Word), 0, 0, limbs.data()); + if (result[i] >= ring.q) throw std::invalid_argument("Noncanonical packed polynomial coefficient"); + } + return result; +} + +inline std::string export_packed(const Polynomial& poly, const mpz_class& q) { + ProfileScope timing(Phase::wire_export); + const auto bits = mpz_sizeinbase(q.get_mpz_t(), 2), words = (bits + 63) / 64; + if (bits > 512) throw std::invalid_argument("Unsupported packed modulus"); + std::vector packed((poly.size() * bits + 63) / 64 + 1); + for (std::size_t i = 0; i < poly.size(); ++i) { + if (poly[i] < 0 || poly[i] >= q) throw std::logic_error("Invalid native output coefficient"); + std::array limbs{}; + mpz_export(limbs.data(), nullptr, -1, sizeof(Word), 0, 0, poly[i].get_mpz_t()); + auto offset = i * bits / 64, shift = i * bits % 64; + for (std::size_t j = 0; j < words; ++j) { + packed[offset + j] |= limbs[j] << shift; + if (shift) packed[offset + j + 1] |= limbs[j] >> (64 - shift); + } + } +#if __BYTE_ORDER__ != __ORDER_LITTLE_ENDIAN__ + for (auto& word : packed) word = __builtin_bswap64(word); +#endif + return std::string(reinterpret_cast(packed.data()), (poly.size() * bits + 7) / 8); +} +} // namespace xtrace_bfv diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h new file mode 100644 index 0000000..0bb1290 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h @@ -0,0 +1,62 @@ +// Opt-in, per-thread native timings. Nested scopes report exclusive time so +// transforms are not also charged to CRT reconstruction or key switching. +#pragma once +#include +#include +#include + +namespace xtrace_bfv { +enum class Phase : std::size_t { + forward_ntt, inverse_ntt, to_residues, crt_reconstruct, crt_exact_fallback, gadget_decompose, + pointwise, scale_round, mod_q, buffers, automorphism, add_sub, wire_import, wire_export, other, count +}; +inline constexpr const char* phase_names[] = { + "forward_ntt", "inverse_ntt", "to_residues", "crt_reconstruct", "crt_exact_fallback", "gadget_decompose", + "pointwise", "scale_round", "mod_q", "buffers", "automorphism", "add_sub", "wire_import", "wire_export", "other" +}; +struct NativeProfile { + using Clock = std::chrono::steady_clock; + std::array(Phase::count)> ns{}, calls{}; + std::size_t current = static_cast(Phase::other); + Clock::time_point last; + void tick() { + auto now = Clock::now(); + ns[current] += std::chrono::duration_cast(now - last).count(); + last = now; + } +}; +inline thread_local NativeProfile* active_profile = nullptr; + +class ProfileScope { + NativeProfile* profile_ = active_profile; + std::size_t parent_ = 0; +public: + explicit ProfileScope(Phase phase) { + if (profile_) { + profile_->tick(); + parent_ = profile_->current; + profile_->current = static_cast(phase); + ++profile_->calls[profile_->current]; + } + } + ~ProfileScope() { + if (!profile_) return; + profile_->tick(); + profile_->current = parent_; + } + ProfileScope(const ProfileScope&) = delete; + ProfileScope& operator=(const ProfileScope&) = delete; +}; + +class ProfileSession { + NativeProfile* previous_profile_ = active_profile; +public: + explicit ProfileSession(NativeProfile& profile) { + active_profile = &profile; + profile.last = NativeProfile::Clock::now(); + } + ~ProfileSession() { active_profile = previous_profile_; } + ProfileSession(const ProfileSession&) = delete; + ProfileSession& operator=(const ProfileSession&) = delete; +}; +} // namespace xtrace_bfv diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h index 70731f8..b3a4543 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h @@ -9,6 +9,7 @@ #include #include #include +#include "profile.h" namespace xtrace_bfv { static_assert(sizeof(unsigned long) == 8 && sizeof(std::size_t) == 8, @@ -38,19 +39,29 @@ struct Multiplier { // most one below floor(input*value/p). The remainder is in [0,2p), even // though both low-word products wrap modulo 2^64. Here p < 2^60. Word multiply(Word input, Word modulus) const { - Word estimate = (Wide(input) * quotient) >> 64; - Word remainder = input * value - estimate * modulus; + Word remainder = multiply_lazy(input, modulus); return remainder >= modulus ? remainder - modulus : remainder; } + + Word multiply_lazy(Word input, Word modulus) const { + Word estimate = (Wide(input) * quotient) >> 64; + return input * value - estimate * modulus; + } }; class PrimeNTT { std::size_t n_; + bool lazy_; + Word reciprocal_low_, reciprocal_high_; std::vector roots_, inverse_roots_, twists_, inverse_scales_; public: const Word modulus; - PrimeNTT(std::size_t n, Word prime) : n_(n), modulus(prime) { + PrimeNTT(std::size_t n, Word prime, bool lazy = false) : n_(n), lazy_(lazy), modulus(prime) { + // For an odd prime, floor((2^128-1)/p) == floor(2^128/p). + Wide reciprocal = ~Wide(0) / prime; + reciprocal_low_ = Word(reciprocal); + reciprocal_high_ = Word(reciprocal >> 64); Word psi = 0; for (Word candidate = 2; !psi; ++candidate) { Word root = power_mod(candidate, (prime - 1) / (2 * n), prime); @@ -83,6 +94,27 @@ class PrimeNTT { // is bit-reversed; keys and query digits use the same order, so no explicit // bit-reversal permutation is needed between forward and inverse NTTs. void forward(std::vector& values) const { + ProfileScope timing(Phase::forward_ntt); + if (lazy_) { + // Internal values stay below 2p. One correction per butterfly; + // canonicalize only at the boundary so gadget accumulation keeps + // its original <2^120 product and 256-term overflow bounds. + const Word twice = 2 * modulus; + for (std::size_t i = 0; i < n_; ++i) + values[i] = twists_[i].multiply_lazy(values[i], modulus); + for (std::size_t length = n_; length > 1; length >>= 1) { + std::size_t half = length / 2, stride = n_ / length; + for (std::size_t start = 0; start < n_; start += length) + for (std::size_t j = 0; j < half; ++j) { + Word a = values[start + j], b = values[start + j + half]; + Word sum = a + b; + values[start + j] = sum >= twice ? sum - twice : sum; + values[start + j + half] = roots_[j * stride].multiply_lazy(a + twice - b, modulus); + } + } + for (auto& value : values) if (value >= modulus) value -= modulus; + return; + } for (std::size_t i = 0; i < n_; ++i) values[i] = twists_[i].multiply(values[i], modulus); for (std::size_t length = n_; length > 1; length >>= 1) { @@ -101,6 +133,27 @@ class PrimeNTT { // Complementary decimation-in-time inverse, fusing 1/N and the inverse // negacyclic twist in the final coefficient pass. void inverse(std::vector& values) const { + ProfileScope timing(Phase::inverse_ntt); + if (lazy_) { + // Inputs to each butterfly are below 4p. Reduce a into [0,2p), + // obtain b*w in [0,2p), then both outputs fit [0,4p). With p<2^60 + // these operations and Shoup's low-word arithmetic cannot overflow. + const Word twice = 2 * modulus; + for (std::size_t length = 2; length <= n_; length <<= 1) { + std::size_t half = length / 2, stride = n_ / length; + for (std::size_t start = 0; start < n_; start += length) + for (std::size_t j = 0; j < half; ++j) { + Word a = values[start + j]; + if (a >= twice) a -= twice; + Word b = inverse_roots_[j * stride].multiply_lazy(values[start + j + half], modulus); + values[start + j] = a + b; + values[start + j + half] = a + twice - b; + } + } + for (std::size_t i = 0; i < n_; ++i) + values[i] = inverse_scales_[i].multiply(values[i], modulus); + return; + } for (std::size_t length = 2; length <= n_; length <<= 1) { std::size_t half = length / 2, stride = n_ / length; for (std::size_t start = 0; start < n_; start += length) { @@ -118,11 +171,37 @@ class PrimeNTT { } std::size_t bytes() const { return 3 * n_ * sizeof(Multiplier); } + + // Barrett reduction for any unsigned 128-bit input. Only the low word of + // floor(x*floor(2^128/p)/2^128) is needed: subtraction wraps mod 2^64, + // while the actual residual lies in [0,2p). The estimate is at most one low. + Word reduce(Wide value) const { + Word low = Word(value), high = Word(value >> 64); + Wide p00 = Wide(low) * reciprocal_low_; + Wide p01 = Wide(low) * reciprocal_high_, p10 = Wide(high) * reciprocal_low_; + Wide middle = (p00 >> 64) + Word(p01) + Word(p10); + Word quotient = high * reciprocal_high_ + Word(p01 >> 64) + Word(p10 >> 64) + Word(middle >> 64); + Word remainder = low - quotient * modulus; + return remainder >= modulus ? remainder - modulus : remainder; + } + + Word remainder(Wide value) const { return lazy_ ? reduce(value) : value % modulus; } + + // Exact floor(y*2^64/p) for y> 64); + if (Word(0) - quotient * modulus >= modulus) ++quotient; + return quotient; + } }; struct CRTBase { mpz_class product, half; std::vector weights; + mpz_class product_mod_q; + std::vector partial_mod_q; + std::vector inverses; }; class Ring { @@ -132,13 +211,14 @@ class Ring { const std::size_t n, coefficient_bytes, digits; const unsigned digit_bits; const mpz_class q; + const bool fast; std::vector transforms; std::size_t switch_prime_count; - Ring(std::size_t n, const mpz_class& q, unsigned bits) + Ring(std::size_t n, const mpz_class& q, unsigned bits, bool fast = false) : n(n), coefficient_bytes((mpz_sizeinbase(q.get_mpz_t(), 2) + 7) / 8), digits(bits ? (mpz_sizeinbase(q.get_mpz_t(), 2) + bits - 1) / bits : 0), - digit_bits(bits), q(q) { + digit_bits(bits), q(q), fast(fast) { if (n < 8 || n > 32768 || (n & (n - 1)) || q < 3 || mpz_sizeinbase(q.get_mpz_t(), 2) > 512 || bits < 1 || bits > mpz_sizeinbase(q.get_mpz_t(), 2)) throw std::invalid_argument("Invalid native BFV ring parameters"); @@ -157,19 +237,24 @@ class Ring { prime -= 2 * n; candidate = prime; } - transforms.emplace_back(n, prime); + transforms.emplace_back(n, prime, fast); product *= prime; prime -= 2 * n; } product = 1; for (std::size_t count = 1; count <= transforms.size(); ++count) { product *= transforms[count - 1].modulus; - CRTBase base{product, product / 2, {}}; + CRTBase base{product, product / 2, {}, product % q, {}, {}}; for (std::size_t j = 0; j < count; ++j) { Word p = transforms[j].modulus; mpz_class partial = product / p; Word residue = mpz_fdiv_ui(partial.get_mpz_t(), p); - base.weights.push_back(partial * power_mod(residue, p - 2, p)); + Word inverse = power_mod(residue, p - 2, p); + base.weights.push_back(partial * inverse); + if (fast) { + base.partial_mod_q.push_back(partial % q); + base.inverses.emplace_back(inverse, p); + } } bases_.push_back(std::move(base)); } @@ -183,7 +268,9 @@ class Ring { } Residues encode(const Polynomial& poly, std::size_t count) const { - Residues result(count, std::vector(n)); + ProfileScope timing(Phase::to_residues); + Residues result; + { ProfileScope allocation(Phase::buffers); result.assign(count, std::vector(n)); } for (std::size_t j = 0; j < count; ++j) { for (std::size_t i = 0; i < n; ++i) result[j][i] = mpz_fdiv_ui(poly[i].get_mpz_t(), transforms[j].modulus); @@ -193,6 +280,7 @@ class Ring { } Polynomial decode(Residues values) const { + ProfileScope timing(Phase::crt_reconstruct); std::size_t count = values.size(); const auto& base = bases_.at(count - 1); for (std::size_t j = 0; j < count; ++j) transforms[j].inverse(values[j]); @@ -211,12 +299,63 @@ class Ring { mpz_class bound = n * *std::max_element(lhs.begin(), lhs.end()) * *std::max_element(rhs.begin(), rhs.end()); std::size_t count = prime_count(bound); auto a = encode(lhs, count), b = encode(rhs, count); + { ProfileScope timing(Phase::pointwise); for (std::size_t j = 0; j < count; ++j) for (std::size_t i = 0; i < n; ++i) - a[j][i] = Wide(a[j][i]) * b[j][i] % transforms[j].modulus; + a[j][i] = transforms[j].remainder(Wide(a[j][i]) * b[j][i]); + } return decode(std::move(a)); } + Polynomial decode_mod_q(Residues values) const { + if (!fast) { + auto result = decode(std::move(values)); + ProfileScope timing(Phase::mod_q); + for (auto& value : result) mpz_mod(value.get_mpz_t(), value.get_mpz_t(), q.get_mpz_t()); + return result; + } + const auto count = values.size(); + const auto& base = bases_.at(count - 1); + for (std::size_t j = 0; j < count; ++j) transforms[j].inverse(values[j]); + Polynomial result(n); + { + ProfileScope timing(Phase::crt_reconstruct); + std::vector terms(count); + for (std::size_t i = 0; i < n; ++i) { + // y_j = residue_j / M_j mod p_j; theta = sum(y_j / p_j). + // The centered integer is sum(y_j*M_j) - round(theta)*M. + // We only need that result modulo q, so use M_j mod q. + // Integer fixed-point bounds certify round(theta); no floats + // or unproved approximate CRT are used. + Wide fraction = 0; + for (std::size_t j = 0; j < count; ++j) { + Word p = transforms[j].modulus; + terms[j] = base.inverses[j].multiply(values[j][i], p); + fraction += transforms[j].fraction(terms[j]); + } + // fraction/2^64 <= theta < (fraction+count)/2^64. + // A rounding boundary inside that interval needs exact CRT. + const Wide half = Wide(1) << 63; + Word nearest = (fraction + half) >> 64; + auto& value = result[i]; + if (nearest != ((fraction + count + half) >> 64)) { + ProfileScope fallback(Phase::crt_exact_fallback); + for (std::size_t j = 0; j < count; ++j) + mpz_addmul_ui(value.get_mpz_t(), base.weights[j].get_mpz_t(), values[j][i]); + value %= base.product; + if (value > base.half) value -= base.product; + } else { + for (std::size_t j = 0; j < count; ++j) + mpz_addmul_ui(value.get_mpz_t(), base.partial_mod_q[j].get_mpz_t(), terms[j]); + mpz_submul_ui(value.get_mpz_t(), base.product_mod_q.get_mpz_t(), nearest); + } + } + } + { ProfileScope timing(Phase::mod_q); + for (auto& value : result) mpz_mod(value.get_mpz_t(), value.get_mpz_t(), q.get_mpz_t()); } + return result; + } + std::array multiply( const std::array& lhs, const std::array& rhs, Word t, bool square) const { @@ -225,21 +364,24 @@ class Ring { Residues b0, b1; if (!square) { b0 = encode(rhs[0], count); b1 = encode(rhs[1], count); } std::array output; - for (auto& component : output) component.assign(count, std::vector(n)); + { ProfileScope allocation(Phase::buffers); + for (auto& component : output) component.assign(count, std::vector(n)); } + { ProfileScope timing(Phase::pointwise); for (std::size_t j = 0; j < count; ++j) { - Word p = transforms[j].modulus; for (std::size_t i = 0; i < n; ++i) { Word x0 = a0[j][i], x1 = a1[j][i]; Word y0 = square ? x0 : b0[j][i], y1 = square ? x1 : b1[j][i]; - output[0][j][i] = Wide(x0) * y0 % p; - output[1][j][i] = (Wide(x0) * y1 + Wide(x1) * y0) % p; - output[2][j][i] = Wide(x1) * y1 % p; + output[0][j][i] = transforms[j].remainder(Wide(x0) * y0); + output[1][j][i] = transforms[j].remainder(Wide(x0) * y1 + Wide(x1) * y0); + output[2][j][i] = transforms[j].remainder(Wide(x1) * y1); } } + } std::array result; mpz_class denominator = 2 * q; for (std::size_t k = 0; k < 3; ++k) { result[k] = decode(std::move(output[k])); + ProfileScope timing(Phase::scale_round); for (auto& value : result[k]) { // Preserve the reference's exact signed scale-and-round BEFORE // reducing mod q: floor((2*t*integer_product + q) / (2*q)). @@ -275,10 +417,12 @@ class SwitchKey { std::array apply(const Polynomial& poly) const { const auto& r = *ring; std::array output; - for (auto& component : output) component.assign(r.switch_prime_count, std::vector(r.n)); + { ProfileScope allocation(Phase::buffers); + for (auto& component : output) component.assign(r.switch_prime_count, std::vector(r.n)); } // Decompose once, then reuse each integer digit across auxiliary primes. std::vector decomposition; std::vector> small_digits; + { ProfileScope timing(Phase::gadget_decompose); #if GMP_NUMB_BITS == 64 if (r.digit_bits <= 60) { small_digits.assign(r.digits, std::vector(r.n)); @@ -302,6 +446,8 @@ class SwitchKey { mpz_fdiv_r_2exp(value.get_mpz_t(), value.get_mpz_t(), r.digit_bits); } } + } + { ProfileScope timing(Phase::pointwise); for (std::size_t j = 0; j < r.switch_prime_count; ++j) { Word p = r.transforms[j].modulus; std::array, 2> accum{std::vector(r.n), std::vector(r.n)}; @@ -325,15 +471,15 @@ class SwitchKey { // Longer gadgets (e.g. 512 one-bit digits) need intermediate reduction. if ((digit + 1) % 256 == 0) for (auto& component : accum) - for (auto& value : component) value %= p; + for (auto& value : component) value = r.transforms[j].remainder(value); } for (std::size_t k = 0; k < 2; ++k) - for (std::size_t i = 0; i < r.n; ++i) output[k][j][i] = accum[k][i] % p; + for (std::size_t i = 0; i < r.n; ++i) output[k][j][i] = r.transforms[j].remainder(accum[k][i]); + } } std::array result; for (std::size_t k = 0; k < 2; ++k) { - result[k] = r.decode(std::move(output[k])); - for (auto& value : result[k]) mpz_mod(value.get_mpz_t(), value.get_mpz_t(), r.q.get_mpz_t()); + result[k] = r.decode_mod_q(std::move(output[k])); } return result; } @@ -342,6 +488,7 @@ class SwitchKey { }; inline void add_inplace(Ciphertext& lhs, const Ciphertext& rhs, const Ring& ring) { + ProfileScope timing(Phase::add_sub); for (std::size_t k = 0; k < 2; ++k) for (std::size_t i = 0; i < ring.n; ++i) { lhs[k][i] += rhs[k][i]; @@ -352,6 +499,7 @@ inline void add_inplace(Ciphertext& lhs, const Ciphertext& rhs, const Ring& ring inline Ciphertext rotate(const Ciphertext& ciphertext, Word exponent, const SwitchKey& key) { const auto& r = *key.ring; Ciphertext permuted{Polynomial(r.n), Polynomial(r.n)}; + { ProfileScope timing(Phase::automorphism); for (std::size_t k = 0; k < 2; ++k) for (std::size_t i = 0; i < r.n; ++i) { std::size_t index = (i * exponent) % (2 * r.n); @@ -359,6 +507,7 @@ inline Ciphertext rotate(const Ciphertext& ciphertext, Word exponent, const Swit value = ciphertext[k][i]; if (index >= r.n && value != 0) value = r.q - value; } + } auto switched = key.apply(permuted[1]); for (std::size_t i = 0; i < r.n; ++i) { switched[0][i] += permuted[0][i]; @@ -375,11 +524,13 @@ inline Ciphertext distance_tile( const std::vector>>& rotations, const Polynomial& mask, Word t) { const auto& r = *relin.ring; + { ProfileScope timing(Phase::add_sub); for (std::size_t k = 0; k < 2; ++k) for (std::size_t i = 0; i < r.n; ++i) { query[k][i] -= tile[k][i]; if (query[k][i] < 0) query[k][i] += r.q; } + } auto quadratic = r.multiply(query, query, t, true); auto distance = relin.apply(quadratic[2]); Ciphertext linear{std::move(quadratic[0]), std::move(quadratic[1])}; @@ -390,6 +541,7 @@ inline Ciphertext distance_tile( } for (auto& component : distance) { component = r.product(component, mask); + ProfileScope timing(Phase::mod_q); for (auto& value : component) mpz_mod(value.get_mpz_t(), value.get_mpz_t(), r.q.get_mpz_t()); } return distance; diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py index 86824e6..ee358e4 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py @@ -21,7 +21,7 @@ BFVSwitchKey, ) -BFVServerBackend = Literal["optimized", "reference", "rns"] +BFVServerBackend = Literal["optimized", "reference", "rns", "native"] @dataclass(frozen=True) @@ -76,11 +76,12 @@ class BFVEvaluator: ``optimized`` lazily caches packed evaluation keys and fuses gadget products; ``reference`` delegates to the original BFV routines. ``rns`` uses the - optional native C++ RNS/NTT kernels. All backends return identical + optional native C++ RNS/NTT kernels. ``native`` enables their faster exact + arithmetic and lets BFVClient run complete searches inside C++. All backends return identical ciphertexts. Reuse one evaluator across queries to amortize preparation. :param pk: Valid public key from key generation or public-key deserialization. - :param backend: ``optimized`` (default), ``reference``, or ``rns``. + :param backend: ``optimized`` (default), ``reference``, ``rns``, or ``native``. The key dictionaries are snapshotted; immutable polynomial tuples are shared. Construct a new evaluator when changing keys. The cache is bounded by the @@ -88,13 +89,17 @@ class BFVEvaluator: """ def __init__(self, pk: BFVPublicKey, backend: BFVServerBackend = "optimized") -> None: - if backend not in ("optimized", "reference", "rns"): - raise ValueError("server_backend must be 'optimized', 'reference', or 'rns'") + if backend not in ("optimized", "reference", "rns", "native"): + raise ValueError("server_backend must be 'optimized', 'reference', 'rns', or 'native'") self.backend = backend self._pk: BFVPublicKey = {**pk, "galois_keys": dict(pk["galois_keys"])} # Exponent zero identifies relinearization; Galois exponents are odd. self._switch_keys: dict[int, _PackedSwitchKey] = {} - self._rns = BFVRNSArithmetic(self._pk) if backend == "rns" else None + self._rns = ( + BFVRNSArithmetic(self._pk, fast=backend == "native") + if backend in ("rns", "native") + else None + ) def cache_info(self) -> dict[str, Any]: """Report prepared public-key payloads; excludes peak temporary allocations.""" diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py new file mode 100644 index 0000000..bc08864 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py @@ -0,0 +1,125 @@ +"""Complete C++ public-key evaluation behind the existing BFV wire interface. + +Only the framing and packed byte buffers cross Python. Polynomial objects, +rotations, the merge tree and response compaction stay inside the native server. +Key generation, encryption and decryption use the existing BFV implementation. +""" + +from collections.abc import Sequence +from numbers import Integral +from typing import Any + +from xtrace_sdk.x_vec.crypto.encryption.bfv import _coefficient_modulus +from xtrace_sdk.x_vec.crypto.encryption.bfv_rns import BFVRNSArithmetic +from xtrace_sdk.x_vec.utils.xtrace_types import EncryptedVector + + +class BFVNativeServer: + """Immutable native plan with public evaluation keys and a prepared slot mask.""" + + def __init__( + self, arithmetic: BFVRNSArithmetic, padded_embed_len: int, response_modulus_bits: int + ) -> None: + self.arithmetic = arithmetic + self.padded_embed_len = padded_embed_len + self.response_modulus_bits = response_modulus_bits + pk = arithmetic._pk + if ( + type(padded_embed_len) is not int + or not 1 <= padded_embed_len <= arithmetic.n // 2 + or padded_embed_len & (padded_embed_len - 1) + ): + raise ValueError("Invalid native server padded dimension") + if ( + type(response_modulus_bits) is not int + or not pk["params"].plain_modulus.bit_length() + 8 + <= response_modulus_bits + <= pk["q"].bit_length() + ): + raise ValueError("Invalid target modulus bit length") + self._target = _coefficient_modulus(response_modulus_bits) + self._key_id = int(pk["key_id"], 16) + self._packed_bytes = (arithmetic.n * pk["q"].bit_length() + 7) // 8 + self._capacity = arithmetic.n // padded_embed_len + exponents = { + pow(3, sign * (self._capacity // 2) * (1 << i) % (arithmetic.n // 2), 2 * arithmetic.n) + for sign in (1, -1) + for i in range(padded_embed_len.bit_length() - 1) + } - {1} + if not exponents.issubset(pk["galois_keys"]): + raise ValueError("Public key lacks native server rotation keys") + self._server = arithmetic._native.create_server( + arithmetic._prepare_key(0, pk["relin_key"]), + tuple((g, arithmetic._prepare_key(g, pk["galois_keys"][g])) for g in sorted(exponents)), + padded_embed_len, + pk["params"].plain_modulus, + format(self._target, "x"), + ) + + def _wire(self, values: Sequence[int | bytes]) -> tuple[bytes, bytes]: + if len(values) != 8: + raise ValueError("Native server requires two-component BFV ciphertext framing") + if any(not isinstance(v, (Integral, bytes)) for v in values): + raise ValueError("Ciphertext entries must be nonnegative integers or bytes") + data = [int.from_bytes(v, "little") if isinstance(v, bytes) else int(v) for v in values] + if any(v < 0 for v in data): + raise ValueError("Ciphertext entries must be nonnegative") + expected = [ + 0x58424656, + 1, + self.arithmetic.n, + int(self.arithmetic._pk["q"]), + self._key_id, + 2, + ] + if data[:6] != expected: + raise ValueError("Incompatible BFV ciphertext header, modulus or key") + if any(v.bit_length() > self._packed_bytes * 8 for v in data[6:]): + raise ValueError("Packed polynomial exceeds N coefficients") + # Coefficient bounds are checked in C++ while importing the bit-packed + # buffers. Do not construct or validate N Python integers per component. + return data[6].to_bytes(self._packed_bytes, "little"), data[7].to_bytes( + self._packed_bytes, "little" + ) + + def search( + self, + query: Sequence[int | bytes], + index: Sequence[Sequence[int | bytes]], + vector_count: int, + *, + compact: bool = True, + profile: dict[str, Any] | None = None, + ) -> list[EncryptedVector]: + """Search with unchanged wire bytes; optionally collect native phase timings. + + Profiling is opt-in and should be excluded from performance samples. + """ + if ( + type(vector_count) is not int + or vector_count < 0 + or len(index) != (vector_count + self._capacity - 1) // self._capacity + ): + raise ValueError("vector_count does not match the packed ciphertext count") + args = ( + self._server, + self._wire(query), + tuple(self._wire(v) for v in index), + vector_count, + compact, + ) + native = self.arithmetic._native + if profile is None: + packed = native.packed_search(*args) + else: + packed, timings = native.profile_packed_search(*args) + profile.update(timings) + q = self._target if compact else self.arithmetic._pk["q"] + header = [0x58424656, 1, self.arithmetic.n, int(q), self._key_id, 2] + return [ + header + [int.from_bytes(b, "little"), int.from_bytes(a, "little")] for b, a in packed + ] + + def cache_bytes(self) -> int: + """Payload of prepared plaintext tables and mask, excluding evaluation keys.""" + return self.arithmetic._native.server_bytes(self._server) diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py index 5609445..f358aaf 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py @@ -15,7 +15,7 @@ class BFVRNSArithmetic: """Bind optional C++ kernels to one validated BFV public-key context.""" - def __init__(self, pk: BFVPublicKey) -> None: + def __init__(self, pk: BFVPublicKey, *, fast: bool = False) -> None: try: from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns except ImportError as error: @@ -24,14 +24,16 @@ def __init__(self, pk: BFVPublicKey) -> None: "make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON=/path/to/python; " "or select server_backend='optimized' for the Python/GMP evaluator." ) from error - if _bfv_rns.ABI_VERSION != 1: + if _bfv_rns.ABI_VERSION != 2: raise ImportError("BFV RNS CPU extension has an incompatible ABI; rebuild it") self._native = _bfv_rns self._pk = pk params = pk["params"] self.n = params.poly_modulus_degree self.width = (pk["q"].bit_length() + 7) // 8 - self._ring = _bfv_rns.create_ring(self.n, format(pk["q"], "x"), params.decomposition_bits) + self._ring = _bfv_rns.create_ring( + self.n, format(pk["q"], "x"), params.decomposition_bits, fast + ) self._keys: dict[int, object] = {} def _pack(self, poly: BFVPolynomial) -> bytes: diff --git a/tests/x_vec/native/run_binding_sanitizers.py b/tests/x_vec/native/run_binding_sanitizers.py new file mode 100644 index 0000000..1cf768c --- /dev/null +++ b/tests/x_vec/native/run_binding_sanitizers.py @@ -0,0 +1,25 @@ +"""Load an ASan/UBSan extension in place of the regular binary, then run its tests. + +Usage: python tests/x_vec/native/run_binding_sanitizers.py /tmp/bfv_sanitized.so +Start Python with the sanitizer and C++ runtimes in LD_PRELOAD (see CI). +""" + +import importlib.util +from pathlib import Path +import sys + +import xtrace_sdk.x_vec.crypto.bfv_cpu_ext as package + +name = "xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns" +library = Path(sys.argv[1]).resolve() +spec = importlib.util.spec_from_file_location(name, library) +assert spec is not None and spec.loader is not None +module = importlib.util.module_from_spec(spec) +sys.modules[name] = module +spec.loader.exec_module(module) +package._bfv_rns = module +assert Path(module.__file__).resolve() == library + +import pytest # noqa: E402 + +raise SystemExit(pytest.main(["tests/x_vec/test_bfv_native.py", "-q"])) diff --git a/tests/x_vec/native/test_bfv_rns.cpp b/tests/x_vec/native/test_bfv_rns.cpp index 40afc8d..e773247 100644 --- a/tests/x_vec/native/test_bfv_rns.cpp +++ b/tests/x_vec/native/test_bfv_rns.cpp @@ -1,5 +1,6 @@ // Standalone kernel oracles; also run under address/undefined-behavior sanitizers. #include "rns_ntt.h" +#include "packed_wire.h" #include #include #include @@ -19,11 +20,12 @@ int main() { std::mt19937_64 random(1337); gmp_randclass big_random(gmp_randinit_default); big_random.seed(1337); + for (bool fast : {false, true}) { for (unsigned q_bits : {31, 100, 180, 512}) { for (unsigned bits : {1, 15, 30, 59, 60, 61, 127}) { if (bits > q_bits) continue; mpz_class q = (mpz_class(1) << q_bits) - 1; - auto ring = std::make_shared(8, q, bits); + auto ring = std::make_shared(8, q, bits, fast); Polynomial a(8), b(8); for (auto& value : a) value = big_random.get_z_range(q); for (auto& value : b) value = big_random.get_z_range(q); @@ -31,8 +33,13 @@ int main() { Word p = transform.modulus; for (int i = 0; i < 300; ++i) { Word multiplier = i ? random() % p : p - 1; - Word input = i ? random() % (2 * p) : 2 * p - 1; + Word input = i ? random() % (4 * p) : 4 * p - 1; assert(Multiplier(multiplier, p).multiply(input, p) == Wide(input) * multiplier % p); + assert(Multiplier(multiplier, p).multiply_lazy(input, p) < 2 * p); + Wide wide = i ? (Wide(random()) << 64) | random() : ~Wide(0); + assert(transform.reduce(wide) == wide % p); + Word y = i ? random() % p : p - 1; + assert(transform.fraction(y) == (Wide(y) << 64) / p); } std::vector values(8); for (auto& value : values) value = random() % p; @@ -73,7 +80,7 @@ int main() { // Exercise NTT lengths, both signs of X^N=-1, and large transform tables. for (std::size_t n : {16, 64, 8192, 32768}) { mpz_class q = (mpz_class(1) << 180) - 1; - Ring ring(n, q, 30); + Ring ring(n, q, 30, fast); Polynomial a(n), b(n); a[n - 1] = q - 1; b[1] = q - 1; @@ -81,5 +88,33 @@ int main() { assert(product[0] == -(q - 1) * (q - 1)); for (std::size_t i = 1; i < n; ++i) assert(product[i] == 0); } + } + // Exercise the certified CRT interval precisely where an approximate + // rounding decision is insufficient: the two sides of +/-M/2. + for (unsigned bits : {31, 64, 100, 180, 512}) { + mpz_class q = (mpz_class(1) << bits) - 1; + Ring ring(8, q, 15, true); + mpz_class product = 1; + for (std::size_t count = 1; count <= ring.transforms.size(); ++count) { + product *= ring.transforms[count - 1].modulus; + mpz_class half = product / 2; + Polynomial values{0, 1, -1, half, half + 1, -half, -half - 1, product - 1}; + NativeProfile profile; + Polynomial result; + { ProfileSession session(profile); result = ring.decode_mod_q(ring.encode(values, count)); } + if (count >= 2) assert(profile.calls[static_cast(Phase::crt_exact_fallback)] > 0); + for (std::size_t i = 0; i < values.size(); ++i) { + mpz_class expected; + mpz_mod(expected.get_mpz_t(), values[i].get_mpz_t(), product.get_mpz_t()); + if (expected > half) expected -= product; + mpz_mod(expected.get_mpz_t(), expected.get_mpz_t(), q.get_mpz_t()); + assert(result[i] == expected); + } + } + Polynomial values(8); + for (auto& c : values) c = big_random.get_z_range(q); + values[0] = 0; values[7] = q - 1; + assert(import_packed(export_packed(values, q), ring) == values); + } std::cout << "Native NTT, Shoup, signed CRT and gadget oracles passed\n"; } diff --git a/tests/x_vec/test_bfv_client.py b/tests/x_vec/test_bfv_client.py index c0d0bdb..8ec352f 100644 --- a/tests/x_vec/test_bfv_client.py +++ b/tests/x_vec/test_bfv_client.py @@ -66,9 +66,9 @@ def test_base_interface_and_individual_batch() -> None: @pytest.mark.parametrize("dimension", [1, 3, 9, 16]) -@pytest.mark.parametrize("backend", ["optimized", "rns"]) +@pytest.mark.parametrize("backend", ["optimized", "rns", "native"]) def test_packed_tile_row_and_response_boundaries(dimension: int, backend: str) -> None: - if backend == "rns": + if backend in ("rns", "native"): pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") client = small_client(dimension) server = public_server(client, backend) @@ -113,9 +113,9 @@ def test_partial_tile_padding_is_zero_and_compaction_preserves_distances() -> No assert sum(v.bit_length() for v in compact) < sum(v.bit_length() for v in response[0]) / 2 -@pytest.mark.parametrize("backend", ["optimized", "rns"]) +@pytest.mark.parametrize("backend", ["optimized", "rns", "native"]) def test_public_only_evaluator_in_separate_process(tmp_path: Path, backend: str) -> None: - if backend == "rns": + if backend in ("rns", "native"): pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") client = small_client(9) vectors, query, expected = data(35, 9) diff --git a/tests/x_vec/test_bfv_evaluator.py b/tests/x_vec/test_bfv_evaluator.py index ef37e3e..d7baf5d 100644 --- a/tests/x_vec/test_bfv_evaluator.py +++ b/tests/x_vec/test_bfv_evaluator.py @@ -55,9 +55,9 @@ def test_fused_switch_against_schoolbook(q_bits: int, digit_bits: int) -> None: assert actual == tuple(tuple(value % q for value in row) for row in expected) -@pytest.mark.parametrize("backend", ["optimized", "rns"]) +@pytest.mark.parametrize("backend", ["optimized", "rns", "native"]) def test_operations_identical_to_reference(keys: BFVKeyPair, backend: str) -> None: - if backend == "rns": + if backend in ("rns", "native"): pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") pk, rng = keys["pk"], random.Random(20) evaluator = BFVEvaluator(pk, backend) @@ -82,11 +82,11 @@ def test_operations_identical_to_reference(keys: BFVKeyPair, backend: str) -> No assert evaluator.subtract(compact, compact) == BFV.subtract(compact, compact, pk) -@pytest.mark.parametrize("backend", ["optimized", "rns"]) +@pytest.mark.parametrize("backend", ["optimized", "rns", "native"]) def test_reject_malformed_ciphertexts_and_keep_integral_support( keys: BFVKeyPair, backend: str ) -> None: - if backend == "rns": + if backend in ("rns", "native"): pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") pk = keys["pk"] evaluator = BFVEvaluator(pk, backend) diff --git a/tests/x_vec/test_bfv_native.py b/tests/x_vec/test_bfv_native.py new file mode 100644 index 0000000..7f95a15 --- /dev/null +++ b/tests/x_vec/test_bfv_native.py @@ -0,0 +1,166 @@ +"""Complete native server: public-only evaluation, wire boundaries and profiling.""" + +import gc +import json +from concurrent.futures import ThreadPoolExecutor + +import pytest + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.encryption.bfv_rns import BFVRNSArithmetic + +native = pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + + +@pytest.fixture() +def client() -> BFVClient: + return BFVClient(3, 16, 97, 100, 15, response_modulus_bits=40, server_backend="native") + + +def test_native_individual_wire_bytes_and_backend_changes(client: BFVClient) -> None: + query, other = client.encrypt_vec_one([0, 1, 0]), client.encrypt_vec_one([1, 0, 1]) + responses = [] + for backend in ("native", "rns", "reference", "optimized", "native"): + client.server_backend = backend + responses.append(client.encode_hamming_server(query, other)) + assert all(response == responses[0] for response in responses) + assert client.decode_hamming_client_one(responses[0]) == 3 + byte_query = [v.to_bytes((v.bit_length() + 7) // 8, "little") for v in query] + assert client.encode_hamming_server(byte_query, other) == responses[0] + client.response_modulus_bits = 45 + changed = client.encode_hamming_server(query, other) + assert changed[3].bit_length() == 45 + assert client.decode_hamming_client_one(changed) == 3 + client.response_modulus_bits = 10 + with pytest.raises(ValueError, match="target modulus"): + client.encode_hamming_server(query, other) + + +def test_complete_search_skips_python_polynomial_conversions( + client: BFVClient, monkeypatch +) -> None: + from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV + + query = client.encrypt_vec_one([0, 1, 0]) + vectors = [[0, 1, 0], [1, 0, 1], [0, 0, 0]] * 13 + index = client.encrypt_vec_packed(vectors) + server = BFVClient(skip_key_gen=True, server_backend="native") + server.load_config(json.loads(client.stringify_config())) + server.load_stringified_keys(client.stringify_pk()) + assert server.keys is None + + def forbidden(*args, **kwargs): + raise AssertionError("Server must keep intermediate polynomials in C++") + + with monkeypatch.context() as patch: + for name in ( + "ciphertext_from_ints", + "ciphertext_to_ints", + "batch_encode", + "modulus_switch", + ): + patch.setattr(BFV, name, forbidden) + response = server.encode_hamming_server_packed(query, index, len(vectors)) + assert client.decode_hamming_client_packed(response, len(vectors)) == [0, 3, 1] * 13 + old_plan = server._native() + server.load_stringified_keys(client.stringify_pk()) + assert server._native() is not old_plan + assert server.encode_hamming_server_packed(query, index, len(vectors)) == response + + +def test_profile_is_transparent_and_thread_local(client: BFVClient) -> None: + query = client.encrypt_vec_one([0, 1, 0]) + index = client.encrypt_vec_packed([[0, 1, 0], [1, 0, 1]] * 5) + evaluator = client._native() + expected = evaluator.search(query, index, 10) + + def evaluate(_: int): + profile = {} + result = evaluator.search(query, index, 10, profile=profile) + assert result == expected + assert sum(v["seconds"] for v in profile.values()) > 0 + assert all(v["seconds"] >= 0 and v["calls"] >= 0 for v in profile.values()) + assert profile["wire_import"]["calls"] == 2 * (1 + len(index)) + assert profile["wire_export"]["calls"] == 2 + return {name: value["calls"] for name, value in profile.items()} + + with ThreadPoolExecutor(max_workers=2) as pool: + counts = list(pool.map(evaluate, range(6))) + assert all(count == counts[0] for count in counts) + assert evaluator.search(query, index, 10) == expected + with pytest.raises(ZeroDivisionError): + native.profile_call(lambda: 1 / 0) + result, profile = native.profile_call(lambda: evaluator.search(query, index, 10)) + assert result == expected + assert profile["wire_import"]["calls"] == 2 * (1 + len(index)) + + +def test_native_rejects_malformed_wire(client: BFVClient) -> None: + query = client.encrypt_vec_one([0, 1, 0]) + q = int(client._pk()["q"]) + changes = [ + (0, 0), + (1, 2), + (2, 32), + (3, q - 2), + (4, query[4] + 1), + (5, 3), + (6, -1), + (6, 0.5), + (6, 1 << 1600), + (6, q << 1500), + ] + for offset, value in changes: + malformed = query[:] + malformed[offset] = value + with pytest.raises(ValueError): + client.encode_hamming_server_packed(query, [malformed], 1) + for count in (-1, True, 1.0, 5): + with pytest.raises(ValueError): + client.encode_hamming_server_packed(query, [query], count) + with pytest.raises(ValueError): + client.encode_hamming_server(query[:-1], query) + other = BFVClient(3, 16, 97, 100, 15, response_modulus_bits=40) + with pytest.raises(ValueError, match="key"): + client.encode_hamming_server(query, other.encrypt_vec_one([0, 1, 0])) + + +def test_private_native_boundary_and_plan_lifetime(client: BFVClient) -> None: + evaluator = client._native() + arithmetic = evaluator.arithmetic + pk = client._pk() + relin = arithmetic._prepare_key(0, pk["relin_key"]) + keys = tuple((g, arithmetic._prepare_key(g, key)) for g, key in pk["galois_keys"].items()) + for padded, t, target in ( + (0, 97, "ffff"), + (3, 97, "ffff"), + (32, 97, "ffff"), + (4, 33, "ffff"), + (4, 65, "ffff"), + (4, 97, "0"), + (4, 97, "!"), + ): + with pytest.raises(ValueError): + native.create_server(relin, keys, padded, t, target) + with pytest.raises(ValueError, match="Missing"): + native.create_server(relin, (), 4, 97, "ffff") + with pytest.raises(ValueError, match="Duplicate"): + native.create_server(relin, keys + keys[:1], 4, 97, "ffff") + other = BFVRNSArithmetic(pk) + other_key = other._prepare_key(0, pk["relin_key"]) + with pytest.raises(ValueError, match="another native ring"): + native.create_server(relin, ((3, other_key),), 4, 97, "ffff") + wire = evaluator._wire(client.encrypt_vec_one([0, 1, 0])) + plan = evaluator._server + for index, count in (((), 1), ((wire,), -1), ((wire,), 1 << 65), (((b"", b""),), 1)): + with pytest.raises((ValueError, OverflowError)): + native.packed_search(plan, wire, index, count, True) + # Use a fresh ring with no BFVClient owner to test the plan's ownership. + owned = BFVRNSArithmetic(pk) + relin = owned._prepare_key(0, pk["relin_key"]) + keys = tuple((g, owned._prepare_key(g, key)) for g, key in pk["galois_keys"].items()) + plan = native.create_server(relin, keys, 4, 97, format(evaluator._target, "x")) + expected = native.packed_search(plan, wire, (wire,), 1, True) + del owned, relin, keys + gc.collect() + assert native.packed_search(plan, wire, (wire,), 1, True) == expected diff --git a/tests/x_vec/test_bfv_rns.py b/tests/x_vec/test_bfv_rns.py index 1ff800f..d2d2b6a 100644 --- a/tests/x_vec/test_bfv_rns.py +++ b/tests/x_vec/test_bfv_rns.py @@ -43,11 +43,12 @@ def schoolbook(a: tuple, b: tuple) -> tuple: (100, 100), ], ) -def test_rns_switch_and_products_against_integer_oracle(q_bits: int, bits: int) -> None: +@pytest.mark.parametrize("fast", [False, True]) +def test_rns_switch_and_products_against_integer_oracle(q_bits: int, bits: int, fast: bool) -> None: keys = BFV.key_gen(8, 17, q_bits, bits) pk, rng = keys["pk"], random.Random(bits) q = pk["q"] - arithmetic = BFVRNSArithmetic(pk) + arithmetic = BFVRNSArithmetic(pk, fast=fast) maximum = (q - 1,) * 8 random_poly = tuple(mpz(rng.randrange(q)) for _ in range(8)) inputs = [(mpz(0),) * 8, maximum, (mpz(0),) * 7 + (q - 1,), random_poly] From e5c075106c91d52d927af65e3d324f516c8c53b0 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Thu, 10 Sep 2026 22:31:20 -0400 Subject: [PATCH 06/12] Keep BFV server ciphertexts in RNS across key switches and rotations --- .github/workflows/ci.yml | 4 +- benchmarks/bfv_client_matrix.py | 12 +- benchmarks/bfv_server.py | 32 +- .../results/native_bfv_residue_1024.json | 407 ++++++++++++++++++ benchmarks/results/native_bfv_residue_32.json | 407 ++++++++++++++++++ ...native_bfv_residue_prime_control_1024.json | 221 ++++++++++ docs/index.rst | 1 + docs/research/native-bfv-residue.md | 272 ++++++++++++ docs/research/native-bfv-server.md | 17 +- src/xtrace_sdk/x_vec/crypto/bfv_client.py | 35 +- .../x_vec/crypto/bfv_cpu_ext/Makefile | 2 +- .../x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi | 4 +- .../x_vec/crypto/bfv_cpu_ext/bfv_residue.h | 238 ++++++++++ .../x_vec/crypto/bfv_cpu_ext/bfv_server.h | 85 ++-- .../x_vec/crypto/bfv_cpu_ext/bindings.cpp | 19 +- .../x_vec/crypto/bfv_cpu_ext/profile.h | 4 +- .../x_vec/crypto/bfv_cpu_ext/rns_ntt.h | 60 ++- src/xtrace_sdk/x_vec/crypto/encryption/bfv.py | 75 +++- .../x_vec/crypto/encryption/bfv_evaluator.py | 22 +- .../x_vec/crypto/encryption/bfv_native.py | 2 +- .../x_vec/crypto/encryption/bfv_rns.py | 15 +- src/xtrace_sdk/x_vec/utils/xtrace_types.py | 1 + tests/x_vec/native/run_binding_sanitizers.py | 4 +- tests/x_vec/native/test_bfv_rns.cpp | 65 ++- tests/x_vec/test_bfv_native.py | 25 +- tests/x_vec/test_bfv_residue.py | 109 +++++ 26 files changed, 2014 insertions(+), 124 deletions(-) create mode 100644 benchmarks/results/native_bfv_residue_1024.json create mode 100644 benchmarks/results/native_bfv_residue_32.json create mode 100644 benchmarks/results/native_bfv_residue_prime_control_1024.json create mode 100644 docs/research/native-bfv-residue.md create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h create mode 100644 tests/x_vec/test_bfv_residue.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9d2ebea..756b860 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -67,8 +67,8 @@ jobs: sudo apt-get update sudo apt-get install -y libgmp-dev make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" - uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 2' - uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_client.py + uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 3' + uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py - name: BFV native arithmetic sanitizers run: | diff --git a/benchmarks/bfv_client_matrix.py b/benchmarks/bfv_client_matrix.py index 9805683..a4eec1d 100644 --- a/benchmarks/bfv_client_matrix.py +++ b/benchmarks/bfv_client_matrix.py @@ -105,6 +105,7 @@ def benchmark( coeff_modulus_bits=args.coeff_modulus_bits, decomposition_bits=args.decomposition_bits, response_modulus_bits=args.response_modulus_bits, + rns_modulus=args.rns_modulus, device="cpu", server_backend=args.server_backend, ) @@ -237,7 +238,9 @@ def main() -> None: parser.add_argument("--response-modulus-bits", type=int, default=50) parser.add_argument("--no-compact", action="store_true") parser.add_argument( - "--server-backend", choices=("optimized", "reference", "rns", "native"), default="optimized" + "--server-backend", + choices=("optimized", "reference", "rns", "native", "residue"), + default="optimized", ) parser.add_argument("--seed", type=int, default=1337) parser.add_argument( @@ -252,7 +255,14 @@ def main() -> None: help="Comma-separated: " + ", ".join(VARIANTS), ) parser.add_argument("--json-out", type=Path) + parser.add_argument( + "--rns-modulus", + action="store_true", + help="Use fresh BFV keys with a product of 60-bit NTT primes", + ) args = parser.parse_args() + if args.server_backend == "residue" and not args.rns_modulus: + parser.error("residue requires --rns-modulus") variants = [v.strip() for v in args.variants.split(",")] if args.num_vectors < 1 or args.embed_len < 1 or args.repeats < 1: parser.error("num-vectors, embed-len and repeats must be positive") diff --git a/benchmarks/bfv_server.py b/benchmarks/bfv_server.py index de3bb77..dcec946 100644 --- a/benchmarks/bfv_server.py +++ b/benchmarks/bfv_server.py @@ -26,7 +26,7 @@ from bfv_client_matrix import REPO_ROOT, make_data, packet, time_call, unpack_packet from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient -from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _rns_coefficient_primes from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVServerBackend @@ -38,6 +38,11 @@ def main() -> None: parser.add_argument("--plain-modulus", type=int, default=65537) parser.add_argument("--coeff-modulus-bits", type=int, default=180) parser.add_argument("--decomposition-bits", type=int, default=30) + parser.add_argument( + "--rns-modulus", + action="store_true", + help="Generate fresh keys with a product of 60-bit NTT primes; required by residue", + ) parser.add_argument("--response-modulus-bits", type=int, default=50) parser.add_argument("--seed", type=int, default=1337) parser.add_argument( @@ -50,7 +55,7 @@ def main() -> None: parser.add_argument( "--backends", default="reference,optimized", - help="Comma-separated native backends: reference,optimized,rns,native", + help="Comma-separated native backends: reference,optimized,rns,native,residue", ) parser.add_argument( "--seal", @@ -74,13 +79,15 @@ def main() -> None: if ( not names or len(set(names)) != len(names) - or any(b not in ("reference", "optimized", "rns", "native") for b in names) + or any(b not in ("reference", "optimized", "rns", "native", "residue") for b in names) ): parser.error("backends must be distinct native backend names") if args.seal and (args.poly_modulus_degree != 8192 or args.plain_modulus != 65537): parser.error("the SEAL comparison requires N=8192 and t=65537") - if args.native_profile and not any(b in ("rns", "native") for b in names): - parser.error("native-profile requires the rns or native backend") + if args.native_profile and not any(b in ("rns", "native", "residue") for b in names): + parser.error("native-profile requires the rns, native, or residue backend") + if "residue" in names and not args.rns_modulus: + parser.error("residue requires --rns-modulus (fresh keys and index)") backends = tuple(cast(BFVServerBackend, name) for name in names) vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) print("Generating one key set and encrypting the shared input", file=sys.stderr, flush=True) @@ -93,6 +100,7 @@ def main() -> None: coeff_modulus_bits=args.coeff_modulus_bits, decomposition_bits=args.decomposition_bits, response_modulus_bits=args.response_modulus_bits, + rns_modulus=args.rns_modulus, device="cpu", ) ) @@ -209,7 +217,7 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns for backend in backends: - if backend in ("rns", "native"): + if backend in ("rns", "native", "residue"): response, phases = _bfv_rns.profile_call(partial(evaluate, backend)) assert response == reference_response native_phases[backend] = phases @@ -235,7 +243,7 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile", REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", ] - if any(b in ("rns", "native") for b in backends): + if any(b in ("rns", "native", "residue") for b in backends): source_paths.extend((REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.so")) if args.seal: source_paths.append(REPO_ROOT / "experiments/bfv/packed_hamming.py") @@ -266,6 +274,12 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: "command": sys.argv, "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", "config": config, + "ciphertext_modulus_hex": format(client._pk()["q"], "x"), + "ciphertext_modulus_primes": _rns_coefficient_primes( + args.poly_modulus_degree, args.coeff_modulus_bits + ) + if args.rns_modulus + else None, "seed": args.seed, "vectors": len(vectors), "setup_timings": setup, @@ -301,7 +315,9 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: "caches": { backend: { **server._evaluator().cache_info(), - "server_plan_bytes": server._native().cache_bytes() if backend == "native" else 0, + "server_plan_bytes": server._native().cache_bytes() + if backend in ("native", "residue") + else 0, } for backend, server in servers.items() }, diff --git a/benchmarks/results/native_bfv_residue_1024.json b/benchmarks/results/native_bfv_residue_1024.json new file mode 100644 index 0000000..fc3be65 --- /dev/null +++ b/benchmarks/results/native_bfv_residue_1024.json @@ -0,0 +1,407 @@ +{ + "environment": { + "utc": "2026-09-11T02:22:03.335196+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "28170049a93a15b1a06a42f5ae89d74e80f50be2", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "1024", + "--rns-modulus", + "--backends", + "native,residue", + "--seal", + "--repeats", + "4", + "--native-profile", + "--json-out", + "benchmarks/results/native_bfv_residue_1024.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "ciphertext_modulus_hex": "ffffffffffbc0030000004bff78002fff1004bffbc001", + "ciphertext_modulus_primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137 + ], + "seed": 1337, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 4.5589400180615485, + "index_encrypt_s": 6.888210566015914, + "query_encrypt_s": 0.10452251299284399, + "public_key_export_s": 0.8208927081432194, + "native_public_key_import_s": 0.8457178140524775, + "residue_public_key_import_s": 0.8620186080224812 + }, + "runs": [ + { + "backend": "native", + "repeat": 0, + "server_compute_s": 5.6354313138872385 + }, + { + "backend": "residue", + "repeat": 0, + "server_compute_s": 3.1145015170332044 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 2.4679608230944723 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 2.463819060008973 + }, + { + "backend": "residue", + "repeat": 1, + "server_compute_s": 2.8064445089548826 + }, + { + "backend": "native", + "repeat": 1, + "server_compute_s": 5.272516414988786 + }, + { + "backend": "native", + "repeat": 2, + "server_compute_s": 5.297504244139418 + }, + { + "backend": "residue", + "repeat": 2, + "server_compute_s": 2.8145669419318438 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 2.4813549709506333 + }, + { + "backend": "seal", + "repeat": 3, + "server_compute_s": 2.4831731480080634 + }, + { + "backend": "residue", + "repeat": 3, + "server_compute_s": 2.8025826739612967 + }, + { + "backend": "native", + "repeat": 3, + "server_compute_s": 5.278038259129971 + } + ], + "summary": { + "native": { + "first_search_s": 5.6354313138872385, + "warm_searches_s": [ + 5.272516414988786, + 5.297504244139418, + 5.278038259129971 + ], + "warm_median_s": 5.278038259129971 + }, + "residue": { + "first_search_s": 3.1145015170332044, + "warm_searches_s": [ + 2.8064445089548826, + 2.8145669419318438, + 2.8025826739612967 + ], + "warm_median_s": 2.8064445089548826 + }, + "seal": { + "first_search_s": 2.4679608230944723, + "warm_searches_s": [ + 2.463819060008973, + 2.4813549709506333, + 2.4831731480080634 + ], + "warm_median_s": 2.4813549709506333 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "native": { + "first": 1.0, + "warm_median": 1.0 + }, + "residue": { + "first": 1.809416782450441, + "warm_median": 1.8806850597931501 + }, + "seal": { + "first": 2.283436293296264, + "warm_median": 2.1270790841779075 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 27680261, + "query_bytes": 432638, + "public_key_bytes": 39584794, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 105892, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "native_phases": { + "native": { + "forward_ntt": { + "seconds": 1.580388313, + "calls": 18280 + }, + "inverse_ntt": { + "seconds": 0.597422036, + "calls": 7480 + }, + "to_residues": { + "seconds": 0.131717274, + "calls": 256 + }, + "crt_reconstruct": { + "seconds": 0.734090161, + "calls": 1726 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.063746828, + "calls": 703 + }, + "pointwise": { + "seconds": 0.45272087000000005, + "calls": 895 + }, + "scale_round": { + "seconds": 0.16394777900000002, + "calls": 193 + }, + "mod_q": { + "seconds": 0.476317101, + "calls": 1534 + }, + "buffers": { + "seconds": 0.023535914, + "calls": 1023 + }, + "automorphism": { + "seconds": 0.365672055, + "calls": 639 + }, + "add_sub": { + "seconds": 0.187055671, + "calls": 767 + }, + "wire_import": { + "seconds": 0.029517076000000003, + "calls": 130 + }, + "wire_export": { + "seconds": 0.00015730300000000002, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.5200640830000001, + "calls": 1 + } + }, + "residue": { + "forward_ntt": { + "seconds": 1.211404176, + "calls": 13934 + }, + "inverse_ntt": { + "seconds": 0.478378373, + "calls": 5946 + }, + "to_residues": { + "seconds": 0.140344466, + "calls": 320 + }, + "crt_reconstruct": { + "seconds": 0.17903116700000002, + "calls": 194 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.081616798, + "calls": 13357 + }, + "pointwise": { + "seconds": 0.28731105500000004, + "calls": 1151 + }, + "scale_round": { + "seconds": 0.162540489, + "calls": 193 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.020592492, + "calls": 895 + }, + "automorphism": { + "seconds": 0.031140883, + "calls": 639 + }, + "add_sub": { + "seconds": 0.05861510600000001, + "calls": 1406 + }, + "wire_import": { + "seconds": 0.029655261000000002, + "calls": 130 + }, + "wire_export": { + "seconds": 0.000174463, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.05888882300000001, + "calls": 705 + }, + "other": { + "seconds": 0.082117907, + "calls": 1 + } + } + }, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", + "sizes": { + "encrypted_index_bytes": 23598495, + "public_keys_bytes": 85603326, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "caches": { + "native": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 0, + "switch_keys": 18, + "key_payload_bytes": 56623104, + "server_plan_bytes": 655360 + }, + "residue": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 589824 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "846b003afc224e40d2c5edea25ffd76fc2235c343fb6f37dcfe5dffef4a56219", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "499ed412000271c75c1980a4f2f312ef62d5da069f7f0995957a0de77dd2ea0a", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "0d8b6a02fddf6b58996da91146ad161ee88783475d87edefe93946ae3defb0c6", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "f76e007558fbd1f6dd6ed6929770cddc6573568fc1bec9af2b6eca31afbd24e8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "1f468bf78fba83753839eddda17b1709f05abfb9b75b4950b7a782065268b15f", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "48c2b2c7c8fcc8c957009cf328963026b15baf03f765a35cb616ffe02e875c00", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "2b6e18633fd8e07e7ec5527b7d4ccba2de5a4aa866453e4c32a4e48bc731c8a3", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "a2fe4ca8273a3f051ba44b0a1614bc19ce757939eeb37fc57835285392682356", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "99b513895a7baca7eebb65f5a1cb3a7d10b297b14479a327269828aa1f91058f", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "fa0315210c10b41cae83fad5fb4bf493354d76837cd898a7410d32ca623e92a5", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "4ff544e8e7371aa0800c7076d1fde9f9ba4bc20143444596618be5ae1cedad47", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "d9505ac8de6f7c356512111eee9e3d8ae19dd046034a81658b021db4596ecea8", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/benchmarks/results/native_bfv_residue_32.json b/benchmarks/results/native_bfv_residue_32.json new file mode 100644 index 0000000..ac3e2c3 --- /dev/null +++ b/benchmarks/results/native_bfv_residue_32.json @@ -0,0 +1,407 @@ +{ + "environment": { + "utc": "2026-09-11T02:19:09.439953+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "28170049a93a15b1a06a42f5ae89d74e80f50be2", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "32", + "--rns-modulus", + "--backends", + "native,residue", + "--seal", + "--repeats", + "4", + "--native-profile", + "--json-out", + "benchmarks/results/native_bfv_residue_32.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "ciphertext_modulus_hex": "ffffffffffbc0030000004bff78002fff1004bffbc001", + "ciphertext_modulus_primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137 + ], + "seed": 1337, + "vectors": 32, + "setup_timings": { + "key_generation_s": 4.348404427990317, + "index_encrypt_s": 0.21356452419422567, + "query_encrypt_s": 0.10177573701366782, + "public_key_export_s": 0.7944609080441296, + "native_public_key_import_s": 0.8522930489853024, + "residue_public_key_import_s": 0.8435139840003103 + }, + "runs": [ + { + "backend": "native", + "repeat": 0, + "server_compute_s": 0.5181663730181754 + }, + { + "backend": "residue", + "repeat": 0, + "server_compute_s": 0.397010528948158 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 0.07721250806935132 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 0.07963385712355375 + }, + { + "backend": "residue", + "repeat": 1, + "server_compute_s": 0.0875147629994899 + }, + { + "backend": "native", + "repeat": 1, + "server_compute_s": 0.15932174399495125 + }, + { + "backend": "native", + "repeat": 2, + "server_compute_s": 0.15907482896000147 + }, + { + "backend": "residue", + "repeat": 2, + "server_compute_s": 0.08726914902217686 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 0.07568373787216842 + }, + { + "backend": "seal", + "repeat": 3, + "server_compute_s": 0.07749695307575166 + }, + { + "backend": "residue", + "repeat": 3, + "server_compute_s": 0.08695215196348727 + }, + { + "backend": "native", + "repeat": 3, + "server_compute_s": 0.157100300071761 + } + ], + "summary": { + "native": { + "first_search_s": 0.5181663730181754, + "warm_searches_s": [ + 0.15932174399495125, + 0.15907482896000147, + 0.157100300071761 + ], + "warm_median_s": 0.15907482896000147 + }, + "residue": { + "first_search_s": 0.397010528948158, + "warm_searches_s": [ + 0.0875147629994899, + 0.08726914902217686, + 0.08695215196348727 + ], + "warm_median_s": 0.08726914902217686 + }, + "seal": { + "first_search_s": 0.07721250806935132, + "warm_searches_s": [ + 0.07963385712355375, + 0.07568373787216842, + 0.07749695307575166 + ], + "warm_median_s": 0.07749695307575166 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "native": { + "first": 1.0, + "warm_median": 1.0 + }, + "residue": { + "first": 1.3051703550306546, + "warm_median": 1.8228071516954671 + }, + "seal": { + "first": 6.71091233758091, + "warm_median": 2.052659138798775 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 865128, + "query_bytes": 432548, + "public_key_bytes": 39585389, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 103294, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "native_phases": { + "native": { + "forward_ntt": { + "seconds": 0.047128805, + "calls": 548 + }, + "inverse_ntt": { + "seconds": 0.017303183, + "calls": 226 + }, + "to_residues": { + "seconds": 0.004098931, + "calls": 8 + }, + "crt_reconstruct": { + "seconds": 0.021912418000000003, + "calls": 52 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.001514979, + "calls": 21 + }, + "pointwise": { + "seconds": 0.012786347, + "calls": 27 + }, + "scale_round": { + "seconds": 0.0062369140000000005, + "calls": 7 + }, + "mod_q": { + "seconds": 0.014188331, + "calls": 46 + }, + "buffers": { + "seconds": 0.000676216, + "calls": 31 + }, + "automorphism": { + "seconds": 0.01051699, + "calls": 19 + }, + "add_sub": { + "seconds": 0.0055407170000000006, + "calls": 23 + }, + "wire_import": { + "seconds": 0.001330797, + "calls": 6 + }, + "wire_export": { + "seconds": 0.00015322300000000002, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.015019135000000001, + "calls": 1 + } + }, + "residue": { + "forward_ntt": { + "seconds": 0.035708193, + "calls": 418 + }, + "inverse_ntt": { + "seconds": 0.014244742000000001, + "calls": 180 + }, + "to_residues": { + "seconds": 0.004256384, + "calls": 10 + }, + "crt_reconstruct": { + "seconds": 0.0058745360000000005, + "calls": 8 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.0022139670000000003, + "calls": 399 + }, + "pointwise": { + "seconds": 0.008133528000000001, + "calls": 35 + }, + "scale_round": { + "seconds": 0.006246996, + "calls": 7 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.00052473, + "calls": 27 + }, + "automorphism": { + "seconds": 0.000875448, + "calls": 19 + }, + "add_sub": { + "seconds": 0.001730596, + "calls": 42 + }, + "wire_import": { + "seconds": 0.001332047, + "calls": 6 + }, + "wire_export": { + "seconds": 0.000152533, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.0018317960000000001, + "calls": 23 + }, + "other": { + "seconds": 0.0030155610000000004, + "calls": 1 + } + } + }, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", + "sizes": { + "encrypted_index_bytes": 737480, + "public_keys_bytes": 85602527, + "query_bytes": 368754, + "response_bytes": 102494, + "query_plus_response_bytes": 471248 + }, + "caches": { + "native": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 0, + "switch_keys": 18, + "key_payload_bytes": 56623104, + "server_plan_bytes": 655360 + }, + "residue": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 589824 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 31, + "distance": 0 + }, + { + "index": 19, + "distance": 241 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "846b003afc224e40d2c5edea25ffd76fc2235c343fb6f37dcfe5dffef4a56219", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "499ed412000271c75c1980a4f2f312ef62d5da069f7f0995957a0de77dd2ea0a", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "0d8b6a02fddf6b58996da91146ad161ee88783475d87edefe93946ae3defb0c6", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "f76e007558fbd1f6dd6ed6929770cddc6573568fc1bec9af2b6eca31afbd24e8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "1f468bf78fba83753839eddda17b1709f05abfb9b75b4950b7a782065268b15f", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "48c2b2c7c8fcc8c957009cf328963026b15baf03f765a35cb616ffe02e875c00", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "2b6e18633fd8e07e7ec5527b7d4ccba2de5a4aa866453e4c32a4e48bc731c8a3", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "a2fe4ca8273a3f051ba44b0a1614bc19ce757939eeb37fc57835285392682356", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "99b513895a7baca7eebb65f5a1cb3a7d10b297b14479a327269828aa1f91058f", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "fa0315210c10b41cae83fad5fb4bf493354d76837cd898a7410d32ca623e92a5", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "4ff544e8e7371aa0800c7076d1fde9f9ba4bc20143444596618be5ae1cedad47", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "d9505ac8de6f7c356512111eee9e3d8ae19dd046034a81658b021db4596ecea8", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/benchmarks/results/native_bfv_residue_prime_control_1024.json b/benchmarks/results/native_bfv_residue_prime_control_1024.json new file mode 100644 index 0000000..2bd3110 --- /dev/null +++ b/benchmarks/results/native_bfv_residue_prime_control_1024.json @@ -0,0 +1,221 @@ +{ + "environment": { + "utc": "2026-09-11T02:23:26.903404+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "28170049a93a15b1a06a42f5ae89d74e80f50be2", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "1024", + "--backends", + "native", + "--repeats", + "4", + "--native-profile", + "--json-out", + "benchmarks/results/native_bfv_residue_prime_control_1024.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "response_modulus_bits": 50, + "device": "cpu" + }, + "ciphertext_modulus_hex": "fffffffffffffffffffffffffffffffffffffffffffd1", + "ciphertext_modulus_primes": null, + "seed": 1337, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 4.5147744179703295, + "index_encrypt_s": 7.154086224967614, + "query_encrypt_s": 0.10949854203499854, + "public_key_export_s": 0.8259541729930788, + "native_public_key_import_s": 0.8619675009977072 + }, + "runs": [ + { + "backend": "native", + "repeat": 0, + "server_compute_s": 5.855862850090489 + }, + { + "backend": "native", + "repeat": 1, + "server_compute_s": 5.464001099811867 + }, + { + "backend": "native", + "repeat": 2, + "server_compute_s": 5.455950492061675 + }, + { + "backend": "native", + "repeat": 3, + "server_compute_s": 5.4547371591906995 + } + ], + "summary": { + "native": { + "first_search_s": 5.855862850090489, + "warm_searches_s": [ + 5.464001099811867, + 5.455950492061675, + 5.4547371591906995 + ], + "warm_median_s": 5.455950492061675 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "native": { + "first": 1.0, + "warm_median": 1.0 + } + }, + "seal_comparison": null, + "native_phases": { + "native": { + "forward_ntt": { + "seconds": 1.5736347460000002, + "calls": 18280 + }, + "inverse_ntt": { + "seconds": 0.5910954350000001, + "calls": 7480 + }, + "to_residues": { + "seconds": 0.132091116, + "calls": 256 + }, + "crt_reconstruct": { + "seconds": 0.854933025, + "calls": 1726 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.062764579, + "calls": 703 + }, + "pointwise": { + "seconds": 0.446116137, + "calls": 895 + }, + "scale_round": { + "seconds": 0.16660401900000002, + "calls": 193 + }, + "mod_q": { + "seconds": 0.504614024, + "calls": 1534 + }, + "buffers": { + "seconds": 0.022925539, + "calls": 1023 + }, + "automorphism": { + "seconds": 0.36388088900000004, + "calls": 639 + }, + "add_sub": { + "seconds": 0.18627094300000002, + "calls": 767 + }, + "wire_import": { + "seconds": 0.029229452000000003, + "calls": 130 + }, + "wire_export": { + "seconds": 0.00016510300000000002, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.5101353780000001, + "calls": 1 + } + } + }, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", + "sizes": { + "encrypted_index_bytes": 23598495, + "public_keys_bytes": 85602362, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "caches": { + "native": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 4, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 0, + "switch_keys": 18, + "key_payload_bytes": 56623104, + "server_plan_bytes": 655360 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "846b003afc224e40d2c5edea25ffd76fc2235c343fb6f37dcfe5dffef4a56219", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "499ed412000271c75c1980a4f2f312ef62d5da069f7f0995957a0de77dd2ea0a", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "0d8b6a02fddf6b58996da91146ad161ee88783475d87edefe93946ae3defb0c6", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "f76e007558fbd1f6dd6ed6929770cddc6573568fc1bec9af2b6eca31afbd24e8", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "1f468bf78fba83753839eddda17b1709f05abfb9b75b4950b7a782065268b15f", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "48c2b2c7c8fcc8c957009cf328963026b15baf03f765a35cb616ffe02e875c00", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "2b6e18633fd8e07e7ec5527b7d4ccba2de5a4aa866453e4c32a4e48bc731c8a3", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "a2fe4ca8273a3f051ba44b0a1614bc19ce757939eeb37fc57835285392682356", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "99b513895a7baca7eebb65f5a1cb3a7d10b297b14479a327269828aa1f91058f", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "fa0315210c10b41cae83fad5fb4bf493354d76837cd898a7410d32ca623e92a5", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "4ff544e8e7371aa0800c7076d1fde9f9ba4bc20143444596618be5ae1cedad47", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "d9505ac8de6f7c356512111eee9e3d8ae19dd046034a81658b021db4596ecea8" + } +} diff --git a/docs/index.rst b/docs/index.rst index dee7bcd..909bef0 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -84,6 +84,7 @@ Choose your path: research/native-bfv research/native-bfv-ntt research/native-bfv-server + research/native-bfv-residue research/bfv-packed-hamming .. toctree:: diff --git a/docs/research/native-bfv-residue.md b/docs/research/native-bfv-residue.md new file mode 100644 index 0000000..ac747bf --- /dev/null +++ b/docs/research/native-bfv-residue.md @@ -0,0 +1,272 @@ +# Persistent RNS BFV server + +`BFVClient(rns_modulus=True, server_backend="residue")` keeps ciphertexts as +arrays of 60-bit residues after the initial BFV tensor scale-and-round. Key +switching, rotations, additions, plaintext masks and the result merge tree run +on these arrays in our own C++. The reference, optimized, `rns` and `native` +backends can evaluate the same keys and ciphertexts and give exactly the same +output coefficients. The existing Paillier and SEAL implementations are retained. + +## Configuration and compatibility + +This is an explicit new cryptographic configuration. It requires **fresh keys +and a fresh encrypted index**. Existing prime-modulus keys cannot be used with +`residue`; they continue to work with the previous backends. Nothing migrates +keys or ciphertexts automatically. + +The default remains `rns_modulus=False` and `server_backend="optimized"`. +When the RNS option is off, parameter serialization and key fingerprints retain +their original representation, including compatibility with saved v1 keys. +When enabled, `rns_modulus: true` is serialized in the configuration and public +key parameters and included in the fingerprint. Ciphertexts keep the same +packed format and contain the actual Q and key identity. + +The first residue layout supports ciphertext modulus sizes 60, 120, ..., 480 +bits. For each ring degree, it selects distinct primes below 2^60 with +`p = 1 mod 2N`, in the same deterministic order as the auxiliary NTT bases. +The default 180-bit Q is the product of three such primes. The binary gadget, +error sampling, plaintext modulus, batching, encryption and exact rounding +algorithms are unchanged. An equal-bit response-compaction request retains Q +instead of increasing it to the same-bit prime. + +This remains experimental leveled BFV with no security audit or security-level +claim. It uses no SEAL code and performs no bootstrapping. The performance +comparison below does not establish equivalent security to SEAL's parameters. + +## Build and use + +The extension interface is now ABI 3. Rebuild an older binary: + +```bash +make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" +``` + +```python +import json +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + +client = BFVClient(embed_len=512, rns_modulus=True, server_backend="residue") +server = BFVClient(skip_key_gen=True, server_backend="residue") +server.load_config(json.loads(client.stringify_config())) +server.load_stringified_keys(client.stringify_pk()) +assert server.keys is None + +query = [0, 1] * 256 +vectors = [query, [1 - bit for bit in query]] +encrypted_query = client.encrypt_vec_one(query) +encrypted_index = client.encrypt_vec_packed(vectors) +response = server.encode_hamming_server_packed(encrypted_query, encrypted_index, 2) +assert client.decode_hamming_client_packed(response, 2) == [0, 512] +``` + +Both individual and packed search methods use the new server. Reuse a server +across queries to amortize public-key and mask preparation. Key generation, +encryption and client decryption continue to use our Python/GMP scheme. +Top-k selection still happens on the client. These local interfaces do not add +BFV support to the production HTTP endpoints. + +## Why the representation matters + +Previously, auxiliary primes only accelerated polynomial products. Their +product M differed from the ciphertext modulus q. Every key switch had to +convert both output components back to GMP integers and reduce them modulo q. +Rotations, additions and the merge tree then operated on those GMP objects. + +With Q equal to the product of the ciphertext primes, the Chinese Remainder +Theorem identifies arithmetic modulo Q with componentwise arithmetic modulo +each prime. A key-switch product needs only these primes, regardless of the +unreduced integer convolution's size: its result is needed modulo Q. The +default key-switch basis shrinks from four primes to three. Ciphertext additions +and automorphisms become small-word loops; masks multiply directly in this +basis. This follows the residue representation described in the textbook's +[RNS operations section](https://fhetextbook.github.io/ApplyingRNSTechniquestoFHEOperations.html). + +Binary gadget decomposition still needs the unique coefficient in [0,Q). +`ResidueArithmetic::compose_words` reconstructs it with exact mixed-radix +(Garner) arithmetic, using fixed 64-bit limbs and 128-bit products. It neither +allocates GMP integers nor estimates a CRT quotient with floating point. For +residues r_j, it finds mixed-radix digits d_j by repeatedly computing +`(value - d_k) * inverse(p_k) mod p_j`, then forms +`d_0 + p_0*(d_1 + p_1*(...))`. Each d_j is in [0,p_j), so this is the canonical +representative. Its binary gadget digits are the same as the reference's. +Digits up to 60 bits are extracted once and reused across primes. Wider gadgets +are reduced in chunks of at most 60 bits; the maximum supported Q uses eight +64-bit limbs. No additional evaluation keys or special key-switch prime are +introduced. + +The scale-and-round in ciphertext multiplication still requires the signed +integer tensor before reducing modulo Q. It uses the existing sufficiently +large auxiliary base, exact GMP CRT and signed rounding. After that step, the +ciphertext remains in RNS until the final response is composed and compacted. +This preserves exact coefficient equality with our reference, including its +rounding choices and noise behavior for the same key. Ciphertexts normally use +coefficient-form residues between operations; transforms are applied for +polynomial multiplication, rather than keeping every operation in NTT form. + +## Reproduce the comparison + +```bash +.venv/bin/python benchmarks/bfv_server.py --num-vectors 1024 \ + --rns-modulus --backends native,residue --seal --repeats 4 \ + --native-profile --json-out /tmp/bfv-residue-1024.json +``` + +Use `--num-vectors 32` for the small workload. Omit `--seal` when TenSEAL is +unavailable. `benchmarks/bfv_client_matrix.py` also accepts `--rns-modulus` and +`--server-backend residue`. Native backends receive identical public keys and +encrypted inputs; every result must match exactly and decrypt to all expected +Hamming distances. SEAL gets the same plaintext vectors with its own parameters. +The first search includes lazy cache setup; three warm searches follow, with +backend order alternating. Additional profiled searches are excluded from +timing medians. Run without concurrent tests or benchmarks. + +## Recorded results + +The final comparisons and phase profiles are recorded below. All times are +seconds on the AMD Ryzen 7 5800X, Python 3.12.3, GMP 6.3.0, GCC 13.3.0 (`-O3`) +and TenSEAL 0.3.16. There is one CPU evaluation thread, with no GPU or SEAL code +in our backend. Each native row uses N=8192, t=65537, a 180-bit product Q, six +30-bit gadget digits, error eta=21 and 50-bit terminal compaction. + +| Vectors | Backend | First search | Median of three warm searches | +| ---: | --- | ---: | ---: | +| 32 | `native`, same product Q | 0.5182 | 0.1591 | +| 32 | `residue` | 0.3970 | 0.0873 | +| 32 | SEAL prototype | 0.0772 | 0.0775 | +| 1,024 | `native`, same product Q | 5.6354 | 5.2780 | +| 1,024 | `residue` | 3.1145 | 2.8064 | +| 1,024 | SEAL prototype | 2.4680 | 2.4814 | + +At 1,024 vectors, residue takes **46.8% less server time** than native with the +same keys and inputs: a **1.88x speedup**. Its warm time is **13.1% longer** than +SEAL's in this run. The small workload improves by 45.1%. Every ciphertext from +our two backends matches exactly, including the compacted response. Every +decoded distance matches the plaintext oracle for every backend. + +SEAL uses active coefficient-prime sizes [43,43,44,44] and an additional 44-bit +special key-switch prime, with its TC128 setting. Our scheme has different +parameters and key switching, with no matching security-level claim. Native +server times include framing, ciphertext decoding/encoding, arithmetic and +compaction, but exclude outer MessagePack. The preserved SEAL prototype also +includes its MessagePack and temporary-file binding I/O. Key import, encryption, +client decryption and network time are excluded. These are practical server +calls, rather than timings of identical cryptographic kernels. + +The 1,024-vector response is 102,496 bytes and query plus response is 471,250 +bytes, matching the preceding native experiment's measured sizes. The encrypted +index is 23,598,495 bytes and serialized public keys are 85,603,326 bytes. This +change targets compute, with the same packed wire structure. Small byte-count +variation between fresh keys/runs is possible. Both native backends have a +26-bit minimum response noise budget here; SEAL reports 19 bits for its own +parameters and noise diagnostic. + +Prepared C++ key payload falls from 56,623,104 to 42,467,328 bytes (25% less), +with the same 18 keys. This is an additional runtime cache, not the serialized +BFV setup-key download. Transform-table payload is 2,752,512 bytes in either +backend; prepared plaintext tables/masks use 655,360 versus 589,824 bytes. +These figures exclude containers, small CRT constants and transient allocations. + +Raw samples, environment, exact modulus/primes, ciphertext checks and source/ +binary hashes are in `benchmarks/results/native_bfv_residue_32.json` and +`benchmarks/results/native_bfv_residue_1024.json`. + +An additional 1,024-vector control kept the original prime-modulus configuration +and `native` backend: first search 5.8559 seconds, warm median 5.4560 seconds. +Its raw data is in `benchmarks/results/native_bfv_residue_prime_control_1024.json`. +This is a separate run with fresh random encryption, not part of the interleaved +same-key comparison used to calculate the 1.88x speedup. The earlier recorded +5.3426-second native result is retained in [the previous report](native-bfv-server.md). + +## What remains slower than SEAL + +The additional exclusive phase profile for 1,024 vectors shows the effect of +retaining residues. These are instrumented extra searches, excluded from the +performance medians above. + +| Phase | `native` | `residue` | +| --- | ---: | ---: | +| Forward NTT | 1.5804 | 1.2114 | +| Inverse NTT | 0.5974 | 0.4784 | +| GMP CRT reconstruction/conversion | 0.7341 | 0.1790 | +| Separately instrumented reduction modulo Q | 0.4763 | 0 | +| Fixed-word RNS composition | 0 | 0.0589 | +| Gadget extraction/conversion | 0.0637 | 0.0816 | +| Automorphisms | 0.3657 | 0.0311 | +| Additions/subtractions | 0.1871 | 0.0586 | +| Pointwise products/accumulation | 0.4527 | 0.2873 | +| Scale-and-round | 0.1639 | 0.1625 | + +Both perform 703 key switches. Forward transforms fall from 18,280 to 13,934; +inverse transforms fall from 7,480 to 5,946. The residue path has 194 GMP CRT +operations: three tensor components for each of 64 tiles, plus two final +response components. The earlier native path has 1,726. Reductions inside the +remaining tensor/compaction work are charged to their own scopes, so the zero +`mod_q` entry does not mean that all big-integer modular arithmetic is gone. +The full JSON retains the other phases and exact call counts. + +Forward/inverse NTT now account for about **60%** of our profiled time. The +remaining GMP CRT and scale-and-round account for about **12%**. The fixed-word +reconstruction needed for binary gadget extraction is about **2%**. Python, +binding and unscoped work together account for about **3%**. + +Source inspection points to two concrete areas for the next optimization: + +- SEAL's [NTT implementation](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/util/ntt.cpp) + and [transform handler](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/util/dwthandler.h) + use transform-ordered root tables and specialized/unrolled Harvey butterflies. + Our kernels still use a separate negacyclic twist pass and generic cyclic + butterfly loops with strided root access. Tuning that kernel now targets most + of our search time. The installed SEAL wheel's optional acceleration features + have not been established; these results do not assume Intel HEXL is enabled. +- SEAL's [BFV evaluator](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/evaluator.cpp) + and [RNS tools](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/util/rns.cpp) + perform BEHZ-style multiplication with RNS base conversion, division and + scaling. Our tensor scale-and-round still reconstructs exact GMP integers. + Replacing that boundary requires preserving BFV's rounding/noise semantics; + dividing each residue independently would be incorrect. + +These are source-based explanations and measured bottlenecks in our code. +We have not instrumented SEAL's internal phases, so we cannot assign an exact +fraction of the remaining 0.325-second gap to each difference. Changes to +transform kernels and RNS multiplication can be measured independently using +the preserved native engine as an oracle. Further changes to private key +generation or encryption would not affect this server-search metric. + +## Implementation files and verification + +| Path relative to the repository | Responsibility | +| --- | --- | +| `src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h` | Persistent RNS ciphertexts, fixed-word reconstruction and complete server | +| `src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h` | Shared NTT/key dot products and optional ciphertext-prime basis | +| `src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h` | Shared layout, masks and streaming merge order | +| `src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp` | Validated ABI 3 boundary and native plan selection | +| `src/xtrace_sdk/x_vec/crypto/encryption/bfv.py` | Optional product modulus and compatible key serialization | +| `src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py` | Compiled arithmetic context and key preparation | +| `src/xtrace_sdk/x_vec/crypto/bfv_client.py` | Client configuration and backend selection | +| `tests/x_vec/test_bfv_residue.py` | Saved-key compatibility, parameters, exact search comparisons | +| `tests/x_vec/native/test_bfv_rns.cpp` | Independent integer/gadget oracles across all supported Q widths | + +The common native boundary tests also run with the residue backend: malformed +inputs, public-only evaluation, direct packed buffers, concurrent searches, +profiling, cache invalidation and capsule lifetimes. Standalone tests compare +mixed-radix reconstruction, gadget digits, key switching and rotations with +GMP and schoolbook integer oracles, including zero, Q-1, prime/word boundaries, +one-bit gadgets, wide gadgets and 480-bit Q. They also assert that a residue +key switch performs no GMP CRT reconstruction or reduction modulo Q. + +Validation passed 192 offline tests, including 110 BFV tests; production API and +GPU tests were excluded. Ruff lint, mypy, and formatting checks for the BFV +modules, tests and benchmarks passed. A broader SDK formatting check reports +existing formatting differences in other modules and unchanged portions of +`xtrace_types.py`; those unrelated sections were left alone. Standalone +arithmetic and Python-binding AddressSanitizer/UndefinedBehaviorSanitizer runs +passed; the latter ran 25 server, compatibility and boundary tests against the +sanitized binary. The local tracing sandbox requires leak checking to be +disabled, as in the previous native-backend verification. + +A built CPython 3.12 platform wheel includes the new header and ABI 3 extension. +An isolated import from that wheel passed public-only residue search and exact +native fallback comparison, without importing SEAL or TenSEAL. +The default N=8192 example above and both matrix-benchmark interfaces passed. +Sphinx built successfully with the existing README cross-reference warning in +`bfv-packed-hamming.md`; no new documentation warning was introduced. diff --git a/docs/research/native-bfv-server.md b/docs/research/native-bfv-server.md index 2ac2aeb..9b85cee 100644 --- a/docs/research/native-bfv-server.md +++ b/docs/research/native-bfv-server.md @@ -14,8 +14,9 @@ experimental leveled BFV without a security audit or bootstrapping. ## Build and use -Rebuild the extension from the repository root. The new interface is ABI version -2; an older binary raises a rebuild error. +Rebuild the extension from the repository root. This backend originally used +ABI 2; the current extension is ABI 3 after adding the +[persistent RNS backend](native-bfv-residue.md). An older binary raises a rebuild error. ```bash make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" @@ -243,8 +244,10 @@ Leak checking is disabled for the Python-hosted sanitizer run; the standalone CI run retains it. As documented for the earlier backend, the local tracing sandbox requires `ASAN_OPTIONS=detect_leaks=0` for standalone runs as well. -Further opportunities include keeping coefficients in RNS across more -operations, reusing more working buffers and moving client-side operations into -C++. This backend still uses GMP internally and is not a fully RNS redesign of -the cryptographic scheme. A rewrite of encryption or key generation would not -directly improve the server-search timing measured here. +The [persistent RNS backend](native-bfv-residue.md) now keeps coefficients in RNS +across key switches, rotations, masks and the merge tree, with an optional new +ciphertext modulus. The `native` backend described here retains its GMP +intermediates and existing-key compatibility. Further opportunities include +reusing more working buffers and moving client-side operations into C++. A +rewrite of encryption or key generation would not directly improve the +server-search timing measured here. diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_client.py index 5791ad5..3a247d7 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_client.py +++ b/src/xtrace_sdk/x_vec/crypto/bfv_client.py @@ -11,7 +11,7 @@ from typing import Any from xtrace_sdk.x_vec.crypto.device import DeviceMode -from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _parameter_dict from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVEvaluator, BFVServerBackend from xtrace_sdk.x_vec.crypto.encryption.bfv_native import BFVNativeServer from xtrace_sdk.x_vec.crypto.hamming_client_base import HammingClientBase @@ -40,8 +40,13 @@ class BFVClient(HammingClientBase): :param skip_key_gen: Construct an empty client for loading saved keys. :param device: ``cpu`` or ``auto``; a CUDA implementation is not yet available. :param server_backend: ``optimized`` (cached GMP), ``reference``, ``rns`` - (compiled arithmetic), or ``native`` (complete C++ server evaluation). - This runtime choice is independent of the serialized crypto configuration. + (compiled arithmetic), ``native`` (complete C++ server evaluation), or + ``residue`` (persistent RNS after the BFV tensor scale-and-round). + ``residue`` requires keys generated with ``rns_modulus=True``. + :param rns_modulus: Use a product of 60-bit NTT primes for the ciphertext + modulus. Requires fresh keys and a fresh encrypted index. Modulus bits + must be a multiple of 60 in [60, 480]. All backends can use these keys; + the default prime-modulus configuration remains available. """ def __init__( @@ -56,19 +61,31 @@ def __init__( skip_key_gen: bool = False, device: DeviceMode = "auto", server_backend: BFVServerBackend = "optimized", + rns_modulus: bool = False, ) -> None: if device not in ("cpu", "auto", "gpu"): raise ValueError("device must be 'auto', 'cpu', or 'gpu'") if device == "gpu": raise NotImplementedError("Native BFV currently supports only the CPU backend") self.device = "cpu" - if server_backend not in ("optimized", "reference", "rns", "native"): - raise ValueError("server_backend must be 'optimized', 'reference', 'rns', or 'native'") + if server_backend not in ("optimized", "reference", "rns", "native", "residue"): + raise ValueError( + "server_backend must be 'optimized', 'reference', 'rns', 'native', or 'residue'" + ) self.server_backend = server_backend + if server_backend == "residue" and not rns_modulus and not skip_key_gen: + raise ValueError( + "The residue backend requires fresh keys generated with rns_modulus=True" + ) self._server_evaluator: BFVEvaluator | None = None self._evaluator_public_key: BFVPublicKey | None = None self.params = BFVParameters( - poly_modulus_degree, plain_modulus, coeff_modulus_bits, decomposition_bits, error_eta + poly_modulus_degree, + plain_modulus, + coeff_modulus_bits, + decomposition_bits, + error_eta, + rns_modulus, ) self.embed_len = embed_len self.response_modulus_bits = response_modulus_bits @@ -162,7 +179,7 @@ def stringify_config(self) -> str: return json.dumps( { "embed_len": self.embed_len, - **asdict(self.params), + **_parameter_dict(self.params), "response_modulus_bits": self.response_modulus_bits, "device": "cpu", } @@ -280,7 +297,7 @@ def encode_hamming_server( compact: bool = True, ) -> EncryptedVector: """Compute an encrypted distance using only public evaluation keys.""" - if self.server_backend == "native": + if self.server_backend in ("native", "residue"): return self._native().search(ct1, [ct2], 1, compact=compact)[0] pk = self._pk() result = self._distance_tile( @@ -307,7 +324,7 @@ def encode_hamming_server_packed( pk = self._pk() capacity = self.vectors_per_ciphertext self._validate_count(vector_count, len(index), capacity) - if self.server_backend == "native": + if self.server_backend in ("native", "residue"): return self._native().search(query, index, vector_count, compact=compact) query_ct = BFV.ciphertext_from_ints(query, pk) evaluator = self._evaluator() diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile index 1d66de0..48d59d5 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile @@ -15,7 +15,7 @@ LDLIBS ?= -lgmpxx -lgmp .PHONY: all clean all: $(TARGET) -$(TARGET): bindings.cpp rns_ntt.h profile.h bfv_server.h packed_wire.h Makefile +$(TARGET): bindings.cpp rns_ntt.h profile.h bfv_server.h bfv_residue.h packed_wire.h Makefile $(CXX) $(CPPFLAGS) $(CXXFLAGS) -std=c++17 -Wall -Wextra -fPIC -shared \ -I$(PY_INC) bindings.cpp $(LDFLAGS) $(LDLIBS) -o $@ diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi index e65625d..a5757a0 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi @@ -2,7 +2,9 @@ from typing import Any ABI_VERSION: int -def create_ring(n: int, q_hex: str, digit_bits: int, fast: bool = False) -> object: ... +def create_ring( + n: int, q_hex: str, digit_bits: int, fast: bool = False, residue: bool = False +) -> object: ... def compile_key(ring: object, key: tuple[tuple[bytes, bytes], ...]) -> object: ... def apply_key(key: object, poly: bytes) -> tuple[bytes, bytes]: ... def ring_product(ring: object, lhs: bytes, rhs: bytes) -> bytes: ... diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h new file mode 100644 index 0000000..61d7ab6 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h @@ -0,0 +1,238 @@ +// Persistent RNS ciphertext arithmetic for Q = p_0 * ... * p_{L-1}. +// Independently implemented. Binary gadget keys and exact BFV rounding are +// unchanged, so the GMP evaluator is a coefficient-for-coefficient oracle. +#pragma once +#include "bfv_server.h" + +namespace xtrace_bfv { +using ResidueCiphertext = std::array; +using CoefficientWords = std::array; // Q has at most 480 bits here. + +class ResidueArithmetic { + const Ring& ring_; + // Inverses p_k^-1 mod p_j for Garner's mixed-radix reconstruction, k < j. + std::vector> inverses_; + +public: + const std::size_t count; + explicit ResidueArithmetic(const Ring& ring) : ring_(ring), count(ring.residue_prime_count) { + if (!count || count > 8) throw std::invalid_argument("Missing persistent RNS modulus"); + for (std::size_t j = 0; j < count; ++j) { + inverses_.emplace_back(); + Word p = ring.transforms[j].modulus; + for (std::size_t k = 0; k < j; ++k) + inverses_.back().emplace_back(power_mod(ring.transforms[k].modulus % p, p - 2, p), p); + } + } + + Residues split(const Polynomial& poly) const { + ProfileScope timing(Phase::to_residues); + Residues result(count, std::vector(ring_.n)); + for (std::size_t j = 0; j < count; ++j) + for (std::size_t i = 0; i < ring_.n; ++i) + result[j][i] = mpz_fdiv_ui(poly[i].get_mpz_t(), ring_.transforms[j].modulus); + return result; + } + + // Recover the UNIQUE representative in [0,Q), not a centered or approximate + // CRT lift. Gadget digits depend on that choice. All arithmetic below uses + // 64/128-bit words, with no floating point, big-integer division or GMP. + std::vector compose_words(const Residues& poly) const { + ProfileScope timing(Phase::rns_compose); + std::vector result(ring_.n); + for (std::size_t i = 0; i < ring_.n; ++i) { + std::array digits{}; + for (std::size_t j = 0; j < count; ++j) { + Word p = ring_.transforms[j].modulus, value = poly[j][i]; + for (std::size_t k = 0; k < j; ++k) { + // Every p is in (2^59,2^60): a mixed-radix digit from an + // earlier prime needs at most one subtraction modulo p. + Word d = digits[k] >= p ? digits[k] - p : digits[k]; + value = inverses_[j][k].multiply(value + p - d, p); + } + digits[j] = value; + } + auto& words = result[i]; + words[0] = digits[count - 1]; + std::size_t used = 1; + for (std::size_t j = count - 1; j > 0; --j) { + Word carry = digits[j - 1], p = ring_.transforms[j - 1].modulus; + for (std::size_t limb = 0; limb < used; ++limb) { + Wide product = Wide(words[limb]) * p + carry; + words[limb] = Word(product); + carry = Word(product >> 64); + } + if (carry) words[used++] = carry; + } + } + return result; + } + + Polynomial compose(const Residues& poly) const { + auto words = compose_words(poly); + ProfileScope timing(Phase::crt_reconstruct); + Polynomial result(ring_.n); + for (std::size_t i = 0; i < ring_.n; ++i) + mpz_import(result[i].get_mpz_t(), 8, -1, sizeof(Word), 0, 0, words[i].data()); + return result; + } + + static Word extract(const CoefficientWords& value, unsigned bit, unsigned width) { + // width <= 60, and the caller never asks for a bit above Q's bit length. + auto limb = bit / 64, offset = bit % 64; + Word result = value[limb] >> offset; + if (offset && limb + 1 < value.size()) result |= value[limb + 1] << (64 - offset); + return result & ((Word(1) << width) - 1); + } + + // Also support wider binary gadgets. Reduction consumes at most 60 bits + // per step, so each intermediate fits 120 bits even for a 480-bit digit. + Word digit_mod(const CoefficientWords& value, unsigned start, unsigned bits, std::size_t j) const { + auto q_bits = unsigned(60 * count); + unsigned width = std::min(bits, q_bits - start); + const auto& transform = ring_.transforms[j]; + if (width <= 60) { + Word result = extract(value, start, width); + return result >= transform.modulus ? result - transform.modulus : result; + } + Word result = 0; + while (width) { + unsigned take = std::min(width, 60U); + result = transform.reduce((Wide(result) << take) | extract(value, start + width - take, take)); + width -= take; + } + return result; + } + + ResidueCiphertext apply(const Residues& poly, const SwitchKey& key) const { + auto words = compose_words(poly); + std::vector> small_digits; + if (ring_.digit_bits <= 60) { + ProfileScope timing(Phase::gadget_decompose); + small_digits.assign(ring_.digits, std::vector(ring_.n)); + for (std::size_t digit = 0; digit < ring_.digits; ++digit) + for (std::size_t i = 0; i < ring_.n; ++i) + small_digits[digit][i] = extract(words[i], digit * ring_.digit_bits, ring_.digit_bits); + } + auto result = key.apply_digits([&](std::size_t j, std::size_t digit, std::vector& values) { + ProfileScope timing(Phase::gadget_decompose); + if (small_digits.empty()) { + for (std::size_t i = 0; i < ring_.n; ++i) + values[i] = digit_mod(words[i], digit * ring_.digit_bits, ring_.digit_bits, j); + } else { + Word p = ring_.transforms[j].modulus; + for (std::size_t i = 0; i < ring_.n; ++i) { + Word value = small_digits[digit][i]; + values[i] = value >= p ? value - p : value; + } + } + }); + for (auto& component : result) + for (std::size_t j = 0; j < count; ++j) ring_.transforms[j].inverse(component[j]); + return result; + } + + void add(ResidueCiphertext& lhs, const ResidueCiphertext& rhs) const { + ProfileScope timing(Phase::add_sub); + for (std::size_t k = 0; k < 2; ++k) + for (std::size_t j = 0; j < count; ++j) { + Word p = ring_.transforms[j].modulus; + for (std::size_t i = 0; i < ring_.n; ++i) { + Word value = lhs[k][j][i] + rhs[k][j][i]; + lhs[k][j][i] = value >= p ? value - p : value; + } + } + } + + ResidueCiphertext rotate(const ResidueCiphertext& value, Word exponent, const SwitchKey& key) const { + ResidueCiphertext permuted; + { ProfileScope timing(Phase::automorphism); + for (std::size_t k = 0; k < 2; ++k) { + permuted[k].assign(count, std::vector(ring_.n)); + for (std::size_t j = 0; j < count; ++j) { + Word p = ring_.transforms[j].modulus; + for (std::size_t i = 0; i < ring_.n; ++i) { + std::size_t at = (i * exponent) & (2 * ring_.n - 1); + Word c = value[k][j][i]; + permuted[k][j][at & (ring_.n - 1)] = at >= ring_.n && c ? p - c : c; + } + } + } } + auto switched = apply(permuted[1], key); + { ProfileScope timing(Phase::add_sub); + for (std::size_t j = 0; j < count; ++j) { + Word p = ring_.transforms[j].modulus; + for (std::size_t i = 0; i < ring_.n; ++i) { + Word c = switched[0][j][i] + permuted[0][j][i]; + switched[0][j][i] = c >= p ? c - p : c; + } + } } + return switched; + } + + void multiply_plain(ResidueCiphertext& value, const Residues& mask_ntt) const { + for (auto& component : value) + for (std::size_t j = 0; j < count; ++j) { + const auto& transform = ring_.transforms[j]; + transform.forward(component[j]); + { ProfileScope timing(Phase::pointwise); + for (std::size_t i = 0; i < ring_.n; ++i) + component[j][i] = transform.remainder(Wide(component[j][i]) * mask_ntt[j][i]); } + transform.inverse(component[j]); + } + } +}; + +class ResidueServer : public HammingServer { + ResidueArithmetic arithmetic_; + Residues full_mask_ntt_; + + ResidueCiphertext tile(const Ciphertext& query, const Ciphertext& indexed, const Residues& selected) const { + Ciphertext difference = query; + { ProfileScope timing(Phase::add_sub); + for (std::size_t k = 0; k < 2; ++k) + for (std::size_t i = 0; i < ring->n; ++i) { + difference[k][i] -= indexed[k][i]; + if (difference[k][i] < 0) difference[k][i] += ring->q; + } } + // One exact tensor and scale/round per tile still uses the original + // auxiliary base and GMP. After this boundary, ciphertexts remain RNS + // through relinearization, all rotations, masks and the merge tree. + auto product = ring->multiply(difference, difference, t, true); + auto distance = arithmetic_.apply(arithmetic_.split(product[2]), *relin_); + arithmetic_.add(distance, {arithmetic_.split(product[0]), arithmetic_.split(product[1])}); + for (auto g : left_) { + auto rotated = arithmetic_.rotate(distance, g, key(g)); + arithmetic_.add(distance, rotated); + } + arithmetic_.multiply_plain(distance, selected); + return distance; + } + +public: + ResidueServer(KeyHandle relin, std::map keys, + std::size_t padded, Word t, const mpz_class& target) + : HammingServer(std::move(relin), std::move(keys), padded, t, target, false), + arithmetic_(*ring), full_mask_ntt_(ring->encode(mask(capacity), arithmetic_.count)) {} + + void search(const Ciphertext& query, std::size_t count, ReadTile read_tile, Emit emit, bool compact_result) const override { + const auto tiles = (count + capacity - 1) / capacity; + Residues partial; + if (count % capacity) partial = ring->encode(mask(count % capacity), arithmetic_.count); + merge_hamming_tiles(tiles, padded, [&](std::size_t at) { + const auto& selected = at + 1 == tiles && !partial.empty() ? partial : full_mask_ntt_; + return tile(query, read_tile(at), selected); + }, [&](const ResidueCiphertext& result, std::size_t level) { + return arithmetic_.rotate(result, right_.at(level), key(right_.at(level))); + }, [&](ResidueCiphertext& lhs, const ResidueCiphertext& rhs) { + arithmetic_.add(lhs, rhs); + }, [&](ResidueCiphertext result) { + Ciphertext output{arithmetic_.compose(result[0]), arithmetic_.compose(result[1])}; + if (compact_result) compact(output); + emit(output); + }); + } + + std::size_t bytes() const override { return HammingServer::bytes() + arithmetic_.count * ring->n * sizeof(Word); } +}; +} // namespace xtrace_bfv diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h index f5e358d..3694c88 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h @@ -4,11 +4,46 @@ #pragma once #include "rns_ntt.h" #include +#include #include namespace xtrace_bfv { using KeyHandle = std::shared_ptr; +// The GMP and persistent-residue servers share the same merge order, including +// partial groups. Only their arithmetic and intermediate ciphertext type differ. +template +void merge_hamming_tiles(std::size_t tiles, std::size_t padded, Tile tile, Rotate rotate, + Add add, Emit emit) { + for (std::size_t start = 0; start < tiles; start += padded) { + std::vector> stack; + for (std::size_t at = start; at < std::min(start + padded, tiles); ++at) { + auto result = tile(at); + std::size_t size = 1, level = 0; + while (!stack.empty() && stack.back().second == size) { + auto rotated = rotate(result, level); + result = std::move(stack.back().first); + stack.pop_back(); + add(result, rotated); + size *= 2; ++level; + } + stack.emplace_back(std::move(result), size); + } + auto result = std::move(stack.back().first); + stack.pop_back(); + while (!stack.empty()) { + auto size = stack.back().second; + std::size_t level = 0; + while ((std::size_t(1) << level) < size) ++level; + auto rotated = rotate(result, level); + result = std::move(stack.back().first); + stack.pop_back(); + add(result, rotated); + } + emit(std::move(result)); + } +} + class PreparedPlain { const Ring& ring_; Residues value_; @@ -29,6 +64,7 @@ class PreparedPlain { }; class HammingServer { +protected: KeyHandle relin_; std::map keys_; PrimeNTT plaintext_; @@ -69,7 +105,7 @@ class HammingServer { const mpz_class target; HammingServer(KeyHandle relin, std::map keys, - std::size_t padded, Word t, const mpz_class& target) + std::size_t padded, Word t, const mpz_class& target, bool prepare_mask = true) : relin_(std::move(relin)), keys_(std::move(keys)), plaintext_(relin_->ring->n, t, relin_->ring->fast), ring(relin_->ring), padded(padded), lanes(ring->n / (2 * padded)), capacity(2 * lanes), t(t), target(target) { @@ -79,9 +115,11 @@ class HammingServer { right_.push_back(exponent(ring->n / 2 - lanes * size)); key(left_.back()); key(right_.back()); } - full_mask_ = std::make_unique(*ring, mask(capacity)); + if (prepare_mask) full_mask_ = std::make_unique(*ring, mask(capacity)); } + virtual ~HammingServer() = default; + Ciphertext tile(const Ciphertext& query, const Ciphertext& indexed, const PreparedPlain& selected) const { Ciphertext difference = query; { ProfileScope timing(Phase::add_sub); @@ -114,41 +152,24 @@ class HammingServer { // read_tile/emit allow the binding to stream packed bytes with the GIL // released. At most O(log(padded)) intermediate ciphertexts are retained. - template - void search(const Ciphertext& query, std::size_t count, ReadTile read_tile, Emit emit, bool compact_result) const { + using ReadTile = std::function; + using Emit = std::function; + virtual void search(const Ciphertext& query, std::size_t count, ReadTile read_tile, Emit emit, bool compact_result) const { const auto tiles = (count + capacity - 1) / capacity; std::unique_ptr partial; if (count % capacity) partial = std::make_unique(*ring, mask(count % capacity)); - for (std::size_t start = 0; start < tiles; start += padded) { - std::vector> stack; - for (std::size_t at = start; at < std::min(start + padded, tiles); ++at) { - const auto& selected = at + 1 == tiles && partial ? *partial : *full_mask_; - auto result = tile(query, read_tile(at), selected); - std::size_t size = 1, level = 0; - while (!stack.empty() && stack.back().second == size) { - auto rotated = rotate(result, right_.at(level), key(right_[level])); - result = std::move(stack.back().first); - stack.pop_back(); - add_inplace(result, rotated, *ring); - size *= 2; ++level; - } - stack.emplace_back(std::move(result), size); - } - auto result = std::move(stack.back().first); - stack.pop_back(); - while (!stack.empty()) { - auto size = stack.back().second; - std::size_t level = 0; - while ((std::size_t(1) << level) < size) ++level; - auto rotated = rotate(result, right_.at(level), key(right_[level])); - result = std::move(stack.back().first); - stack.pop_back(); - add_inplace(result, rotated, *ring); - } + merge_hamming_tiles(tiles, padded, [&](std::size_t at) { + const auto& selected = at + 1 == tiles && partial ? *partial : *full_mask_; + return tile(query, read_tile(at), selected); + }, [&](const Ciphertext& result, std::size_t level) { + return rotate(result, right_.at(level), key(right_.at(level))); + }, [&](Ciphertext& lhs, const Ciphertext& rhs) { + add_inplace(lhs, rhs, *ring); + }, [&](Ciphertext result) { if (compact_result) compact(result); emit(result); - } + }); } - std::size_t bytes() const { return plaintext_.bytes() + full_mask_->bytes(); } + virtual std::size_t bytes() const { return plaintext_.bytes() + (full_mask_ ? full_mask_->bytes() : 0); } }; } // namespace xtrace_bfv diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp index 1a5efcb..6e09bae 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp @@ -3,6 +3,7 @@ #include #include "rns_ntt.h" #include "bfv_server.h" +#include "bfv_residue.h" #include "packed_wire.h" #include @@ -107,15 +108,15 @@ PyObject* create_ring(PyObject*, PyObject* args) { return checked([&]() -> PyObject* { PyObject *n_object, *bits_object; const char* q_hex; - int fast = 0; - if (!PyArg_ParseTuple(args, "OsO|p", &n_object, &q_hex, &bits_object, &fast)) return nullptr; + int fast = 0, residue = 0; + if (!PyArg_ParseTuple(args, "OsO|pp", &n_object, &q_hex, &bits_object, &fast, &residue)) return nullptr; auto n = PyLong_AsUnsignedLongLong(n_object), bits = PyLong_AsUnsignedLongLong(bits_object); if (PyErr_Occurred()) return nullptr; if (n > 32768 || bits > 512) throw std::invalid_argument("Invalid native BFV ring parameters"); mpz_class q; if (q.set_str(q_hex, 16) != 0) throw std::invalid_argument("Invalid ciphertext modulus"); RingPtr ring; - { WithoutGIL release; ring = std::make_shared(n, q, bits, fast); } + { WithoutGIL release; ring = std::make_shared(n, q, bits, fast, residue); } return capsule(std::move(ring), ring_name); }); } @@ -246,10 +247,11 @@ PyObject* ring_info(PyObject*, PyObject* argument) { if (!value) { Py_DECREF(primes); return nullptr; } PyTuple_SET_ITEM(primes, i, value); } - return Py_BuildValue("{s:N,s:n,s:n,s:s,s:s,s:O}", "primes", primes, + return Py_BuildValue("{s:N,s:n,s:n,s:s,s:s,s:O,s:n}", "primes", primes, "switch_prime_count", ring->switch_prime_count, "transform_table_bytes", ring->bytes(), "gmp", gmp_version, - "compiler", __VERSION__, "fast_arithmetic", ring->fast ? Py_True : Py_False); + "compiler", __VERSION__, "fast_arithmetic", ring->fast ? Py_True : Py_False, + "residue_prime_count", ring->residue_prime_count); }); } @@ -286,7 +288,10 @@ PyObject* create_server(PyObject*, PyObject* args) { if (!keys.emplace(g, std::move(key)).second) throw std::invalid_argument("Duplicate rotation key"); } ServerPtr server; - { WithoutGIL release; server = std::make_shared(relin, std::move(keys), padded, t, target); } + { WithoutGIL release; + if (ring.residue_prime_count) + server = std::make_shared(relin, std::move(keys), padded, t, target); + else server = std::make_shared(relin, std::move(keys), padded, t, target); } return capsule(std::move(server), server_name); }); } @@ -405,7 +410,7 @@ PyModuleDef module = {PyModuleDef_HEAD_INIT, "_bfv_rns", "Native BFV RNS/NTT CPU PyMODINIT_FUNC PyInit__bfv_rns() { PyObject* result = PyModule_Create(&module); - if (result && PyModule_AddIntConstant(result, "ABI_VERSION", 2) < 0) { + if (result && PyModule_AddIntConstant(result, "ABI_VERSION", 3) < 0) { Py_DECREF(result); return nullptr; } return result; diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h index 0bb1290..6ddd7ce 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h @@ -8,11 +8,11 @@ namespace xtrace_bfv { enum class Phase : std::size_t { forward_ntt, inverse_ntt, to_residues, crt_reconstruct, crt_exact_fallback, gadget_decompose, - pointwise, scale_round, mod_q, buffers, automorphism, add_sub, wire_import, wire_export, other, count + pointwise, scale_round, mod_q, buffers, automorphism, add_sub, wire_import, wire_export, rns_compose, other, count }; inline constexpr const char* phase_names[] = { "forward_ntt", "inverse_ntt", "to_residues", "crt_reconstruct", "crt_exact_fallback", "gadget_decompose", - "pointwise", "scale_round", "mod_q", "buffers", "automorphism", "add_sub", "wire_import", "wire_export", "other" + "pointwise", "scale_round", "mod_q", "buffers", "automorphism", "add_sub", "wire_import", "wire_export", "rns_compose", "other" }; struct NativeProfile { using Clock = std::chrono::steady_clock; diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h index b3a4543..3347fb9 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h @@ -214,8 +214,9 @@ class Ring { const bool fast; std::vector transforms; std::size_t switch_prime_count; + std::size_t residue_prime_count = 0; - Ring(std::size_t n, const mpz_class& q, unsigned bits, bool fast = false) + Ring(std::size_t n, const mpz_class& q, unsigned bits, bool fast = false, bool residue = false) : n(n), coefficient_bytes((mpz_sizeinbase(q.get_mpz_t(), 2) + 7) / 8), digits(bits ? (mpz_sizeinbase(q.get_mpz_t(), 2) + bits - 1) / bits : 0), digit_bits(bits), q(q), fast(fast) { @@ -259,6 +260,16 @@ class Ring { bases_.push_back(std::move(base)); } switch_prime_count = prime_count(switch_bound); + if (residue) { + // The residue backend uses the first L primes as the actual BFV + // modulus Q. Products modulo Q then need only those L transforms; + // the larger auxiliary base is still needed for BFV scale/round. + for (std::size_t count = 1; count <= bases_.size(); ++count) + if (bases_[count - 1].product == q) residue_prime_count = count; + if (!residue_prime_count || residue_prime_count > 8) + throw std::invalid_argument("Residue backend requires Q to be a product of the first 60-bit NTT primes"); + switch_prime_count = residue_prime_count; + } } std::size_t prime_count(const mpz_class& bound) const { @@ -416,9 +427,6 @@ class SwitchKey { std::array apply(const Polynomial& poly) const { const auto& r = *ring; - std::array output; - { ProfileScope allocation(Phase::buffers); - for (auto& component : output) component.assign(r.switch_prime_count, std::vector(r.n)); } // Decompose once, then reuse each integer digit across auxiliary primes. std::vector decomposition; std::vector> small_digits; @@ -447,22 +455,38 @@ class SwitchKey { } } } + auto output_ntt = apply_digits([&](std::size_t j, std::size_t digit, std::vector& digit_values) { + Word p = r.transforms[j].modulus; + for (std::size_t i = 0; i < r.n; ++i) { + if (small_digits.empty()) + digit_values[i] = mpz_fdiv_ui(decomposition[digit][i].get_mpz_t(), p); + else { + Word value = small_digits[digit][i]; + digit_values[i] = value >= p ? value - p : value; + } + } + }); + std::array result; + for (std::size_t k = 0; k < 2; ++k) + result[k] = r.decode_mod_q(std::move(output_ntt[k])); + return result; + } + + // Share the cached key dot product with the persistent-residue evaluator. + // Its digit reader uses fixed-size limbs, without constructing GMP objects. + // Both readers must supply canonical residues in [0,p). + template + std::array apply_digits(ReadDigit read_digit) const { + const auto& r = *ring; + std::array output; + { ProfileScope allocation(Phase::buffers); + for (auto& component : output) component.assign(r.switch_prime_count, std::vector(r.n)); } { ProfileScope timing(Phase::pointwise); for (std::size_t j = 0; j < r.switch_prime_count; ++j) { - Word p = r.transforms[j].modulus; std::array, 2> accum{std::vector(r.n), std::vector(r.n)}; std::vector digit_values(r.n); for (std::size_t digit = 0; digit < r.digits; ++digit) { - for (std::size_t i = 0; i < r.n; ++i) { - if (small_digits.empty()) - digit_values[i] = mpz_fdiv_ui(decomposition[digit][i].get_mpz_t(), p); - else { - // Both the digit and p are at most 60 bits; p is close - // to 2^60, so one conditional subtraction suffices. - Word value = small_digits[digit][i]; - digit_values[i] = value >= p ? value - p : value; - } - } + read_digit(j, digit, digit_values); r.transforms[j].forward(digit_values); for (std::size_t k = 0; k < 2; ++k) for (std::size_t i = 0; i < r.n; ++i) @@ -477,11 +501,7 @@ class SwitchKey { for (std::size_t i = 0; i < r.n; ++i) output[k][j][i] = r.transforms[j].remainder(accum[k][i]); } } - std::array result; - for (std::size_t k = 0; k < 2; ++k) { - result[k] = r.decode_mod_q(std::move(output[k])); - } - return result; + return output; } std::size_t bytes() const { return ring->digits * 2 * ring->switch_prime_count * ring->n * sizeof(Word); } diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py index f37e72d..a55f730 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py @@ -1,7 +1,7 @@ """Native, leveled BFV over Z_q[X]/(X^N + 1), using GMP integer arithmetic. -The implementation deliberately keeps a single large ciphertext modulus. GMP -Kronecker substitution handles polynomial products; a small-modulus NTT handles +The default uses a single prime ciphertext modulus. An opt-in product of 60-bit +primes also supports the residue server backend. GMP Kronecker substitution handles polynomial products; a small-modulus NTT handles plaintext batching. No SEAL/TenSEAL code is called. This is an experimental, variable-time implementation for learning and optimization, not audited crypto. """ @@ -127,6 +127,46 @@ def _coefficient_modulus(bits: int) -> mpz: return candidate +@lru_cache(maxsize=16) +def _rns_coefficient_primes(n: int, bits: int) -> tuple[int, ...]: + """Deterministic 60-bit NTT primes, in the native Ring's auxiliary-base order. + + This first residue layout supports 60, 120, ..., 480 modulus bits. It does + not change the gadget, error distribution, encryption or rounding rules. + """ + if bits < 60 or bits > 480 or bits % 60: + raise ValueError( + "rns_modulus requires coeff_modulus_bits to be a multiple of 60 in [60, 480]" + ) + result = [] + candidate = (1 << 60) - 2 * n + 1 + for _ in range(bits // 60): + while not gmpy2.is_prime(candidate): + candidate -= 2 * n + if candidate <= 1 << 59: + raise ValueError("No suitable 60-bit RNS prime") + result.append(candidate) + candidate -= 2 * n + return tuple(result) + + +def _parameter_dict(params: BFVParameters) -> dict[str, Any]: + # Preserve the original v1 key fingerprints and JSON when the option is off. + result = asdict(params) + if not params.rns_modulus: + del result["rns_modulus"] + return result + + +def _parameter_modulus(params: BFVParameters) -> mpz: + if params.rns_modulus: + q = mpz(1) + for prime in _rns_coefficient_primes(params.poly_modulus_degree, params.coeff_modulus_bits): + q *= prime + return q + return _coefficient_modulus(params.coeff_modulus_bits) + + def _small_poly(n: int, eta: int, q: mpz) -> BFVPolynomial: # Centered binomial: sum of eta independent bits minus another eta bits. # Each sample uses OS-backed randomness; never use benchmark PRNGs here. @@ -141,7 +181,7 @@ def _ternary_poly(n: int, q: mpz) -> BFVPolynomial: def _key_id(params: BFVParameters, b: BFVPolynomial, a: BFVPolynomial) -> str: - digest = hashlib.sha256(json.dumps(asdict(params), sort_keys=True).encode()) + digest = hashlib.sha256(json.dumps(_parameter_dict(params), sort_keys=True).encode()) width = (params.coeff_modulus_bits + 7) // 8 for poly in (b, a): for c in poly: @@ -163,7 +203,11 @@ def __init__(self, keys: BFVKeyPair) -> None: @staticmethod def validate_parameters(params: BFVParameters) -> None: """Check algebraic constraints, independently of a security assessment.""" - if any(not isinstance(v, int) or isinstance(v, bool) for v in asdict(params).values()): + integer_params = asdict(params) + rns_modulus = integer_params.pop("rns_modulus") + if type(rns_modulus) is not bool: + raise ValueError("rns_modulus must be a boolean") + if any(not isinstance(v, int) or isinstance(v, bool) for v in integer_params.values()): raise ValueError("BFV parameters must be integers") n, t = params.poly_modulus_degree, params.plain_modulus if n < 8 or n > 32768 or n & (n - 1): @@ -176,6 +220,8 @@ def validate_parameters(params: BFVParameters) -> None: raise ValueError("decomposition_bits must be in [1, coeff_modulus_bits]") if not 1 <= params.error_eta <= 64: raise ValueError("error_eta must be in [1, 64]") + if rns_modulus: + _rns_coefficient_primes(n, params.coeff_modulus_bits) @staticmethod def key_gen( @@ -186,20 +232,27 @@ def key_gen( error_eta: int = 21, rotation_steps: Sequence[int] = (), relinearization: bool = True, + rns_modulus: bool = False, ) -> BFVKeyPair: """Generate public/secret keys and requested public evaluation keys. :param rotation_steps: Left rotations within each of the two N/2-slot rows. Negative steps rotate right. Only requested rotations get keys. :param relinearization: Generate the key needed to reduce products to two polynomials. + :param rns_modulus: Use a product of 60-bit NTT primes; requires fresh keys and ciphertexts. :return: PK/SK pair; only the PK (including evaluation keys) goes to the server. :rtype: BFVKeyPair """ params = BFVParameters( - poly_modulus_degree, plain_modulus, coeff_modulus_bits, decomposition_bits, error_eta + poly_modulus_degree, + plain_modulus, + coeff_modulus_bits, + decomposition_bits, + error_eta, + rns_modulus, ) BFV.validate_parameters(params) - n, q = poly_modulus_degree, _coefficient_modulus(coeff_modulus_bits) + n, q = poly_modulus_degree, _parameter_modulus(params) s = _ternary_poly(n, q) a = tuple(mpz(secrets.randbelow(int(q))) for _ in range(n)) error = _small_poly(n, error_eta, q) @@ -518,9 +571,9 @@ def modulus_switch( <= ciphertext.modulus.bit_length() ): raise ValueError("Invalid target modulus bit length") - target = _coefficient_modulus(modulus_bits) - if target > ciphertext.modulus: - raise ValueError("Modulus switching cannot increase the modulus") + target = min(_coefficient_modulus(modulus_bits), ciphertext.modulus) + # With a product modulus, the same-bit prime can be larger than Q. + # An equal-bit request is a no-op; it must never raise the modulus. components = tuple( tuple(_round_div(c * target, ciphertext.modulus) % target for c in poly) for poly in ciphertext.components @@ -606,7 +659,7 @@ def switch(values: BFVSwitchKey) -> list[list[list[str]]]: return json.dumps( { "version": 1, - "params": asdict(pk["params"]), + "params": _parameter_dict(pk["params"]), "key_id": pk["key_id"], "b": poly(pk["b"]), "a": poly(pk["a"]), @@ -629,7 +682,7 @@ def deserialize_public_key(value: str) -> BFVPublicKey: raise ValueError("Unsupported public key version") params = BFVParameters(**data["params"]) BFV.validate_parameters(params) - n, q = params.poly_modulus_degree, _coefficient_modulus(params.coeff_modulus_bits) + n, q = params.poly_modulus_degree, _parameter_modulus(params) digits = ( params.coeff_modulus_bits + params.decomposition_bits - 1 ) // params.decomposition_bits diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py index ee358e4..4214857 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py @@ -21,7 +21,7 @@ BFVSwitchKey, ) -BFVServerBackend = Literal["optimized", "reference", "rns", "native"] +BFVServerBackend = Literal["optimized", "reference", "rns", "native", "residue"] @dataclass(frozen=True) @@ -77,11 +77,13 @@ class BFVEvaluator: ``optimized`` lazily caches packed evaluation keys and fuses gadget products; ``reference`` delegates to the original BFV routines. ``rns`` uses the optional native C++ RNS/NTT kernels. ``native`` enables their faster exact - arithmetic and lets BFVClient run complete searches inside C++. All backends return identical - ciphertexts. Reuse one evaluator across queries to amortize preparation. + arithmetic and lets BFVClient run complete searches inside C++. ``residue`` + keeps those searches in RNS after tensor scale-and-round; it requires an + RNS product modulus. All backends return identical ciphertexts for the same + keys and inputs. Reuse one evaluator across queries to amortize preparation. :param pk: Valid public key from key generation or public-key deserialization. - :param backend: ``optimized`` (default), ``reference``, ``rns``, or ``native``. + :param backend: ``optimized`` (default), ``reference``, ``rns``, ``native``, or ``residue``. The key dictionaries are snapshotted; immutable polynomial tuples are shared. Construct a new evaluator when changing keys. The cache is bounded by the @@ -89,15 +91,19 @@ class BFVEvaluator: """ def __init__(self, pk: BFVPublicKey, backend: BFVServerBackend = "optimized") -> None: - if backend not in ("optimized", "reference", "rns", "native"): - raise ValueError("server_backend must be 'optimized', 'reference', 'rns', or 'native'") + if backend not in ("optimized", "reference", "rns", "native", "residue"): + raise ValueError( + "server_backend must be 'optimized', 'reference', 'rns', 'native', or 'residue'" + ) self.backend = backend self._pk: BFVPublicKey = {**pk, "galois_keys": dict(pk["galois_keys"])} # Exponent zero identifies relinearization; Galois exponents are odd. self._switch_keys: dict[int, _PackedSwitchKey] = {} self._rns = ( - BFVRNSArithmetic(self._pk, fast=backend == "native") - if backend in ("rns", "native") + BFVRNSArithmetic( + self._pk, fast=backend in ("native", "residue"), residue=backend == "residue" + ) + if backend in ("rns", "native", "residue") else None ) diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py index bc08864..798b8d7 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py @@ -37,7 +37,7 @@ def __init__( <= pk["q"].bit_length() ): raise ValueError("Invalid target modulus bit length") - self._target = _coefficient_modulus(response_modulus_bits) + self._target = min(_coefficient_modulus(response_modulus_bits), pk["q"]) self._key_id = int(pk["key_id"], 16) self._packed_bytes = (arithmetic.n * pk["q"].bit_length() + 7) // 8 self._capacity = arithmetic.n // padded_embed_len diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py index f358aaf..2940637 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py @@ -1,7 +1,8 @@ """Optional native RNS/NTT arithmetic with the existing BFV polynomial types. -Auxiliary primes are temporary arithmetic workspaces. The BFV ciphertext modulus, -keys and wire representation remain the same as the Python/GMP implementation. +The default uses auxiliary primes as temporary arithmetic workspaces. The +residue option uses a subset as the actual ciphertext modulus and keeps complete +search intermediates in that basis. Both match Python/GMP for the same keys. """ from typing import Any @@ -15,7 +16,7 @@ class BFVRNSArithmetic: """Bind optional C++ kernels to one validated BFV public-key context.""" - def __init__(self, pk: BFVPublicKey, *, fast: bool = False) -> None: + def __init__(self, pk: BFVPublicKey, *, fast: bool = False, residue: bool = False) -> None: try: from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns except ImportError as error: @@ -24,15 +25,19 @@ def __init__(self, pk: BFVPublicKey, *, fast: bool = False) -> None: "make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON=/path/to/python; " "or select server_backend='optimized' for the Python/GMP evaluator." ) from error - if _bfv_rns.ABI_VERSION != 2: + if _bfv_rns.ABI_VERSION != 3: raise ImportError("BFV RNS CPU extension has an incompatible ABI; rebuild it") self._native = _bfv_rns self._pk = pk params = pk["params"] + if residue and not params.rns_modulus: + raise ValueError( + "The residue backend requires fresh keys generated with rns_modulus=True" + ) self.n = params.poly_modulus_degree self.width = (pk["q"].bit_length() + 7) // 8 self._ring = _bfv_rns.create_ring( - self.n, format(pk["q"], "x"), params.decomposition_bits, fast + self.n, format(pk["q"], "x"), params.decomposition_bits, fast, residue ) self._keys: dict[int, object] = {} diff --git a/src/xtrace_sdk/x_vec/utils/xtrace_types.py b/src/xtrace_sdk/x_vec/utils/xtrace_types.py index fff6946..327bc1e 100644 --- a/src/xtrace_sdk/x_vec/utils/xtrace_types.py +++ b/src/xtrace_sdk/x_vec/utils/xtrace_types.py @@ -79,6 +79,7 @@ class BFVParameters: coeff_modulus_bits: int = 180 decomposition_bits: int = 30 error_eta: int = 21 + rns_modulus: bool = False @dataclass(frozen=True) diff --git a/tests/x_vec/native/run_binding_sanitizers.py b/tests/x_vec/native/run_binding_sanitizers.py index 1cf768c..bca110d 100644 --- a/tests/x_vec/native/run_binding_sanitizers.py +++ b/tests/x_vec/native/run_binding_sanitizers.py @@ -22,4 +22,6 @@ import pytest # noqa: E402 -raise SystemExit(pytest.main(["tests/x_vec/test_bfv_native.py", "-q"])) +raise SystemExit( + pytest.main(["tests/x_vec/test_bfv_native.py", "tests/x_vec/test_bfv_residue.py", "-q"]) +) diff --git a/tests/x_vec/native/test_bfv_rns.cpp b/tests/x_vec/native/test_bfv_rns.cpp index e773247..fddf3b8 100644 --- a/tests/x_vec/native/test_bfv_rns.cpp +++ b/tests/x_vec/native/test_bfv_rns.cpp @@ -1,6 +1,7 @@ // Standalone kernel oracles; also run under address/undefined-behavior sanitizers. #include "rns_ntt.h" #include "packed_wire.h" +#include "bfv_residue.h" #include #include #include @@ -116,5 +117,67 @@ int main() { values[0] = 0; values[7] = q - 1; assert(import_packed(export_packed(values, q), ring) == values); } - std::cout << "Native NTT, Shoup, signed CRT and gadget oracles passed\n"; + // Persistent RNS: exact canonical Garner lifts and binary gadget digits at + // every supported Q width, including word boundaries and the widest gadget. + Ring primes(8, (mpz_class(1) << 480) - 1, 30, true); + mpz_class q = 1; + for (std::size_t count = 1; count <= 8; ++count) { + q *= primes.transforms[count - 1].modulus; + for (unsigned bits : {1, 15, 30, 60, 61, 127, 480}) { + if (bits > 60 * count) continue; + auto ring = std::make_shared(8, q, bits, true, true); + ResidueArithmetic arithmetic(*ring); + assert(ring->switch_prime_count == count); + Polynomial a(8); + for (auto& c : a) c = big_random.get_z_range(q); + a[0] = 0; a[1] = q - 1; a[2] = q / 2; + a[3] = primes.transforms[0].modulus % q; + a[4] = (mpz_class(1) << (60 * count - 1)) % q; + auto residues = arithmetic.split(a); + assert(arithmetic.compose(residues) == a); + auto words = arithmetic.compose_words(residues); + for (std::size_t digit = 0; digit < ring->digits; ++digit) + for (std::size_t j = 0; j < count; ++j) + for (std::size_t i = 0; i < 8; ++i) { + mpz_class expected = a[i] >> (digit * bits); + mpz_fdiv_r_2exp(expected.get_mpz_t(), expected.get_mpz_t(), bits); + assert(arithmetic.digit_mod(words[i], digit * bits, bits, j) == + mpz_fdiv_ui(expected.get_mpz_t(), ring->transforms[j].modulus)); + } + std::vector key(ring->digits, Ciphertext{Polynomial(8), Polynomial(8)}); + for (auto& pair : key) + for (auto& component : pair) + for (auto& c : component) c = big_random.get_z_range(q); + SwitchKey prepared(ring, key); + Ciphertext expected{Polynomial(8), Polynomial(8)}; + for (std::size_t digit = 0; digit < ring->digits; ++digit) { + Polynomial part(8); + for (std::size_t i = 0; i < 8; ++i) { + part[i] = a[i] >> (digit * bits); + mpz_fdiv_r_2exp(part[i].get_mpz_t(), part[i].get_mpz_t(), bits); + } + for (std::size_t k = 0; k < 2; ++k) { + auto product = schoolbook(part, key[digit][k]); + for (std::size_t i = 0; i < 8; ++i) expected[k][i] += product[i]; + } + } + for (auto& component : expected) + for (auto& c : component) mpz_mod(c.get_mpz_t(), c.get_mpz_t(), q.get_mpz_t()); + NativeProfile profile; + ResidueCiphertext actual; + { ProfileSession session(profile); actual = arithmetic.apply(residues, prepared); } + assert(profile.calls[std::size_t(Phase::crt_reconstruct)] == 0); + assert(profile.calls[std::size_t(Phase::mod_q)] == 0); + assert(arithmetic.compose(actual[0]) == expected[0]); + assert(arithmetic.compose(actual[1]) == expected[1]); + assert(prepared.apply(a) == expected); + for (Word g : {3, 9, 15}) { + auto rotated = arithmetic.rotate(actual, g, prepared); + auto reference = rotate(expected, g, prepared); + assert(arithmetic.compose(rotated[0]) == reference[0]); + assert(arithmetic.compose(rotated[1]) == reference[1]); + } + } + } + std::cout << "Native NTT, signed CRT, gadget and persistent RNS oracles passed\n"; } diff --git a/tests/x_vec/test_bfv_native.py b/tests/x_vec/test_bfv_native.py index 7f95a15..0fa209f 100644 --- a/tests/x_vec/test_bfv_native.py +++ b/tests/x_vec/test_bfv_native.py @@ -12,9 +12,19 @@ native = pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") -@pytest.fixture() -def client() -> BFVClient: - return BFVClient(3, 16, 97, 100, 15, response_modulus_bits=40, server_backend="native") +@pytest.fixture(params=["native", "residue"]) +def client(request) -> BFVClient: + residue = request.param == "residue" + return BFVClient( + 3, + 16, + 97, + 180 if residue else 100, + 15, + response_modulus_bits=40, + server_backend=request.param, + rns_modulus=residue, + ) def test_native_individual_wire_bytes_and_backend_changes(client: BFVClient) -> None: @@ -44,7 +54,7 @@ def test_complete_search_skips_python_polynomial_conversions( query = client.encrypt_vec_one([0, 1, 0]) vectors = [[0, 1, 0], [1, 0, 1], [0, 0, 0]] * 13 index = client.encrypt_vec_packed(vectors) - server = BFVClient(skip_key_gen=True, server_backend="native") + server = BFVClient(skip_key_gen=True, server_backend=client.server_backend) server.load_config(json.loads(client.stringify_config())) server.load_stringified_keys(client.stringify_pk()) assert server.keys is None @@ -98,6 +108,7 @@ def evaluate(_: int): def test_native_rejects_malformed_wire(client: BFVClient) -> None: query = client.encrypt_vec_one([0, 1, 0]) q = int(client._pk()["q"]) + packed_bits = 16 * q.bit_length() changes = [ (0, 0), (1, 2), @@ -107,8 +118,8 @@ def test_native_rejects_malformed_wire(client: BFVClient) -> None: (5, 3), (6, -1), (6, 0.5), - (6, 1 << 1600), - (6, q << 1500), + (6, 1 << packed_bits), + (6, q << (packed_bits - q.bit_length())), ] for offset, value in changes: malformed = query[:] @@ -156,7 +167,7 @@ def test_private_native_boundary_and_plan_lifetime(client: BFVClient) -> None: with pytest.raises((ValueError, OverflowError)): native.packed_search(plan, wire, index, count, True) # Use a fresh ring with no BFVClient owner to test the plan's ownership. - owned = BFVRNSArithmetic(pk) + owned = BFVRNSArithmetic(pk, fast=True, residue=pk["params"].rns_modulus) relin = owned._prepare_key(0, pk["relin_key"]) keys = tuple((g, owned._prepare_key(g, key)) for g, key in pk["galois_keys"].items()) plan = native.create_server(relin, keys, 4, 97, format(evaluator._target, "x")) diff --git a/tests/x_vec/test_bfv_residue.py b/tests/x_vec/test_bfv_residue.py new file mode 100644 index 0000000..63fe2a2 --- /dev/null +++ b/tests/x_vec/test_bfv_residue.py @@ -0,0 +1,109 @@ +"""Persistent RNS ciphertexts: compatibility, exact oracles and public-only search.""" + +import hashlib +import json +from dataclasses import asdict, replace + +import gmpy2 +import pytest + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _rns_coefficient_primes +from xtrace_sdk.x_vec.utils.xtrace_types import BFVParameters + + +def test_default_keys_keep_original_fingerprints_and_serialization() -> None: + keys = BFV.key_gen(16, 97, 100, 15) + pk = keys["pk"] + legacy_params = asdict(pk["params"]) + del legacy_params["rns_modulus"] + digest = hashlib.sha256(json.dumps(legacy_params, sort_keys=True).encode()) + for poly in (pk["b"], pk["a"]): + for c in poly: + digest.update(int(c).to_bytes(13, "little")) + assert pk["key_id"] == digest.hexdigest() + encoded = BFV.serialize_public_key(pk) + assert "rns_modulus" not in json.loads(encoded)["params"] + assert BFV.deserialize_public_key(encoded) == pk + client = BFVClient(3, 16, 97, 100, 15, response_modulus_bits=40) + assert "rns_modulus" not in json.loads(client.stringify_config()) + + +@pytest.mark.parametrize("bits", [60, 120, 180, 480]) +def test_rns_modulus_key_roundtrip_and_equal_bit_compaction(bits: int) -> None: + keys = BFV.key_gen(16, 97, bits, 15, rns_modulus=True) + pk = keys["pk"] + assert pk["q"].bit_length() == bits + primes = _rns_coefficient_primes(16, bits) + assert len(primes) == bits // 60 + assert len(set(primes)) == len(primes) + assert all(gmpy2.is_prime(p) and p % 32 == 1 for p in primes) + product = gmpy2.mpz(1) + for p in primes: + product *= p + assert pk["q"] == product + encoded = BFV.serialize_public_key(pk) + assert json.loads(encoded)["params"]["rns_modulus"] is True + assert BFV.deserialize_public_key(encoded) == pk + assert BFV.deserialize_secret_key(BFV.serialize_secret_key(keys["sk"]), pk) == keys["sk"] + slots = list(range(16)) + ct = BFV.encrypt(BFV.batch_encode(slots, pk["params"]), pk) + assert BFV.modulus_switch(ct, bits, pk) == ct + assert BFV.batch_decode(BFV.decrypt(ct, keys), pk["params"]) == slots + + +@pytest.mark.parametrize( + "changes", + [ + {"rns_modulus": 1}, + {"rns_modulus": "yes"}, + {"coeff_modulus_bits": 100}, + {"coeff_modulus_bits": 512}, + ], +) +def test_rns_parameter_validation(changes) -> None: + with pytest.raises(ValueError): + BFV.validate_parameters( + replace(BFVParameters(16, 97, 180, 15, rns_modulus=True), **changes) + ) + + +@pytest.mark.parametrize("bits,gadget", [(60, 4), (120, 15), (180, 30), (180, 61), (480, 127)]) +def test_residue_search_matches_every_backend(bits: int, gadget: int) -> None: + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + client = BFVClient(3, 16, 97, bits, gadget, response_modulus_bits=40, rns_modulus=True) + # Uneven tile and merge counts, two rows, multiple responses and padding. + vectors = [[0, 1, 0], [1, 0, 1], [0, 0, 0]] * 13 + query = client.encrypt_vec_one([0, 1, 0]) + index = client.encrypt_vec_packed(vectors) + results = [] + for backend in ("reference", "optimized", "rns", "native", "residue"): + server = BFVClient(skip_key_gen=True, server_backend=backend) + server.load_config(json.loads(client.stringify_config())) + server.load_stringified_keys(client.stringify_pk()) + assert server.keys is None + for compact in (False, True): + result = server.encode_hamming_server_packed( + query, index, len(vectors), compact=compact + ) + assert client.decode_hamming_client_packed(result, len(vectors)) == [0, 3, 1] * 13 + results.append(result) + assert server.encode_hamming_server_packed(query, [], 0) == [] + if backend == "residue": + info = server._evaluator().cache_info() + assert info["residue_prime_count"] == info["switch_prime_count"] == bits // 60 + assert all(result == results[0] for result in results[::2]) + assert all(result == results[1] for result in results[1::2]) + + +def test_residue_rejects_existing_prime_modulus_keys() -> None: + native = pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + with pytest.raises(ValueError, match="fresh keys"): + BFVClient(server_backend="residue") + client = BFVClient(3, 16, 97, 100, 15, response_modulus_bits=40) + client.server_backend = "residue" + query = client.encrypt_vec_one([0, 1, 0]) + with pytest.raises(ValueError, match="fresh keys"): + client.encode_hamming_server(query, query) + with pytest.raises(ValueError, match="product"): + native.create_ring(16, format(client._pk()["q"], "x"), 15, True, True) From 06d8d172c168f37dd77d1572f68df2088d2d82d7 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Thu, 10 Sep 2026 23:06:58 -0400 Subject: [PATCH 07/12] Fuse BFV transforms and scale tensor products exactly in RNS --- .github/workflows/ci.yml | 2 +- benchmarks/bfv_server.py | 47 +- benchmarks/results/native_bfv_fused_1024.json | 562 ++++++++++++++++++ benchmarks/results/native_bfv_fused_32.json | 562 ++++++++++++++++++ docs/index.rst | 1 + docs/research/native-bfv-fused-rns.md | 124 ++++ docs/research/native-bfv-residue.md | 4 + .../x_vec/crypto/bfv_cpu_ext/Makefile | 2 +- .../x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi | 3 +- .../x_vec/crypto/bfv_cpu_ext/bfv_residue.h | 25 +- .../x_vec/crypto/bfv_cpu_ext/bindings.cpp | 15 +- .../x_vec/crypto/bfv_cpu_ext/profile.h | 4 +- .../x_vec/crypto/bfv_cpu_ext/rns_ntt.h | 122 +++- .../x_vec/crypto/bfv_cpu_ext/rns_scale.h | 214 +++++++ .../x_vec/crypto/encryption/bfv_evaluator.py | 15 +- .../x_vec/crypto/encryption/bfv_rns.py | 21 +- tests/x_vec/native/test_bfv_rns.cpp | 57 +- tests/x_vec/test_bfv_residue.py | 34 ++ 18 files changed, 1767 insertions(+), 47 deletions(-) create mode 100644 benchmarks/results/native_bfv_fused_1024.json create mode 100644 benchmarks/results/native_bfv_fused_32.json create mode 100644 docs/research/native-bfv-fused-rns.md create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 756b860..f6a1433 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -67,7 +67,7 @@ jobs: sudo apt-get update sudo apt-get install -y libgmp-dev make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" - uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 3' + uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 4' uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py - name: BFV native arithmetic sanitizers diff --git a/benchmarks/bfv_server.py b/benchmarks/bfv_server.py index dcec946..fd9a705 100644 --- a/benchmarks/bfv_server.py +++ b/benchmarks/bfv_server.py @@ -27,7 +27,7 @@ from bfv_client_matrix import REPO_ROOT, make_data, packet, time_call, unpack_packet from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _rns_coefficient_primes -from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVServerBackend +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVEvaluator, BFVServerBackend def main() -> None: @@ -44,6 +44,11 @@ def main() -> None: help="Generate fresh keys with a product of 60-bit NTT primes; required by residue", ) parser.add_argument("--response-modulus-bits", type=int, default=50) + parser.add_argument( + "--residue-levels", + default="2", + help="Residue controls: 0=previous kernels, 1=fused NTT, 2=also exact RNS scaling", + ) parser.add_argument("--seed", type=int, default=1337) parser.add_argument( "--repeats", @@ -88,7 +93,18 @@ def main() -> None: parser.error("native-profile requires the rns, native, or residue backend") if "residue" in names and not args.rns_modulus: parser.error("residue requires --rns-modulus (fresh keys and index)") - backends = tuple(cast(BFVServerBackend, name) for name in names) + levels = args.residue_levels.split(",") + if len(set(levels)) != len(levels) or any(level not in ("0", "1", "2") for level in levels): + parser.error("residue-levels must contain distinct values from 0,1,2") + variants: dict[str, tuple[BFVServerBackend, int | None]] = {} + for name in names: + if name == "residue": + for level in levels: + label = "residue" if len(levels) == 1 else f"residue-{level}" + variants[label] = ("residue", int(level)) + else: + variants[name] = (cast(BFVServerBackend, name), None) + backends = tuple(variants) vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) print("Generating one key set and encrypting the shared input", file=sys.stderr, flush=True) setup: dict[str, float] = {} @@ -111,17 +127,21 @@ def main() -> None: setup["public_key_export_s"], public_json = time_call(client.stringify_pk) config = json.loads(client.stringify_config()) servers = {} - for backend in backends: - server = BFVClient(skip_key_gen=True, server_backend=backend, **config) - setup[f"{backend}_public_key_import_s"], _ = time_call( + for label, (actual_backend, level) in variants.items(): + server = BFVClient(skip_key_gen=True, server_backend=actual_backend, **config) + setup[f"{label}_public_key_import_s"], _ = time_call( partial(server.load_stringified_keys, public_json) ) + setup[f"{label}_arithmetic_context_s"], server._server_evaluator = time_call( + partial(BFVEvaluator, server._pk(), actual_backend, kernel_level=level) + ) + server._evaluator_public_key = server._pk() assert server.keys is None - servers[backend] = server + servers[label] = server key_bytes = len(public_json.encode()) del public_json - def evaluate(backend: BFVServerBackend) -> list[list[int]]: + def evaluate(backend: str) -> list[list[int]]: return servers[backend].encode_hamming_server_packed( server_query, server_index, len(vectors) ) @@ -184,9 +204,7 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: seal_client.noise_budgets(seal_response) ) else: - elapsed, response = time_call( - partial(evaluate, cast(BFVServerBackend, backend_name)) - ) + elapsed, response = time_call(partial(evaluate, backend_name)) if reference_response is None: reference_response = response assert response == reference_response, "Native backend ciphertexts differ" @@ -217,7 +235,7 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns for backend in backends: - if backend in ("rns", "native", "residue"): + if variants[backend][0] in ("rns", "native", "residue"): response, phases = _bfv_rns.profile_call(partial(evaluate, backend)) assert response == reference_response native_phases[backend] = phases @@ -243,7 +261,7 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile", REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", ] - if any(b in ("rns", "native", "residue") for b in backends): + if any(b in ("rns", "native", "residue") for b, _ in variants.values()): source_paths.extend((REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.so")) if args.seal: source_paths.append(REPO_ROOT / "experiments/bfv/packed_hamming.py") @@ -272,7 +290,8 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: ), }, "command": sys.argv, - "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, arithmetic context creation (reported under setup), key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "backend_variants": variants, "config": config, "ciphertext_modulus_hex": format(client._pk()["q"], "x"), "ciphertext_modulus_primes": _rns_coefficient_primes( @@ -316,7 +335,7 @@ def evaluate(backend: BFVServerBackend) -> list[list[int]]: backend: { **server._evaluator().cache_info(), "server_plan_bytes": server._native().cache_bytes() - if backend in ("native", "residue") + if variants[backend][0] in ("native", "residue") else 0, } for backend, server in servers.items() diff --git a/benchmarks/results/native_bfv_fused_1024.json b/benchmarks/results/native_bfv_fused_1024.json new file mode 100644 index 0000000..c2d35ae --- /dev/null +++ b/benchmarks/results/native_bfv_fused_1024.json @@ -0,0 +1,562 @@ +{ + "environment": { + "utc": "2026-09-11T03:04:45.774330+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "e5c075106c91d52d927af65e3d324f516c8c53b0", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "1024", + "--rns-modulus", + "--backends", + "residue", + "--residue-levels", + "0,1,2", + "--seal", + "--repeats", + "4", + "--native-profile", + "--json-out", + "benchmarks/results/native_bfv_fused_1024.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, arithmetic context creation (reported under setup), key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "backend_variants": { + "residue-0": [ + "residue", + 0 + ], + "residue-1": [ + "residue", + 1 + ], + "residue-2": [ + "residue", + 2 + ] + }, + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "ciphertext_modulus_hex": "ffffffffffbc0030000004bff78002fff1004bffbc001", + "ciphertext_modulus_primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137 + ], + "seed": 1337, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 4.47719966317527, + "index_encrypt_s": 6.908126533031464, + "query_encrypt_s": 0.10573444119654596, + "public_key_export_s": 0.8328375099226832, + "residue-0_public_key_import_s": 0.874006818048656, + "residue-0_arithmetic_context_s": 0.001661886926740408, + "residue-1_public_key_import_s": 0.8809227580204606, + "residue-1_arithmetic_context_s": 0.0008880100212991238, + "residue-2_public_key_import_s": 0.8828790578991175, + "residue-2_arithmetic_context_s": 0.0008801498916000128 + }, + "runs": [ + { + "backend": "residue-0", + "repeat": 0, + "server_compute_s": 3.12221663701348 + }, + { + "backend": "residue-1", + "repeat": 0, + "server_compute_s": 2.9633277570828795 + }, + { + "backend": "residue-2", + "repeat": 0, + "server_compute_s": 2.7569567230530083 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 2.458954274887219 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 2.4774899801705033 + }, + { + "backend": "residue-2", + "repeat": 1, + "server_compute_s": 2.4064054479822516 + }, + { + "backend": "residue-1", + "repeat": 1, + "server_compute_s": 2.645552079193294 + }, + { + "backend": "residue-0", + "repeat": 1, + "server_compute_s": 2.7930076429620385 + }, + { + "backend": "residue-0", + "repeat": 2, + "server_compute_s": 2.799179068999365 + }, + { + "backend": "residue-1", + "repeat": 2, + "server_compute_s": 2.645475624827668 + }, + { + "backend": "residue-2", + "repeat": 2, + "server_compute_s": 2.4265225289855152 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 2.4921487709507346 + }, + { + "backend": "seal", + "repeat": 3, + "server_compute_s": 2.4789128969423473 + }, + { + "backend": "residue-2", + "repeat": 3, + "server_compute_s": 2.4004873901139945 + }, + { + "backend": "residue-1", + "repeat": 3, + "server_compute_s": 2.638776435982436 + }, + { + "backend": "residue-0", + "repeat": 3, + "server_compute_s": 2.7942729440983385 + } + ], + "summary": { + "residue-0": { + "first_search_s": 3.12221663701348, + "warm_searches_s": [ + 2.7930076429620385, + 2.799179068999365, + 2.7942729440983385 + ], + "warm_median_s": 2.7942729440983385 + }, + "residue-1": { + "first_search_s": 2.9633277570828795, + "warm_searches_s": [ + 2.645552079193294, + 2.645475624827668, + 2.638776435982436 + ], + "warm_median_s": 2.645475624827668 + }, + "residue-2": { + "first_search_s": 2.7569567230530083, + "warm_searches_s": [ + 2.4064054479822516, + 2.4265225289855152, + 2.4004873901139945 + ], + "warm_median_s": 2.4064054479822516 + }, + "seal": { + "first_search_s": 2.458954274887219, + "warm_searches_s": [ + 2.4774899801705033, + 2.4921487709507346, + 2.4789128969423473 + ], + "warm_median_s": 2.4789128969423473 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "residue-0": { + "first": 1.0, + "warm_median": 1.0 + }, + "residue-1": { + "first": 1.0536183955861202, + "warm_median": 1.0562459611701633 + }, + "residue-2": { + "first": 1.1324866331438053, + "warm_median": 1.1611812741037926 + }, + "seal": { + "first": 1.269733507816725, + "warm_median": 1.1272170746882542 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 27681091, + "query_bytes": 432663, + "public_key_bytes": 39585150, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 105961, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "native_phases": { + "residue-0": { + "forward_ntt": { + "seconds": 1.195090398, + "calls": 13934 + }, + "inverse_ntt": { + "seconds": 0.46048345, + "calls": 5946 + }, + "to_residues": { + "seconds": 0.138519197, + "calls": 320 + }, + "crt_reconstruct": { + "seconds": 0.18281814100000002, + "calls": 194 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.07851556500000001, + "calls": 13357 + }, + "pointwise": { + "seconds": 0.29695649500000004, + "calls": 1151 + }, + "scale_round": { + "seconds": 0.16776511300000002, + "calls": 193 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.017771753, + "calls": 895 + }, + "automorphism": { + "seconds": 0.031365696000000005, + "calls": 639 + }, + "add_sub": { + "seconds": 0.058091784, + "calls": 1406 + }, + "wire_import": { + "seconds": 0.029809831000000002, + "calls": 130 + }, + "wire_export": { + "seconds": 0.00016202300000000002, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.060537628, + "calls": 705 + }, + "rns_scale": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.08418196400000001, + "calls": 1 + } + }, + "residue-1": { + "forward_ntt": { + "seconds": 1.116043798, + "calls": 13934 + }, + "inverse_ntt": { + "seconds": 0.399708159, + "calls": 5946 + }, + "to_residues": { + "seconds": 0.141233763, + "calls": 320 + }, + "crt_reconstruct": { + "seconds": 0.188225854, + "calls": 194 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.07928863300000001, + "calls": 13357 + }, + "pointwise": { + "seconds": 0.298050792, + "calls": 1151 + }, + "scale_round": { + "seconds": 0.173725757, + "calls": 193 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.01836928, + "calls": 895 + }, + "automorphism": { + "seconds": 0.032022763, + "calls": 639 + }, + "add_sub": { + "seconds": 0.059361043, + "calls": 1406 + }, + "wire_import": { + "seconds": 0.030430777000000003, + "calls": 130 + }, + "wire_export": { + "seconds": 0.000163553, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.060993114, + "calls": 705 + }, + "rns_scale": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.085690057, + "calls": 1 + } + }, + "residue-2": { + "forward_ntt": { + "seconds": 1.108225623, + "calls": 13934 + }, + "inverse_ntt": { + "seconds": 0.396151433, + "calls": 5946 + }, + "to_residues": { + "seconds": 0.08412324700000001, + "calls": 128 + }, + "crt_reconstruct": { + "seconds": 0.00049237, + "calls": 2 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.077781504, + "calls": 13357 + }, + "pointwise": { + "seconds": 0.294543106, + "calls": 1151 + }, + "scale_round": { + "seconds": 0.001360969, + "calls": 1 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.020847512000000002, + "calls": 895 + }, + "automorphism": { + "seconds": 0.03390283, + "calls": 639 + }, + "add_sub": { + "seconds": 0.058100396000000006, + "calls": 1406 + }, + "wire_import": { + "seconds": 0.029341752000000002, + "calls": 130 + }, + "wire_export": { + "seconds": 0.000183684, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.060095020000000006, + "calls": 705 + }, + "rns_scale": { + "seconds": 0.21518545200000003, + "calls": 192 + }, + "other": { + "seconds": 0.061856571000000006, + "calls": 1 + } + } + }, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", + "sizes": { + "encrypted_index_bytes": 23598495, + "public_keys_bytes": 85602740, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "caches": { + "residue-0": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "kernel_level": 0, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 589824 + }, + "residue-1": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 1835232, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "kernel_level": 1, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 589824 + }, + "residue-2": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 1835232, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "kernel_level": 2, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 591648 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "ad70ee74af27d055ac9653a58ed46b3c58381aef2de875568cdd15eeb5a9fac6", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "499ed412000271c75c1980a4f2f312ef62d5da069f7f0995957a0de77dd2ea0a", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "0d8b6a02fddf6b58996da91146ad161ee88783475d87edefe93946ae3defb0c6", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "cabbd600ca70b75fa0c608bd3fc6f40dca43a695ddc8e5aeb146277790da36fa", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "48c2b2c7c8fcc8c957009cf328963026b15baf03f765a35cb616ffe02e875c00", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "039e8e26c193be99f4034f6bd39c0b6e5b7b95b1b33c4cd022af0f2dbd358632", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "05e30fc3b8191e2fd0f0620ac7dbbc35ccea0a43913fc62f66575b0d44716aeb", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "e002a06dac9b5f33cecc72dd6d08914417d9882430d829967613bb5d5f661299", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/benchmarks/results/native_bfv_fused_32.json b/benchmarks/results/native_bfv_fused_32.json new file mode 100644 index 0000000..6f8f773 --- /dev/null +++ b/benchmarks/results/native_bfv_fused_32.json @@ -0,0 +1,562 @@ +{ + "environment": { + "utc": "2026-09-11T03:05:33.347461+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "e5c075106c91d52d927af65e3d324f516c8c53b0", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "32", + "--rns-modulus", + "--backends", + "residue", + "--residue-levels", + "0,1,2", + "--seal", + "--repeats", + "4", + "--native-profile", + "--json-out", + "benchmarks/results/native_bfv_fused_32.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, arithmetic context creation (reported under setup), key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "backend_variants": { + "residue-0": [ + "residue", + 0 + ], + "residue-1": [ + "residue", + 1 + ], + "residue-2": [ + "residue", + 2 + ] + }, + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "ciphertext_modulus_hex": "ffffffffffbc0030000004bff78002fff1004bffbc001", + "ciphertext_modulus_primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137 + ], + "seed": 1337, + "vectors": 32, + "setup_timings": { + "key_generation_s": 4.475579569814727, + "index_encrypt_s": 0.21834237920120358, + "query_encrypt_s": 0.1041919409763068, + "public_key_export_s": 0.8049148949794471, + "residue-0_public_key_import_s": 0.8719211141578853, + "residue-0_arithmetic_context_s": 0.0016436860896646976, + "residue-1_public_key_import_s": 0.8792678711470217, + "residue-1_arithmetic_context_s": 0.0009159599430859089, + "residue-2_public_key_import_s": 0.8816855410113931, + "residue-2_arithmetic_context_s": 0.000876349164173007 + }, + "runs": [ + { + "backend": "residue-0", + "repeat": 0, + "server_compute_s": 0.401618508156389 + }, + { + "backend": "residue-1", + "repeat": 0, + "server_compute_s": 0.39036817383021116 + }, + { + "backend": "residue-2", + "repeat": 0, + "server_compute_s": 0.3826875661034137 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 0.08032198995351791 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 0.07701776688918471 + }, + { + "backend": "residue-2", + "repeat": 1, + "server_compute_s": 0.0762762411031872 + }, + { + "backend": "residue-1", + "repeat": 1, + "server_compute_s": 0.082836874993518 + }, + { + "backend": "residue-0", + "repeat": 1, + "server_compute_s": 0.08717664889991283 + }, + { + "backend": "residue-0", + "repeat": 2, + "server_compute_s": 0.08822069107554853 + }, + { + "backend": "residue-1", + "repeat": 2, + "server_compute_s": 0.08271700097247958 + }, + { + "backend": "residue-2", + "repeat": 2, + "server_compute_s": 0.0763571229763329 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 0.08028713893145323 + }, + { + "backend": "seal", + "repeat": 3, + "server_compute_s": 0.07686260412447155 + }, + { + "backend": "residue-2", + "repeat": 3, + "server_compute_s": 0.0761273680254817 + }, + { + "backend": "residue-1", + "repeat": 3, + "server_compute_s": 0.08302503800950944 + }, + { + "backend": "residue-0", + "repeat": 3, + "server_compute_s": 0.08712895796634257 + } + ], + "summary": { + "residue-0": { + "first_search_s": 0.401618508156389, + "warm_searches_s": [ + 0.08717664889991283, + 0.08822069107554853, + 0.08712895796634257 + ], + "warm_median_s": 0.08717664889991283 + }, + "residue-1": { + "first_search_s": 0.39036817383021116, + "warm_searches_s": [ + 0.082836874993518, + 0.08271700097247958, + 0.08302503800950944 + ], + "warm_median_s": 0.082836874993518 + }, + "residue-2": { + "first_search_s": 0.3826875661034137, + "warm_searches_s": [ + 0.0762762411031872, + 0.0763571229763329, + 0.0761273680254817 + ], + "warm_median_s": 0.0762762411031872 + }, + "seal": { + "first_search_s": 0.08032198995351791, + "warm_searches_s": [ + 0.07701776688918471, + 0.08028713893145323, + 0.07686260412447155 + ], + "warm_median_s": 0.07701776688918471 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "residue-0": { + "first": 1.0, + "warm_median": 1.0 + }, + "residue-1": { + "first": 1.0288198041756118, + "warm_median": 1.0523893967117233 + }, + "residue-2": { + "first": 1.0494684011966555, + "warm_median": 1.1429069870128952 + }, + "seal": { + "first": 5.000106551005576, + "warm_median": 1.1319030974417241 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 865029, + "query_bytes": 432641, + "public_key_bytes": 39586570, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 103298, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "native_phases": { + "residue-0": { + "forward_ntt": { + "seconds": 0.035739227000000005, + "calls": 418 + }, + "inverse_ntt": { + "seconds": 0.013498248000000001, + "calls": 180 + }, + "to_residues": { + "seconds": 0.004273492, + "calls": 10 + }, + "crt_reconstruct": { + "seconds": 0.005981719000000001, + "calls": 8 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.002274541, + "calls": 399 + }, + "pointwise": { + "seconds": 0.008908133, + "calls": 35 + }, + "scale_round": { + "seconds": 0.006335980000000001, + "calls": 7 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.000614412, + "calls": 27 + }, + "automorphism": { + "seconds": 0.0008823390000000001, + "calls": 19 + }, + "add_sub": { + "seconds": 0.001745765, + "calls": 42 + }, + "wire_import": { + "seconds": 0.0013987300000000002, + "calls": 6 + }, + "wire_export": { + "seconds": 0.000160513, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.001924134, + "calls": 23 + }, + "rns_scale": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.003255571, + "calls": 1 + } + }, + "residue-1": { + "forward_ntt": { + "seconds": 0.032785657, + "calls": 418 + }, + "inverse_ntt": { + "seconds": 0.011867049000000001, + "calls": 180 + }, + "to_residues": { + "seconds": 0.004252533, + "calls": 10 + }, + "crt_reconstruct": { + "seconds": 0.006173485, + "calls": 8 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.00225163, + "calls": 399 + }, + "pointwise": { + "seconds": 0.008610774, + "calls": 35 + }, + "scale_round": { + "seconds": 0.006365468, + "calls": 7 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.000553022, + "calls": 27 + }, + "automorphism": { + "seconds": 0.0008768980000000001, + "calls": 19 + }, + "add_sub": { + "seconds": 0.0017867780000000002, + "calls": 42 + }, + "wire_import": { + "seconds": 0.00135716, + "calls": 6 + }, + "wire_export": { + "seconds": 0.000165684, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.0019463630000000002, + "calls": 23 + }, + "rns_scale": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.003454454, + "calls": 1 + } + }, + "residue-2": { + "forward_ntt": { + "seconds": 0.033162801, + "calls": 418 + }, + "inverse_ntt": { + "seconds": 0.011793217, + "calls": 180 + }, + "to_residues": { + "seconds": 0.002709368, + "calls": 4 + }, + "crt_reconstruct": { + "seconds": 0.00033013700000000004, + "calls": 2 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.002518695, + "calls": 399 + }, + "pointwise": { + "seconds": 0.008948329000000001, + "calls": 35 + }, + "scale_round": { + "seconds": 0.0012418260000000001, + "calls": 1 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.000535945, + "calls": 27 + }, + "automorphism": { + "seconds": 0.0009641210000000001, + "calls": 19 + }, + "add_sub": { + "seconds": 0.001791826, + "calls": 42 + }, + "wire_import": { + "seconds": 0.00137702, + "calls": 6 + }, + "wire_export": { + "seconds": 0.00016111400000000002, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.001975412, + "calls": 23 + }, + "rns_scale": { + "seconds": 0.006580942, + "calls": 6 + }, + "other": { + "seconds": 0.0026012870000000003, + "calls": 1 + } + } + }, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", + "sizes": { + "encrypted_index_bytes": 737480, + "public_keys_bytes": 85603026, + "query_bytes": 368754, + "response_bytes": 102494, + "query_plus_response_bytes": 471248 + }, + "caches": { + "residue-0": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "kernel_level": 0, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 589824 + }, + "residue-1": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 1835232, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "kernel_level": 1, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 589824 + }, + "residue-2": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 1835232, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "kernel_level": 2, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 591648 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 31, + "distance": 0 + }, + { + "index": 19, + "distance": 241 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "ad70ee74af27d055ac9653a58ed46b3c58381aef2de875568cdd15eeb5a9fac6", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "499ed412000271c75c1980a4f2f312ef62d5da069f7f0995957a0de77dd2ea0a", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "0d8b6a02fddf6b58996da91146ad161ee88783475d87edefe93946ae3defb0c6", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "cabbd600ca70b75fa0c608bd3fc6f40dca43a695ddc8e5aeb146277790da36fa", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "48c2b2c7c8fcc8c957009cf328963026b15baf03f765a35cb616ffe02e875c00", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "039e8e26c193be99f4034f6bd39c0b6e5b7b95b1b33c4cd022af0f2dbd358632", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "05e30fc3b8191e2fd0f0620ac7dbbc35ccea0a43913fc62f66575b0d44716aeb", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "e002a06dac9b5f33cecc72dd6d08914417d9882430d829967613bb5d5f661299", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/docs/index.rst b/docs/index.rst index 909bef0..5986238 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -85,6 +85,7 @@ Choose your path: research/native-bfv-ntt research/native-bfv-server research/native-bfv-residue + research/native-bfv-fused-rns research/bfv-packed-hamming .. toctree:: diff --git a/docs/research/native-bfv-fused-rns.md b/docs/research/native-bfv-fused-rns.md new file mode 100644 index 0000000..1cb87d4 --- /dev/null +++ b/docs/research/native-bfv-fused-rns.md @@ -0,0 +1,124 @@ +# Fused NTT and exact RNS tensor scaling + +The `residue` server now uses fused negacyclic transforms and exact RNS +scale-and-round. Existing product-Q keys and ciphertexts are compatible; the +arithmetic returns the same ciphertext coefficients as the previous kernels. +Rebuild the extension for **ABI 4**. The other backends and the Paillier and +SEAL examples remain available. + +## Changes + +`bfv_cpu_ext/rns_ntt.h` folds the negacyclic twist into stage roots and folds +inverse normalization into the last butterfly. It preserves the existing +bit-reversed spectrum, uses bounded lazy residues, and unrolls groups of four +butterflies. The implementation follows the transform organization discussed by +[Longa and Naehrig](https://eprint.iacr.org/2016/504.pdf) and inspected in +[SEAL 4.1.2's transform handler](https://github.com/microsoft/SEAL/blob/v4.1.2/native/src/seal/util/dwthandler.h). +No SEAL code is linked or called. + +`bfv_cpu_ext/rns_scale.h` adds an independent exact RNS conversion plan. It +removes GMP reconstruction and division from the tensor scaling loop while +preserving our reference's signed rounding. This is related to the +[textbook's RNS BFV multiplication construction](https://fhetextbook.github.io/ApplyingRNSTechniquestoFHEOperations.html), +but retains our existing binary gadget and auxiliary-prime selection. + +For each signed integer tensor coefficient z, write `z = Q*k + r`, with +`0 <= r < Q`. The desired result is exactly +`t*k + floor(t*r/Q + 1/2) mod Q`. The Q residues recover r. In an auxiliary +base B, compute `k = (z-r)/Q mod B`, then lift k with centered base conversion. +The constructor checks `B > 2*(floor(2*N*(Q-1)^2/Q)+1)`, so the quotient's sign +and value are unambiguous for every permitted tensor coefficient. + +Base conversion uses `u_j = r_j*(M/p_j)^-1 mod p_j` and +`theta = sum(u_j/p_j)`. Integer fixed-point intervals certify the floor or +nearest-integer quotient. An interval that crosses a boundary invokes an exact +18-word integer comparison. Scaling splits each `t*u_j/p_j` into its integer +part and a bounded fraction before summing; increasing t does not amplify an +approximation error. There is no floating-point arithmetic. GMP is used for +public plan constants, incoming coefficients and final response conversion. + +## Reproduce + +```bash +make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" +.venv/bin/python benchmarks/bfv_server.py --num-vectors 1024 \ + --rns-modulus --backends residue --residue-levels 0,1,2 --seal \ + --repeats 4 --native-profile --json-out /tmp/bfv-fused-1024.json +``` + +Level 0 retains the previous kernels, level 1 adds the fused transform, and +level 2 also adds RNS scaling. Level 2 is the `residue` default. The controls +are available through `BFVEvaluator(..., kernel_level=...)` for experiments; +they do not change serialized cryptographic parameters. All three receive the +same keys, encrypted query and index. Arithmetic context creation is now +reported separately under setup; cold search still includes lazy evaluation +key and mask preparation. This changes the cold measurement boundary relative +to older reports, but leaves the warm measurement boundary unchanged. + +## Results + +AMD Ryzen 7 5800X, GCC 13.3.0 `-O3`, Python 3.12.3, GMP 6.3.0, +TenSEAL 0.3.16. One CPU evaluation thread, N=8192, t=65537, 180-bit product Q, +30-bit gadget, eta=21, 50-bit response compaction. No simultaneous tests or +other benchmark jobs. Three warm samples follow one cold search; ordering +alternates. Profiles are separate additional searches. + +| 1,024 vectors × 512 dimensions | First search (s) | Warm median (s) | +| --- | ---: | ---: | +| Previous residue kernels | 3.1222 | 2.7943 | +| Fused NTT only | 2.9633 | 2.6455 | +| Fused NTT + exact RNS scaling | 2.7570 | 2.4064 | +| Preserved SEAL prototype | 2.4590 | 2.4789 | + +The complete change takes **13.9% less warm server time**, a **1.16x speedup**. +It is 2.9% faster than this SEAL prototype in this particular run; this small +gap is not a general claim of outperforming SEAL. SEAL uses different modulus +primes, key switching and a TC128 configuration. Our parameters have no +established equivalent security claim. The native timer excludes outer +MessagePack; SEAL includes its envelope and temporary-file binding I/O. Both +include ciphertext decoding, arithmetic, compaction and encoding, and exclude +key import, private encryption/decryption and network time. + +For 32 vectors, warm medians are 0.08718 s (old), 0.08284 s (NTT only), +0.07628 s (both) and 0.07702 s (SEAL): 12.5% less time than the old kernels. +Cold times are 0.40162, 0.39037, 0.38269 and 0.08032 s respectively. Preparing +our public evaluation-key cache still dominates the small cold search. + +All native output ciphertexts match exactly, and every backend returns all +1,024 correct distances. The response is 102,496 bytes; query plus response is +471,250 bytes. This optimization does not change the wire format or noise; +the observed minimum response noise budget is 26 bits, a correctness diagnostic +and not a cryptographic security level. + +| Exclusive profile phase | Previous kernels (s) | Both optimizations (s) | +| --- | ---: | ---: | +| Forward + inverse NTT | 1.6556 | 1.5044 | +| Conversion to residues | 0.1385 | 0.0841 | +| GMP CRT reconstruction | 0.1828 (194 calls) | 0.0005 (2 calls) | +| GMP scale-and-round | 0.1678 (193 calls) | 0.0014 (1 call) | +| Exact RNS scale-and-round | 0 | 0.2152 (192 calls) | + +The 192 tensor reconstructions disappear; only the two final response +components are reconstructed. Transform-table payload falls from 2,752,512 +to 1,835,232 bytes. Key-cache payload stays 42,467,328 bytes, and the RNS scaling +constants add 1,824 bytes to the server plan. These are payload counts, not +peak process memory. NTTs still dominate the remaining cost. + +Raw samples, complete phase profiles, exact parameters, environment and source/ +binary hashes are in `benchmarks/results/native_bfv_fused_1024.json` and +`benchmarks/results/native_bfv_fused_32.json`. Fresh encryption changes key +JSON sizes slightly. Old benchmark artifacts are retained. + +## Validation + +The 118 BFV test cases pass, including exact ciphertext equality across the +three kernel levels for 60-, 180- and 480-bit Q. Standalone C++ oracles compare +the fused spectrum with the old transform through N=32768, and compare exact +scaling with GMP at all eight supported Q widths. Cases cover signed convolution +bounds, zero, maximal coefficients, half-integer rounding, large t and exact +CRT fallback decisions. The C++ suite and 33 native Python-binding tests also +pass AddressSanitizer and UndefinedBehaviorSanitizer. Lint and type checks pass. + +These checks provide arithmetic and memory-safety evidence for the exercised +paths. They do not establish a security level, negligible decryption-failure +probability or production readiness. diff --git a/docs/research/native-bfv-residue.md b/docs/research/native-bfv-residue.md index ac747bf..1fc822a 100644 --- a/docs/research/native-bfv-residue.md +++ b/docs/research/native-bfv-residue.md @@ -1,5 +1,9 @@ # Persistent RNS BFV server +The measurements below describe the initial residue backend. The follow-up +[fused NTT and RNS scaling report](native-bfv-fused-rns.md) documents the current +default kernels and ABI 4; the earlier kernels remain available as controls. + `BFVClient(rns_modulus=True, server_backend="residue")` keeps ciphertexts as arrays of 60-bit residues after the initial BFV tensor scale-and-round. Key switching, rotations, additions, plaintext masks and the result merge tree run diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile index 48d59d5..555413e 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile @@ -15,7 +15,7 @@ LDLIBS ?= -lgmpxx -lgmp .PHONY: all clean all: $(TARGET) -$(TARGET): bindings.cpp rns_ntt.h profile.h bfv_server.h bfv_residue.h packed_wire.h Makefile +$(TARGET): bindings.cpp rns_ntt.h profile.h bfv_server.h bfv_residue.h rns_scale.h packed_wire.h Makefile $(CXX) $(CPPFLAGS) $(CXXFLAGS) -std=c++17 -Wall -Wextra -fPIC -shared \ -I$(PY_INC) bindings.cpp $(LDFLAGS) $(LDLIBS) -o $@ diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi index a5757a0..5c88107 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi @@ -3,7 +3,8 @@ from typing import Any ABI_VERSION: int def create_ring( - n: int, q_hex: str, digit_bits: int, fast: bool = False, residue: bool = False + n: int, q_hex: str, digit_bits: int, fast: bool = False, residue: bool = False, + kernel_level: int = 0, ) -> object: ... def compile_key(ring: object, key: tuple[tuple[bytes, bytes], ...]) -> object: ... def apply_key(key: object, poly: bytes) -> tuple[bytes, bytes]: ... diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h index 61d7ab6..9145082 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h @@ -3,6 +3,7 @@ // unchanged, so the GMP evaluator is a coefficient-for-coefficient oracle. #pragma once #include "bfv_server.h" +#include "rns_scale.h" namespace xtrace_bfv { using ResidueCiphertext = std::array; @@ -186,6 +187,7 @@ class ResidueArithmetic { class ResidueServer : public HammingServer { ResidueArithmetic arithmetic_; Residues full_mask_ntt_; + std::unique_ptr scale_; ResidueCiphertext tile(const Ciphertext& query, const Ciphertext& indexed, const Residues& selected) const { Ciphertext difference = query; @@ -195,12 +197,15 @@ class ResidueServer : public HammingServer { difference[k][i] -= indexed[k][i]; if (difference[k][i] < 0) difference[k][i] += ring->q; } } - // One exact tensor and scale/round per tile still uses the original - // auxiliary base and GMP. After this boundary, ciphertexts remain RNS - // through relinearization, all rotations, masks and the merge tree. - auto product = ring->multiply(difference, difference, t, true); - auto distance = arithmetic_.apply(arithmetic_.split(product[2]), *relin_); - arithmetic_.add(distance, {arithmetic_.split(product[0]), arithmetic_.split(product[1])}); + std::array product; + if (scale_) product = scale_->multiply(difference, difference, true); + else { + // Retain the exact GMP route for differential tests and benchmarks. + auto canonical = ring->multiply(difference, difference, t, true); + for (std::size_t k = 0; k < 3; ++k) product[k] = arithmetic_.split(canonical[k]); + } + auto distance = arithmetic_.apply(product[2], *relin_); + arithmetic_.add(distance, {std::move(product[0]), std::move(product[1])}); for (auto g : left_) { auto rotated = arithmetic_.rotate(distance, g, key(g)); arithmetic_.add(distance, rotated); @@ -213,7 +218,9 @@ class ResidueServer : public HammingServer { ResidueServer(KeyHandle relin, std::map keys, std::size_t padded, Word t, const mpz_class& target) : HammingServer(std::move(relin), std::move(keys), padded, t, target, false), - arithmetic_(*ring), full_mask_ntt_(ring->encode(mask(capacity), arithmetic_.count)) {} + arithmetic_(*ring), full_mask_ntt_(ring->encode(mask(capacity), arithmetic_.count)) { + if (ring->kernel_level >= 2) scale_ = std::make_unique(*ring, t); + } void search(const Ciphertext& query, std::size_t count, ReadTile read_tile, Emit emit, bool compact_result) const override { const auto tiles = (count + capacity - 1) / capacity; @@ -233,6 +240,8 @@ class ResidueServer : public HammingServer { }); } - std::size_t bytes() const override { return HammingServer::bytes() + arithmetic_.count * ring->n * sizeof(Word); } + std::size_t bytes() const override { + return HammingServer::bytes() + arithmetic_.count * ring->n * sizeof(Word) + (scale_ ? scale_->bytes() : 0); + } }; } // namespace xtrace_bfv diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp index 6e09bae..6abdb59 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp @@ -106,17 +106,18 @@ template PyObject* checked(Function&& function) { PyObject* create_ring(PyObject*, PyObject* args) { return checked([&]() -> PyObject* { - PyObject *n_object, *bits_object; + PyObject *n_object, *bits_object, *level_object = nullptr; const char* q_hex; int fast = 0, residue = 0; - if (!PyArg_ParseTuple(args, "OsO|pp", &n_object, &q_hex, &bits_object, &fast, &residue)) return nullptr; + if (!PyArg_ParseTuple(args, "OsO|ppO", &n_object, &q_hex, &bits_object, &fast, &residue, &level_object)) return nullptr; auto n = PyLong_AsUnsignedLongLong(n_object), bits = PyLong_AsUnsignedLongLong(bits_object); + auto level = level_object ? PyLong_AsUnsignedLongLong(level_object) : 0; if (PyErr_Occurred()) return nullptr; - if (n > 32768 || bits > 512) throw std::invalid_argument("Invalid native BFV ring parameters"); + if (n > 32768 || bits > 512 || level > 2) throw std::invalid_argument("Invalid native BFV ring parameters or kernel level"); mpz_class q; if (q.set_str(q_hex, 16) != 0) throw std::invalid_argument("Invalid ciphertext modulus"); RingPtr ring; - { WithoutGIL release; ring = std::make_shared(n, q, bits, fast, residue); } + { WithoutGIL release; ring = std::make_shared(n, q, bits, fast, residue, level); } return capsule(std::move(ring), ring_name); }); } @@ -247,11 +248,11 @@ PyObject* ring_info(PyObject*, PyObject* argument) { if (!value) { Py_DECREF(primes); return nullptr; } PyTuple_SET_ITEM(primes, i, value); } - return Py_BuildValue("{s:N,s:n,s:n,s:s,s:s,s:O,s:n}", "primes", primes, + return Py_BuildValue("{s:N,s:n,s:n,s:s,s:s,s:O,s:n,s:I}", "primes", primes, "switch_prime_count", ring->switch_prime_count, "transform_table_bytes", ring->bytes(), "gmp", gmp_version, "compiler", __VERSION__, "fast_arithmetic", ring->fast ? Py_True : Py_False, - "residue_prime_count", ring->residue_prime_count); + "residue_prime_count", ring->residue_prime_count, "kernel_level", ring->kernel_level); }); } @@ -410,7 +411,7 @@ PyModuleDef module = {PyModuleDef_HEAD_INIT, "_bfv_rns", "Native BFV RNS/NTT CPU PyMODINIT_FUNC PyInit__bfv_rns() { PyObject* result = PyModule_Create(&module); - if (result && PyModule_AddIntConstant(result, "ABI_VERSION", 3) < 0) { + if (result && PyModule_AddIntConstant(result, "ABI_VERSION", 4) < 0) { Py_DECREF(result); return nullptr; } return result; diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h index 6ddd7ce..60140bb 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h @@ -8,11 +8,11 @@ namespace xtrace_bfv { enum class Phase : std::size_t { forward_ntt, inverse_ntt, to_residues, crt_reconstruct, crt_exact_fallback, gadget_decompose, - pointwise, scale_round, mod_q, buffers, automorphism, add_sub, wire_import, wire_export, rns_compose, other, count + pointwise, scale_round, mod_q, buffers, automorphism, add_sub, wire_import, wire_export, rns_compose, rns_scale, other, count }; inline constexpr const char* phase_names[] = { "forward_ntt", "inverse_ntt", "to_residues", "crt_reconstruct", "crt_exact_fallback", "gadget_decompose", - "pointwise", "scale_round", "mod_q", "buffers", "automorphism", "add_sub", "wire_import", "wire_export", "rns_compose", "other" + "pointwise", "scale_round", "mod_q", "buffers", "automorphism", "add_sub", "wire_import", "wire_export", "rns_compose", "rns_scale", "other" }; struct NativeProfile { using Clock = std::chrono::steady_clock; diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h index 3347fb9..e0ef1cf 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h @@ -52,12 +52,74 @@ struct Multiplier { class PrimeNTT { std::size_t n_; bool lazy_; + bool fused_; Word reciprocal_low_, reciprocal_high_; std::vector roots_, inverse_roots_, twists_, inverse_scales_; + std::vector stage_roots_, stage_inverse_; + std::unique_ptr inverse_n_, last_inverse_; + + void forward_fused(std::vector& values) const { + const Word twice = 2 * modulus; + // Evaluate at odd powers of psi directly. One root per contiguous + // butterfly group incorporates the negacyclic twist into the transform. + // Stage inputs/outputs are in [0,4p); the fixed multiplier sees <4p. + for (std::size_t groups = 1, gap = n_ / 2; groups < n_; groups *= 2, gap /= 2) { + for (std::size_t group = 0; group < groups; ++group) { + const auto root = stage_roots_[groups + group]; + Word* a = values.data() + 2 * group * gap; + Word* b = a + gap; + auto butterfly = [&](std::size_t j) { + Word u = a[j]; + if (u >= twice) u -= twice; + Word v = root.multiply_lazy(b[j], modulus); + a[j] = u + v; + b[j] = u + twice - v; + }; + std::size_t j = 0; + for (; j + 4 <= gap; j += 4) { + butterfly(j); butterfly(j + 1); butterfly(j + 2); butterfly(j + 3); + } + for (; j < gap; ++j) butterfly(j); + } + } + for (auto& value : values) { + if (value >= twice) value -= twice; + if (value >= modulus) value -= modulus; + } + } + + void inverse_fused(std::vector& values) const { + const Word twice = 2 * modulus; + for (std::size_t groups = n_ / 2, gap = 1; groups > 1; groups /= 2, gap *= 2) { + for (std::size_t group = 0; group < groups; ++group) { + const auto root = stage_inverse_[groups + group]; + Word* a = values.data() + 2 * group * gap; + Word* b = a + gap; + auto butterfly = [&](std::size_t j) { + Word u = a[j], v = b[j], sum = u + v; + a[j] = sum >= twice ? sum - twice : sum; + b[j] = root.multiply_lazy(u + twice - v, modulus); + }; + std::size_t j = 0; + for (; j + 4 <= gap; j += 4) { + butterfly(j); butterfly(j + 1); butterfly(j + 2); butterfly(j + 3); + } + for (; j < gap; ++j) butterfly(j); + } + } + // Inverse stages keep [0,2p). Fuse 1/N into the final butterfly, saving + // N/2 multiplications and the separate inverse-twist/scaling pass. + for (std::size_t j = 0; j < n_ / 2; ++j) { + Word u = values[j], v = values[j + n_ / 2]; + values[j] = inverse_n_->multiply(u + v, modulus); + values[j + n_ / 2] = last_inverse_->multiply(u + twice - v, modulus); + } + } public: const Word modulus; - PrimeNTT(std::size_t n, Word prime, bool lazy = false) : n_(n), lazy_(lazy), modulus(prime) { + PrimeNTT(std::size_t n, Word prime, bool lazy = false, bool fused = false) + : n_(n), lazy_(lazy), fused_(fused), modulus(prime) { // For an odd prime, floor((2^128-1)/p) == floor(2^128/p). Wide reciprocal = ~Wide(0) / prime; reciprocal_low_ = Word(reciprocal); @@ -70,6 +132,26 @@ class PrimeNTT { Word omega = Wide(psi) * psi % prime; Word inverse_omega = power_mod(omega, prime - 2, prime); Word inverse_psi = power_mod(psi, prime - 2, prime); + if (fused_) { + unsigned log_n = 0; + for (auto v = n; v > 1; v /= 2) ++log_n; + std::vector powers(n, 1), inverse_powers(n, 1); + for (std::size_t i = 1; i < n; ++i) { + powers[i] = Wide(powers[i - 1]) * psi % prime; + inverse_powers[i] = Wide(inverse_powers[i - 1]) * inverse_psi % prime; + } + stage_roots_.reserve(n); stage_inverse_.reserve(n); + for (std::size_t i = 0; i < n; ++i) { + Word reversed = 0, index = i; + for (unsigned bit = 0; bit < log_n; ++bit) { reversed = 2 * reversed + (index & 1); index >>= 1; } + stage_roots_.emplace_back(powers[reversed], prime); + stage_inverse_.emplace_back(inverse_powers[reversed], prime); + } + Word inverse_n = power_mod(n, prime - 2, prime); + inverse_n_ = std::make_unique(inverse_n, prime); + last_inverse_ = std::make_unique(Wide(stage_inverse_[1].value) * inverse_n % prime, prime); + return; + } Word root = 1, inverse_root = 1, twist = 1; Word scale = power_mod(n, prime - 2, prime); roots_.reserve(n / 2); @@ -95,6 +177,7 @@ class PrimeNTT { // bit-reversal permutation is needed between forward and inverse NTTs. void forward(std::vector& values) const { ProfileScope timing(Phase::forward_ntt); + if (fused_) { forward_fused(values); return; } if (lazy_) { // Internal values stay below 2p. One correction per butterfly; // canonicalize only at the boundary so gadget accumulation keeps @@ -134,6 +217,7 @@ class PrimeNTT { // negacyclic twist in the final coefficient pass. void inverse(std::vector& values) const { ProfileScope timing(Phase::inverse_ntt); + if (fused_) { inverse_fused(values); return; } if (lazy_) { // Inputs to each butterfly are below 4p. Reduce a into [0,2p), // obtain b*w in [0,2p), then both outputs fit [0,4p). With p<2^60 @@ -170,7 +254,7 @@ class PrimeNTT { values[i] = inverse_scales_[i].multiply(values[i], modulus); } - std::size_t bytes() const { return 3 * n_ * sizeof(Multiplier); } + std::size_t bytes() const { return (fused_ ? 2 * n_ + 2 : 3 * n_) * sizeof(Multiplier); } // Barrett reduction for any unsigned 128-bit input. Only the low word of // floor(x*floor(2^128/p)/2^128) is needed: subtraction wraps mod 2^64, @@ -185,6 +269,19 @@ class PrimeNTT { return remainder >= modulus ? remainder - modulus : remainder; } + // Quotient and remainder when floor(value/p) fits one word. Tensor scaling + // uses value=t*y with t,y<2^60, so the quotient is below 2^61. + std::pair divide(Wide value) const { + Word low = Word(value), high = Word(value >> 64); + Wide p00 = Wide(low) * reciprocal_low_; + Wide p01 = Wide(low) * reciprocal_high_, p10 = Wide(high) * reciprocal_low_; + Wide middle = (p00 >> 64) + Word(p01) + Word(p10); + Word quotient = high * reciprocal_high_ + Word(p01 >> 64) + Word(p10 >> 64) + Word(middle >> 64); + Word remainder = low - quotient * modulus; + if (remainder >= modulus) { remainder -= modulus; ++quotient; } + return {quotient, remainder}; + } + Word remainder(Wide value) const { return lazy_ ? reduce(value) : value % modulus; } // Exact floor(y*2^64/p) for y transforms; std::size_t switch_prime_count; std::size_t residue_prime_count = 0; - Ring(std::size_t n, const mpz_class& q, unsigned bits, bool fast = false, bool residue = false) + Ring(std::size_t n, const mpz_class& q, unsigned bits, bool fast = false, bool residue = false, + unsigned kernel_level = 0) : n(n), coefficient_bytes((mpz_sizeinbase(q.get_mpz_t(), 2) + 7) / 8), digits(bits ? (mpz_sizeinbase(q.get_mpz_t(), 2) + bits - 1) / bits : 0), - digit_bits(bits), q(q), fast(fast) { + digit_bits(bits), q(q), fast(fast), kernel_level(kernel_level) { if (n < 8 || n > 32768 || (n & (n - 1)) || q < 3 || mpz_sizeinbase(q.get_mpz_t(), 2) > 512 || bits < 1 || bits > mpz_sizeinbase(q.get_mpz_t(), 2)) throw std::invalid_argument("Invalid native BFV ring parameters"); + if (kernel_level > 2 || (kernel_level && !residue)) + throw std::invalid_argument("Kernel level must be 0..2, with nonzero levels requiring the residue backend"); // A coefficient of a gadget dot product has |c| <= N*L*(2^b-1)*(q-1). // A BFV cross component is a sum of two products, bounded by 2N(q-1)^2. // Auxiliary modulus M must exceed twice the bound for exact signed CRT. @@ -238,7 +339,7 @@ class Ring { prime -= 2 * n; candidate = prime; } - transforms.emplace_back(n, prime, fast); + transforms.emplace_back(n, prime, fast, kernel_level >= 1); product *= prime; prime -= 2 * n; } @@ -367,9 +468,9 @@ class Ring { return result; } - std::array multiply( + std::array tensor( const std::array& lhs, const std::array& rhs, - Word t, bool square) const { + bool square) const { std::size_t count = prime_count(2 * n * (q - 1) * (q - 1)); auto a0 = encode(lhs[0], count), a1 = encode(lhs[1], count); Residues b0, b1; @@ -388,6 +489,13 @@ class Ring { } } } + return output; + } + + std::array multiply( + const std::array& lhs, const std::array& rhs, + Word t, bool square) const { + auto output = tensor(lhs, rhs, square); std::array result; mpz_class denominator = 2 * q; for (std::size_t k = 0; k < 3; ++k) { diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h new file mode 100644 index 0000000..b4b11dd --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h @@ -0,0 +1,214 @@ +// Exact BFV tensor scale-and-round in RNS. Public plan construction uses GMP; +// evaluation uses small residues and fixed-word corrections at CRT boundaries. +#pragma once +#include "rns_ntt.h" + +namespace xtrace_bfv { +// Even the complete auxiliary product is <= 17*60 bits for 480-bit Q. These +// extra limbs also cover the largest scaled CRT sum used by a correction. +using CRTWords = std::array; +using CRTTerms = std::array; + +inline CRTWords crt_words(const mpz_class& value) { + CRTWords result{}; + if (value < 0 || mpz_sizeinbase(value.get_mpz_t(), 2) > 64 * result.size()) + throw std::invalid_argument("CRT constant exceeds the fixed-word bound"); + mpz_export(result.data(), nullptr, -1, sizeof(Word), 0, 0, value.get_mpz_t()); + return result; +} + +inline void words_addmul(CRTWords& value, const CRTWords& source, Word multiplier) { + Word carry = 0; + for (std::size_t i = 0; i < value.size(); ++i) { + Wide product = Wide(source[i]) * multiplier + value[i] + carry; + value[i] = Word(product); carry = Word(product >> 64); + } + if (carry) throw std::overflow_error("Fixed-word CRT correction overflow"); +} + +inline CRTWords words_mul(const CRTWords& value, Word multiplier) { + CRTWords result{}; + words_addmul(result, value, multiplier); + return result; +} + +inline bool words_ge(const CRTWords& lhs, const CRTWords& rhs) { + for (std::size_t i = lhs.size(); i-- > 0;) + if (lhs[i] != rhs[i]) return lhs[i] > rhs[i]; + return true; +} + +class ExactRNSBase { + const Ring& ring_; + std::vector inverses_, product_mod_target_; + std::vector partial_words_; + std::vector> partial_mod_target_; + CRTWords product_words_; + + CRTWords sum_words(const CRTTerms& terms) const { + CRTWords result{}; + for (std::size_t j = 0; j < count; ++j) words_addmul(result, partial_words_[j], terms[j]); + return result; + } + +public: + const std::size_t start, count, target_start, target_count; + mpz_class product = 1; + struct Normalized { CRTTerms terms{}; Word alpha = 0; }; + + ExactRNSBase(const Ring& ring, std::size_t start, std::size_t count, + std::size_t target_start, std::size_t target_count) + : ring_(ring), start(start), count(count), target_start(target_start), target_count(target_count) { + if (!count || count > 17 || !target_count || target_count > 17 || + start > ring.transforms.size() || count > ring.transforms.size() - start || + target_start > ring.transforms.size() || target_count > ring.transforms.size() - target_start) + throw std::invalid_argument("Invalid exact RNS conversion bases"); + for (std::size_t j = 0; j < count; ++j) product *= ring.transforms[start + j].modulus; + product_words_ = crt_words(product); + partial_mod_target_.assign(target_count, std::vector(count)); + for (std::size_t j = 0; j < count; ++j) { + Word p = ring.transforms[start + j].modulus; + mpz_class partial = product / p; + partial_words_.push_back(crt_words(partial)); + inverses_.emplace_back(power_mod(mpz_fdiv_ui(partial.get_mpz_t(), p), p - 2, p), p); + for (std::size_t k = 0; k < target_count; ++k) + partial_mod_target_[k][j] = mpz_fdiv_ui(partial.get_mpz_t(), ring.transforms[target_start + k].modulus); + } + for (std::size_t k = 0; k < target_count; ++k) { + Word p = ring.transforms[target_start + k].modulus; + product_mod_target_.emplace_back(mpz_fdiv_ui(product.get_mpz_t(), p), p); + } + } + + // A=sum(u_j*M_j), theta=A/M. floor(theta) selects [0,M), while + // round(theta) selects (-M/2,M/2). Fixed-point bounds certify the quotient; + // an ambiguous interval is resolved by an exact fixed-word comparison. + Normalized normalize(const CRTTerms& residues, bool centered) const { + Normalized result; + Wide fraction = 0; + for (std::size_t j = 0; j < count; ++j) { + const auto& transform = ring_.transforms[start + j]; + result.terms[j] = inverses_[j].multiply(residues[j], transform.modulus); + fraction += transform.fraction(result.terms[j]); + } + Wide bias = centered ? Wide(1) << 63 : 0; + result.alpha = Word((fraction + bias) >> 64); + if (result.alpha != ((fraction + count + bias) >> 64)) { + ProfileScope fallback(Phase::crt_exact_fallback); + auto sum = sum_words(result.terms); + if (centered) { + if (words_ge(words_mul(sum, 2), words_mul(product_words_, 2 * result.alpha + 1))) ++result.alpha; + } else if (words_ge(sum, words_mul(product_words_, result.alpha + 1))) ++result.alpha; + } + return result; + } + + CRTTerms convert(const Normalized& value) const { + CRTTerms output{}; + for (std::size_t k = 0; k < target_count; ++k) { + const auto& transform = ring_.transforms[target_start + k]; + Wide sum = 0; + // At most 17 products below 2^120 fit in the 128-bit accumulator. + for (std::size_t j = 0; j < count; ++j) sum += Wide(value.terms[j]) * partial_mod_target_[k][j]; + Word residue = transform.reduce(sum); + Word correction = product_mod_target_[k].multiply(value.alpha, transform.modulus); + Word result = residue + transform.modulus - correction; + output[k] = result >= transform.modulus ? result - transform.modulus : result; + } + return output; + } + + // For the canonical lift r=A-alpha*M, round(t*r/M) is + // round(t*theta)-t*alpha. Split each t*u_j/p_j into its integer part and a + // rigorously bounded 64-bit fraction. No precision is lost as t increases. + Word round_fraction(const Normalized& value, Word t) const { + Word whole = 0; + Wide fraction = 0; + for (std::size_t j = 0; j < count; ++j) { + const auto& transform = ring_.transforms[start + j]; + auto divided = transform.divide(Wide(t) * value.terms[j]); + whole += divided.first; + fraction += transform.fraction(divided.second); + } + const Wide half = Wide(1) << 63; + Word rounded = whole + Word((fraction + half) >> 64); + if (((fraction + half) >> 64) != ((fraction + count + half) >> 64)) { + ProfileScope fallback(Phase::crt_exact_fallback); + if (words_ge(words_mul(sum_words(value.terms), 2 * t), words_mul(product_words_, 2 * rounded + 1))) ++rounded; + } + return rounded - t * value.alpha; + } + + std::size_t bytes() const { + return (count + target_count) * sizeof(Multiplier) + (count + 1) * sizeof(CRTWords) + + target_count * count * sizeof(Word); + } +}; + +class RNSScale { + const Ring& ring_; + const Word t_; + const std::size_t tensor_count_; + ExactRNSBase q_to_b_, b_to_q_; + std::vector inverse_q_, plaintext_; + +public: + RNSScale(const Ring& ring, Word t) + : ring_(ring), t_(t), tensor_count_(ring.prime_count(2 * ring.n * (ring.q - 1) * (ring.q - 1))), + q_to_b_(ring, 0, ring.residue_prime_count, ring.residue_prime_count, tensor_count_ - ring.residue_prime_count), + b_to_q_(ring, ring.residue_prime_count, tensor_count_ - ring.residue_prime_count, 0, ring.residue_prime_count) { + if (t < 2 || t >= (Word(1) << 60) || t >= ring.q || q_to_b_.product != ring.q || q_to_b_.count > 8) + throw std::invalid_argument("Invalid RNS scale parameters"); + mpz_class quotient_bound = 2 * ring.n * (ring.q - 1) * (ring.q - 1) / ring.q + 1; + if (b_to_q_.product <= 2 * quotient_bound) + throw std::invalid_argument("RNS quotient basis is too small for exact signed scaling"); + for (std::size_t j = 0; j < b_to_q_.count; ++j) { + Word p = ring.transforms[b_to_q_.start + j].modulus; + inverse_q_.emplace_back(power_mod(mpz_fdiv_ui(ring.q.get_mpz_t(), p), p - 2, p), p); + } + for (std::size_t j = 0; j < q_to_b_.count; ++j) { + Word p = ring.transforms[j].modulus; + plaintext_.emplace_back(t % p, p); + } + } + + Residues scale(Residues tensor) const { + ProfileScope timing(Phase::rns_scale); + if (tensor.size() != tensor_count_ || + std::any_of(tensor.begin(), tensor.end(), [&](const auto& p) { return p.size() != ring_.n; })) + throw std::invalid_argument("Incorrect RNS tensor dimensions"); + for (std::size_t j = 0; j < tensor_count_; ++j) ring_.transforms[j].inverse(tensor[j]); + Residues result(q_to_b_.count, std::vector(ring_.n)); + for (std::size_t i = 0; i < ring_.n; ++i) { + CRTTerms remainder{}, quotient{}; + for (std::size_t j = 0; j < q_to_b_.count; ++j) remainder[j] = tensor[j][i]; + auto normalized = q_to_b_.normalize(remainder, false); + auto remainder_b = q_to_b_.convert(normalized); + Word rounded_remainder = q_to_b_.round_fraction(normalized, t_); + for (std::size_t j = 0; j < b_to_q_.count; ++j) { + Word p = ring_.transforms[b_to_q_.start + j].modulus; + Word residue = tensor[b_to_q_.start + j][i] + p - remainder_b[j]; + quotient[j] = inverse_q_[j].multiply(residue, p); + } + // z=Q*k+r with 0<=r= p ? rounded_remainder - p : rounded_remainder; + Word c = plaintext_[j].multiply(quotient_q[j], p) + r; + result[j][i] = c >= p ? c - p : c; + } + } + return result; + } + + std::array multiply(const Ciphertext& lhs, const Ciphertext& rhs, bool square) const { + auto output = ring_.tensor(lhs, rhs, square); + for (auto& component : output) component = scale(std::move(component)); + return output; + } + + std::size_t bytes() const { return q_to_b_.bytes() + b_to_q_.bytes() + (inverse_q_.size() + plaintext_.size()) * sizeof(Multiplier); } +}; +} // namespace xtrace_bfv diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py index 4214857..2c866de 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py @@ -90,18 +90,29 @@ class BFVEvaluator: number of evaluation keys in this snapshot and never stores query results. """ - def __init__(self, pk: BFVPublicKey, backend: BFVServerBackend = "optimized") -> None: + def __init__( + self, + pk: BFVPublicKey, + backend: BFVServerBackend = "optimized", + *, + kernel_level: int | None = None, + ) -> None: if backend not in ("optimized", "reference", "rns", "native", "residue"): raise ValueError( "server_backend must be 'optimized', 'reference', 'rns', 'native', or 'residue'" ) + if kernel_level is not None and backend != "residue": + raise ValueError("kernel_level is a residue backend experiment control") self.backend = backend self._pk: BFVPublicKey = {**pk, "galois_keys": dict(pk["galois_keys"])} # Exponent zero identifies relinearization; Galois exponents are odd. self._switch_keys: dict[int, _PackedSwitchKey] = {} self._rns = ( BFVRNSArithmetic( - self._pk, fast=backend in ("native", "residue"), residue=backend == "residue" + self._pk, + fast=backend in ("native", "residue"), + residue=backend == "residue", + kernel_level=kernel_level, ) if backend in ("rns", "native", "residue") else None diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py index 2940637..0016907 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py @@ -16,7 +16,14 @@ class BFVRNSArithmetic: """Bind optional C++ kernels to one validated BFV public-key context.""" - def __init__(self, pk: BFVPublicKey, *, fast: bool = False, residue: bool = False) -> None: + def __init__( + self, + pk: BFVPublicKey, + *, + fast: bool = False, + residue: bool = False, + kernel_level: int | None = None, + ) -> None: try: from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns except ImportError as error: @@ -25,7 +32,7 @@ def __init__(self, pk: BFVPublicKey, *, fast: bool = False, residue: bool = Fals "make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON=/path/to/python; " "or select server_backend='optimized' for the Python/GMP evaluator." ) from error - if _bfv_rns.ABI_VERSION != 3: + if _bfv_rns.ABI_VERSION != 4: raise ImportError("BFV RNS CPU extension has an incompatible ABI; rebuild it") self._native = _bfv_rns self._pk = pk @@ -36,8 +43,16 @@ def __init__(self, pk: BFVPublicKey, *, fast: bool = False, residue: bool = Fals ) self.n = params.poly_modulus_degree self.width = (pk["q"].bit_length() + 7) // 8 + if kernel_level is None: + kernel_level = 2 if residue else 0 + if ( + type(kernel_level) is not int + or kernel_level not in (0, 1, 2) + or (kernel_level and not residue) + ): + raise ValueError("kernel_level must be 0..2; nonzero levels require residue arithmetic") self._ring = _bfv_rns.create_ring( - self.n, format(pk["q"], "x"), params.decomposition_bits, fast, residue + self.n, format(pk["q"], "x"), params.decomposition_bits, fast, residue, kernel_level ) self._keys: dict[int, object] = {} diff --git a/tests/x_vec/native/test_bfv_rns.cpp b/tests/x_vec/native/test_bfv_rns.cpp index fddf3b8..a73b1f8 100644 --- a/tests/x_vec/native/test_bfv_rns.cpp +++ b/tests/x_vec/native/test_bfv_rns.cpp @@ -41,6 +41,9 @@ int main() { assert(transform.reduce(wide) == wide % p); Word y = i ? random() % p : p - 1; assert(transform.fraction(y) == (Wide(y) << 64) / p); + Wide numerator = Wide(y) * (i ? random() % p : p - 1); + auto divided = transform.divide(numerator); + assert(divided.first == numerator / p && divided.second == numerator % p); } std::vector values(8); for (auto& value : values) value = random() % p; @@ -179,5 +182,57 @@ int main() { } } } - std::cout << "Native NTT, signed CRT, gadget and persistent RNS oracles passed\n"; + // The fused transform must preserve the existing bit-reversed spectrum, + // including maximal residues, not just invert its own forward operation. + for (std::size_t n : {8, 16, 64, 8192, 32768}) { + Ring ring(n, (mpz_class(1) << 180) - 1, 30, true); + for (const auto& original : ring.transforms) { + PrimeNTT fused(n, original.modulus, true, true); + for (int sample = 0; sample < 3; ++sample) { + std::vector input(n); + for (auto& c : input) c = sample == 0 ? 0 : sample == 1 ? original.modulus - 1 : random() % original.modulus; + auto old = input, next = input; + original.forward(old); fused.forward(next); + assert(old == next); + fused.inverse(next); + assert(next == input); + } + } + } + // Independent GMP oracle for exact signed tensor scaling at every Q width. + // The +/- convolution bounds and half-integer rounding cases exercise the + // certified fixed-point intervals and their fixed-word exact corrections. + q = 1; + for (std::size_t count = 1; count <= 8; ++count) { + q *= primes.transforms[count - 1].modulus; + Ring ring(8, q, 30, true, true, 2); + ResidueArithmetic arithmetic(ring); + mpz_class bound = 2 * ring.n * (q - 1) * (q - 1); + auto tensor_count = ring.prime_count(bound); + for (Word t : {Word(2), Word(97), Word(65537), (Word(1) << 59) - 1}) { + RNSScale scaler(ring, t); + NativeProfile profile; + for (int sample = 0; sample < 100; ++sample) { + Polynomial z(8); + for (auto& c : z) { c = big_random.get_z_range(2 * bound + 1); c -= bound; } + if (sample == 0) z = {0, 1, -1, mpz_class(q / 2), mpz_class(q / 2 + 1), mpz_class(-q / 2), mpz_class(-q / 2 - 1), bound}; + if (sample == 1) z = {-bound, q, -q, mpz_class(2 * q), mpz_class(-2 * q), mpz_class(q - 1), mpz_class(1 - q), mpz_class(bound - 1)}; + Residues scaled; + { ProfileSession session(profile); scaled = scaler.scale(ring.encode(z, tensor_count)); } + auto result = arithmetic.compose(scaled); + for (std::size_t i = 0; i < z.size(); ++i) { + mpz_class expected = 2 * t * z[i] + q, denominator = 2 * q; + mpz_fdiv_q(expected.get_mpz_t(), expected.get_mpz_t(), denominator.get_mpz_t()); + mpz_mod(expected.get_mpz_t(), expected.get_mpz_t(), q.get_mpz_t()); + assert(result[i] == expected); + } + } + assert(profile.calls[std::size_t(Phase::crt_reconstruct)] == 0); + if (count > 1) assert(profile.calls[std::size_t(Phase::crt_exact_fallback)] > 0); + bool rejected = false; + try { scaler.scale(Residues{}); } catch (const std::invalid_argument&) { rejected = true; } + assert(rejected); + } + } + std::cout << "Native NTT, signed CRT, gadget, persistent RNS and exact scaling oracles passed\n"; } diff --git a/tests/x_vec/test_bfv_residue.py b/tests/x_vec/test_bfv_residue.py index 63fe2a2..54f3a5f 100644 --- a/tests/x_vec/test_bfv_residue.py +++ b/tests/x_vec/test_bfv_residue.py @@ -9,6 +9,7 @@ from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _rns_coefficient_primes +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVEvaluator from xtrace_sdk.x_vec.utils.xtrace_types import BFVParameters @@ -107,3 +108,36 @@ def test_residue_rejects_existing_prime_modulus_keys() -> None: client.encode_hamming_server(query, query) with pytest.raises(ValueError, match="product"): native.create_ring(16, format(client._pk()["q"], "x"), 15, True, True) + + +@pytest.mark.parametrize("bits", [60, 180, 480]) +def test_residue_kernel_levels_return_identical_ciphertexts(bits: int) -> None: + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + client = BFVClient(3, 16, 97, bits, 15, response_modulus_bits=40, rns_modulus=True) + vectors = [[0, 0, 0], [1, 1, 1], [0, 1, 0]] * 13 + query = client.encrypt_vec_one([0, 1, 0]) + index = client.encrypt_vec_packed(vectors) + responses = [] + for level in range(3): + server = BFVClient(skip_key_gen=True, server_backend="residue") + server.load_config(json.loads(client.stringify_config())) + server.load_stringified_keys(client.stringify_pk()) + server._server_evaluator = BFVEvaluator(server._pk(), "residue", kernel_level=level) + server._evaluator_public_key = server._pk() + assert server._evaluator().cache_info()["kernel_level"] == level + response = server.encode_hamming_server_packed(query, index, len(vectors)) + assert client.decode_hamming_client_packed(response, len(vectors)) == [1, 2, 0] * 13 + responses.append(response) + assert responses[0] == responses[1] == responses[2] + + +@pytest.mark.parametrize("level", [-1, 3, True, "2", 1 << 64]) +def test_residue_kernel_level_validation(level) -> None: + native = pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + keys = BFV.key_gen(16, 97, 120, 15, rns_modulus=True) + with pytest.raises(ValueError): + BFVEvaluator(keys["pk"], "residue", kernel_level=level) + # bool is accepted by the low-level PyLong ABI, but rejected by the wrapper. + if level is not True: + with pytest.raises((TypeError, ValueError, OverflowError)): + native.create_ring(16, format(keys["pk"]["q"], "x"), 15, True, True, level) From b7841e45333772d1e470939a1c2351fea17a639c Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Thu, 10 Sep 2026 23:23:35 -0400 Subject: [PATCH 08/12] Harden BFV input parsing and document production security blockers --- .github/workflows/ci.yml | 2 +- .../results/native_bfv_security_1024.json | 432 ++++++++++++++++++ docs/index.rst | 1 + docs/research/native-bfv-fused-rns.md | 8 + docs/research/native-bfv-residue.md | 5 +- docs/research/native-bfv-security.md | 347 ++++++++++++++ docs/research/native-bfv.md | 6 +- src/xtrace_sdk/x_vec/crypto/bfv_client.py | 31 +- .../x_vec/crypto/bfv_cpu_ext/bindings.cpp | 25 +- src/xtrace_sdk/x_vec/crypto/encryption/bfv.py | 142 ++++-- .../x_vec/crypto/encryption/bfv_evaluator.py | 3 + .../x_vec/crypto/encryption/bfv_native.py | 11 +- tests/x_vec/native/run_binding_sanitizers.py | 9 +- tests/x_vec/native/test_bfv_rns.cpp | 2 +- tests/x_vec/test_bfv_security.py | 316 +++++++++++++ 15 files changed, 1284 insertions(+), 56 deletions(-) create mode 100644 benchmarks/results/native_bfv_security_1024.json create mode 100644 docs/research/native-bfv-security.md create mode 100644 tests/x_vec/test_bfv_security.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f6a1433..8ccbe04 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -68,7 +68,7 @@ jobs: sudo apt-get install -y libgmp-dev make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 4' - uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py + uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py tests/x_vec/test_bfv_security.py - name: BFV native arithmetic sanitizers run: | diff --git a/benchmarks/results/native_bfv_security_1024.json b/benchmarks/results/native_bfv_security_1024.json new file mode 100644 index 0000000..5ceeb01 --- /dev/null +++ b/benchmarks/results/native_bfv_security_1024.json @@ -0,0 +1,432 @@ +{ + "environment": { + "utc": "2026-09-11T03:20:46.201242+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "06d8d172c168f37dd77d1572f68df2088d2d82d7", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_server.py", + "--num-vectors", + "1024", + "--rns-modulus", + "--backends", + "residue", + "--residue-levels", + "0,2", + "--seal", + "--repeats", + "4", + "--native-profile", + "--json-out", + "benchmarks/results/native_bfv_security_1024.json" + ], + "measurement_notes": "One CPU process, identical public keys and encrypted inputs for all native backends. Server times include native ciphertext decoding/encoding, terminal modulus switching and lazy cache preparation; exclude outer MessagePack, arithmetic context creation (reported under setup), key import, encryption and client decryption. First search starts with empty key and mask caches. Warm searches reuse these caches. Order alternates. Profiles, if requested, are additional untimed searches. No network or equal-security comparison to other schemes.", + "backend_variants": { + "residue-0": [ + "residue", + 0 + ], + "residue-2": [ + "residue", + 2 + ] + }, + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "ciphertext_modulus_hex": "ffffffffffbc0030000004bff78002fff1004bffbc001", + "ciphertext_modulus_primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137 + ], + "seed": 1337, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 4.603943218942732, + "index_encrypt_s": 6.865148832090199, + "query_encrypt_s": 0.10382973193190992, + "public_key_export_s": 0.8147107770200819, + "residue-0_public_key_import_s": 1.466961293015629, + "residue-0_arithmetic_context_s": 0.0017840259242802858, + "residue-2_public_key_import_s": 1.4780206978321075, + "residue-2_arithmetic_context_s": 0.0010454009752720594 + }, + "runs": [ + { + "backend": "residue-0", + "repeat": 0, + "server_compute_s": 3.0709580299444497 + }, + { + "backend": "residue-2", + "repeat": 0, + "server_compute_s": 2.7449227031320333 + }, + { + "backend": "seal", + "repeat": 0, + "server_compute_s": 2.484233553055674 + }, + { + "backend": "seal", + "repeat": 1, + "server_compute_s": 2.4849505240563303 + }, + { + "backend": "residue-2", + "repeat": 1, + "server_compute_s": 2.4235800479073077 + }, + { + "backend": "residue-0", + "repeat": 1, + "server_compute_s": 2.7881676598917693 + }, + { + "backend": "residue-0", + "repeat": 2, + "server_compute_s": 2.787437440128997 + }, + { + "backend": "residue-2", + "repeat": 2, + "server_compute_s": 2.426682349992916 + }, + { + "backend": "seal", + "repeat": 2, + "server_compute_s": 2.4957599870394915 + }, + { + "backend": "seal", + "repeat": 3, + "server_compute_s": 2.4768296871334314 + }, + { + "backend": "residue-2", + "repeat": 3, + "server_compute_s": 2.420860161073506 + }, + { + "backend": "residue-0", + "repeat": 3, + "server_compute_s": 2.7841470839921385 + } + ], + "summary": { + "residue-0": { + "first_search_s": 3.0709580299444497, + "warm_searches_s": [ + 2.7881676598917693, + 2.787437440128997, + 2.7841470839921385 + ], + "warm_median_s": 2.787437440128997 + }, + "residue-2": { + "first_search_s": 2.7449227031320333, + "warm_searches_s": [ + 2.4235800479073077, + 2.426682349992916, + 2.420860161073506 + ], + "warm_median_s": 2.4235800479073077 + }, + "seal": { + "first_search_s": 2.484233553055674, + "warm_searches_s": [ + 2.4849505240563303, + 2.4957599870394915, + 2.4768296871334314 + ], + "warm_median_s": 2.4849505240563303 + } + }, + "first_search_speedup": null, + "warm_median_speedup": null, + "speedups_relative_to_first_native_backend": { + "residue-0": { + "first": 1.0, + "warm_median": 1.0 + }, + "residue-2": { + "first": 1.1187775985241408, + "warm_median": 1.150132194946839 + }, + "seal": { + "first": 1.236179273952358, + "warm_median": 1.1217275407072893 + } + }, + "seal_comparison": { + "tenseal": "0.3.16", + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "active_coeff_modulus_bits": [ + 43, + 43, + 44, + 44 + ], + "key_coeff_modulus_bits": [ + 43, + 43, + 44, + 44, + 44 + ], + "security_setting": "SEAL TC128; no equivalent-security claim for native parameters", + "index_bytes": 27680519, + "query_bytes": 432613, + "public_key_bytes": 39585853, + "timing_notes": "SEAL search includes its serialization, MessagePack and binding temporary-file I/O. Native times exclude outer MessagePack; both include ciphertext decoding, arithmetic, compaction and ciphertext encoding. These are practical server calls with different parameters/formats, not identical arithmetic kernels.", + "response_bytes": 105854, + "all_distances_correct": true, + "minimum_response_noise_budget_bits": 19 + }, + "native_phases": { + "residue-0": { + "forward_ntt": { + "seconds": 1.1965777880000001, + "calls": 13934 + }, + "inverse_ntt": { + "seconds": 0.457320508, + "calls": 5946 + }, + "to_residues": { + "seconds": 0.13917984900000002, + "calls": 320 + }, + "crt_reconstruct": { + "seconds": 0.18413337100000002, + "calls": 194 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.079418079, + "calls": 13357 + }, + "pointwise": { + "seconds": 0.29637719700000004, + "calls": 1151 + }, + "scale_round": { + "seconds": 0.16709886100000002, + "calls": 193 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.019559113, + "calls": 895 + }, + "automorphism": { + "seconds": 0.033134543, + "calls": 639 + }, + "add_sub": { + "seconds": 0.058943964, + "calls": 1406 + }, + "wire_import": { + "seconds": 0.029082671, + "calls": 130 + }, + "wire_export": { + "seconds": 0.000167453, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.061222676000000004, + "calls": 705 + }, + "rns_scale": { + "seconds": 0.0, + "calls": 0 + }, + "other": { + "seconds": 0.07879342, + "calls": 1 + } + }, + "residue-2": { + "forward_ntt": { + "seconds": 1.1098617400000002, + "calls": 13934 + }, + "inverse_ntt": { + "seconds": 0.39522169, + "calls": 5946 + }, + "to_residues": { + "seconds": 0.08329718, + "calls": 128 + }, + "crt_reconstruct": { + "seconds": 0.00039382800000000004, + "calls": 2 + }, + "crt_exact_fallback": { + "seconds": 0.0, + "calls": 0 + }, + "gadget_decompose": { + "seconds": 0.080379433, + "calls": 13357 + }, + "pointwise": { + "seconds": 0.29532529300000004, + "calls": 1151 + }, + "scale_round": { + "seconds": 0.001434909, + "calls": 1 + }, + "mod_q": { + "seconds": 0.0, + "calls": 0 + }, + "buffers": { + "seconds": 0.018826629, + "calls": 895 + }, + "automorphism": { + "seconds": 0.039910895, + "calls": 639 + }, + "add_sub": { + "seconds": 0.058128008, + "calls": 1406 + }, + "wire_import": { + "seconds": 0.029310127000000002, + "calls": 130 + }, + "wire_export": { + "seconds": 0.00017654400000000002, + "calls": 2 + }, + "rns_compose": { + "seconds": 0.060768088000000005, + "calls": 705 + }, + "rns_scale": { + "seconds": 0.21904145000000003, + "calls": 192 + }, + "other": { + "seconds": 0.056853829, + "calls": 1 + } + } + }, + "native_profile_note": "Additional warm searches, excluded from performance samples. Exclusive wall times and call counts; other includes Python, binding overhead and unscoped native work. buffers covers explicit buffer setup; GMP allocations remain charged to their arithmetic phases. Profiles are per-thread and restored after exceptions.", + "sizes": { + "encrypted_index_bytes": 23598496, + "public_keys_bytes": 85604170, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "caches": { + "residue-0": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 2752512, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "kernel_level": 0, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 589824 + }, + "residue-2": { + "primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137, + 1152921504606601217, + 1152921504606584833, + 1152921504606109697, + 1152921504605962241 + ], + "switch_prime_count": 3, + "transform_table_bytes": 1835232, + "gmp": "6.3.0", + "compiler": "13.3.0", + "fast_arithmetic": true, + "residue_prime_count": 3, + "kernel_level": 2, + "switch_keys": 18, + "key_payload_bytes": 42467328, + "server_plan_bytes": 591648 + } + }, + "cache_size_note": "Additional to the existing public key: GMP object payloads including folding constants, or native NTT coefficient arrays and transform tables. Excludes containers, CRT constants and transient allocations; not peak RSS.", + "minimum_response_noise_budget_bits": 26, + "identical_ciphertexts": true, + "all_distances_correct": true, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 1023, + "distance": 0 + }, + { + "index": 81, + "distance": 220 + } + ], + "source_sha256": { + "benchmarks/bfv_server.py": "ad70ee74af27d055ac9653a58ed46b3c58381aef2de875568cdd15eeb5a9fac6", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "05e30fc3b8191e2fd0f0620ac7dbbc35ccea0a43913fc62f66575b0d44716aeb", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "experiments/bfv/packed_hamming.py": "5c940efe69617a822d0e6fce8afff92d14804cd49431cc87705553b71a709e01" + } +} diff --git a/docs/index.rst b/docs/index.rst index 5986238..0aeabe7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -86,6 +86,7 @@ Choose your path: research/native-bfv-server research/native-bfv-residue research/native-bfv-fused-rns + research/native-bfv-security research/bfv-packed-hamming .. toctree:: diff --git a/docs/research/native-bfv-fused-rns.md b/docs/research/native-bfv-fused-rns.md index 1cb87d4..b6da2d7 100644 --- a/docs/research/native-bfv-fused-rns.md +++ b/docs/research/native-bfv-fused-rns.md @@ -109,6 +109,14 @@ binary hashes are in `benchmarks/results/native_bfv_fused_1024.json` and `benchmarks/results/native_bfv_fused_32.json`. Fresh encryption changes key JSON sizes slightly. Old benchmark artifacts are retained. +After the [security hardening](native-bfv-security.md), a fresh interleaved +1,024-vector run measured 2.7874 s (previous kernels), 2.4236 s (both +optimizations) and 2.4850 s (SEAL) as warm medians. The improvement remains +**13.1%**, with identical native ciphertexts and all distances correct. +`benchmarks/results/native_bfv_security_1024.json` records that final source +state and all samples. Public-key import, measured separately, took about +1.48 s with the stricter parser and the default ~86 MB bundle. + ## Validation The 118 BFV test cases pass, including exact ciphertext equality across the diff --git a/docs/research/native-bfv-residue.md b/docs/research/native-bfv-residue.md index 1fc822a..81d904b 100644 --- a/docs/research/native-bfv-residue.md +++ b/docs/research/native-bfv-residue.md @@ -33,8 +33,9 @@ error sampling, plaintext modulus, batching, encryption and exact rounding algorithms are unchanged. An equal-bit response-compaction request retains Q instead of increasing it to the same-bit prime. -This remains experimental leveled BFV with no security audit or security-level -claim. It uses no SEAL code and performs no bootstrapping. The performance +This remains experimental leveled BFV with no independent audit or security-level +claim; see the [internal security review](native-bfv-security.md). It uses no SEAL +code and performs no bootstrapping. The performance comparison below does not establish equivalent security to SEAL's parameters. ## Build and use diff --git a/docs/research/native-bfv-security.md b/docs/research/native-bfv-security.md new file mode 100644 index 0000000..aee54ea --- /dev/null +++ b/docs/research/native-bfv-security.md @@ -0,0 +1,347 @@ +# Internal BFV security review — 2026-09-10 + +**Decision: keep this implementation experimental. It is not ready for a +production security claim.** The arithmetic and input handling have stronger +test coverage, but protocol integrity, parameter assurance and private-key +implementation assurance remain open. This is an internal, AI-assisted source +review and regression exercise, not an independent cryptographic audit, +penetration test of a deployed service, or proof of security. + +The optimization baseline reviewed is commit `06d8d17`, following +`e5c0751`. This report and the accompanying hardening are on +`research/bfv-packed-hamming`. No production endpoint or `main` change is part +of this work. The Paillier implementations and SEAL experiment are preserved. + +## Scope and trust model + +The review covers `crypto/encryption/bfv.py`, `bfv_evaluator.py`, `bfv_rns.py`, +`bfv_native.py`, `crypto/bfv_client.py`, the BFV types in +`x_vec/utils/xtrace_types.py`, and every source header plus `bindings.cpp` in +`crypto/bfv_cpu_ext/`. It follows key generation, encryption, public evaluation, +packed serialization, response compaction, decryption, input checks and native +ownership/concurrency. Existing BFV tests, build rules and the benchmark's +secret/public separation were also inspected. Paillier, SEAL internals, +production HTTP services, dependency implementations and deployment +infrastructure are outside this review's audit scope. + +The assumed data owner controls a trusted private client and its secret key. +The evaluator receives public encryption/evaluation keys, encrypted index and +encrypted queries. An adversary may replace, replay, reorder or corrupt the +server's output or network messages. A separate honest-but-curious model assumes +the server follows the specified circuit and only observes its inputs and +execution. Local side-channel attackers are considered a deployment concern, +but no timing, cache, power or fault-injection exploit was attempted. + +The local BFV API supplies no authenticated network session, tenant policy, +query identifier, index-version binding or verified computation. The wire's +key ID is public. Decrypted distances and top-k selection are private-client +operations. A later document fetch, telemetry event, error response or retry +can cross that privacy boundary even if this arithmetic module never makes a +network call. + +## Findings + +Severity below describes impact **if the stated integration exposes the +relevant surface**. These are not claims that a deployed XTrace endpoint is +currently exploitable. + +| ID | Severity / status | Finding and precondition | +| --- | --- | --- | +| BFV-01 | High; open deployment blocker | An attacker able to observe chosen-ciphertext decryptions can recover the secret. Error/noise feedback must also be treated as secret-dependent. | +| BFV-02 | High; open for an untrusted evaluator | Ciphertext validation and distance range checks do not establish correct computation or prevent replay. | +| BFV-03 | High; open assurance blocker | Algebraically accepted parameters have no established security level or negligible failure-probability guarantee. | +| BFV-04 | High with a local observation surface; open | Private Python/GMP operations are variable time, with no secret-memory erasure guarantee. Secret serialization is plaintext. | +| BFV-05 | Medium; locally hardened | Oversized byte/hex fields were converted before size rejection; key JSON lacked an import-size bound and admitted ambiguous encodings. Service-wide resource limits remain necessary. | +| BFV-06 | High if setup is unauthenticated; documented, open | The key fingerprint does not authenticate the public bundle and does not cover evaluation keys. | +| BFV-07 | Medium; open protocol policy | Distances, query-derived access, counts and stable identifiers have application-level leakage; shared-key query access does not enforce database confidentiality from that query holder. | + +### BFV-01: a decryption oracle can disclose the key + +For our formula `Decrypt(c) = round(t*(c0+c1*s)/Q) mod t`, construct +`c0=0` and the constant polynomial `c1=floor(Q/t)`. This requires only public +parameters and the public key ID. For a ternary secret and the test parameters, +decrypting yields each secret coefficient modulo t; mapping `t-1` to `-1` +recovers the entire secret. The regression test demonstrates this with a fresh +local key and no attack on an external system. The small ring makes the test +fast; the construction is algebraic, not a brute-force attack on its dimension. + +`test_decryption_oracle_would_reveal_secret_key` intentionally **passes when +this disclosure is possible**. It records a forbidden integration pattern; +it is not a mitigation test. The demonstration uses the raw polynomial +decryption API, not a claim that one call to the range-checked Hamming decoder +returns the full key. Rejecting some decoded values can itself create a +secret-dependent success/failure oracle if that outcome is observable. + +Do not expose `BFV.decrypt`, decoded distances, `noise_budget`, per-response +errors or decryption-dependent telemetry to the untrusted evaluator. A +production protocol must analyze any result-dependent follow-up, including +which document IDs are fetched. SEAL's own guidance likewise treats decryptor +outputs and noise diagnostics as private, and Apple's BFV guidance explicitly +warns about decryption-oracle key recovery. +[SEAL security guidance](https://github.com/microsoft/SEAL/blob/main/SECURITY.md), +[Apple BFV guidance](https://github.com/apple/swift-homomorphic-encryption/blob/main/README.md#homomorphic-encryption-he). + +Client response parsing now limits accepted Hamming responses to two +components at the configured terminal modulus or original Q, before private +decryption. This rejects unexpected protocol shapes. **It does not provide +chosen-ciphertext security.** Blanket rejection of all-zero ciphertexts would +also reject legitimate homomorphic results and would not prevent an attacker +from constructing other chosen ciphertexts. + +### BFV-02: plausible results and replays remain unauthenticated + +Two tests show that the client accepts both a transparent zero ciphertext and +a fresh public-key encryption of the plausible distance 1. Neither proves +that the encrypted query was evaluated against the intended index. A third +test replays the response to one query while the correct answer to another +query is different. Key identity, shape and `0 <= distance <= embed_len` +checks all still pass. There is no query or index epoch in the response frame. + +Authenticated transport and authenticated setup are necessary to exclude an +outside message replacement attack. Binding messages to a tenant, key, query +nonce, index version, ordered IDs, layout and parameters can exclude accidental +mixups and outsider replay when those bindings are authenticated. **A signature +from the evaluator alone does not prove it computed the right circuit.** An +actively malicious evaluator requires a reviewed verifiable-computation or +other protocol mechanism, or an explicit weaker trust assumption. No bespoke +authentication scheme was added around homomorphic ciphertexts in this change. +SEAL also distinguishes loading/decrypting a ciphertext from authenticating it. +[SEAL ciphertext authenticity guidance](https://github.com/microsoft/SEAL/blob/main/SECURITY.md#ciphertext-authenticity). + +### BFV-03: parameter and failure assurance is incomplete + +`BFV.validate_parameters` checks algebra, supported bounds and types. It +deliberately accepts insecure tiny rings for tests, and also accepts large +Q/small-N combinations that must not be selected for production merely because +the method returns successfully. The API does not advertise a 128-bit security +setting, enforce a deployment allowlist or estimate lattice attacks. + +The optimized benchmark configuration is: + +| Item | Actual construction | +| --- | --- | +| Ring | `Z_Q[X]/(X^8192+1)` | +| Q | Product of three 60-bit NTT primes; 180 total bits | +| Plaintext | t=65537, two-row batching | +| Secret and encryption ephemeral u | Independent uniform coefficients in `{-1,0,1}` | +| Error | Centered binomial, eta=21; variance 10.5 and standard deviation about 3.240 | +| Evaluation keys | Six 30-bit gadget digits for s² and required automorphic secrets, under the same secret | +| Terminal response | Exact componentwise rounding to a 50-bit prime | + +Q's three primes are 1152921504606830593, 1152921504606748673 and +1152921504606683137. They are the actual encryption modulus; the larger tensor +NTT basis is only an arithmetic workspace. Counting its product as the RLWE +encryption modulus would be incorrect. + +The older HE standard's classical ternary-secret table lists log2(Q)=218 at +N=8192 for its 128-bit target and specified error assumptions. Our Q=180 is +below that reference bound, but our binomial sampler and secret-dependent +evaluation keys still require a justified analysis. This table comparison +**does not establish security for our implementation**. +[HE Standard v1.1, Table 1, ternary section on PDF page 27](https://homomorphicencryption.org/wp-content/uploads/2018/11/HomomorphicEncryptionStandardv1.1.pdf). +The consortium points to newer parameter guidance; parameter selection must +cover both attacks and functional correctness. +[Current guidance index](https://homomorphicencryption.org/security-guidelines/), +[Security Guidelines for Implementing Homomorphic Encryption](https://cic.iacr.org/p/1/4/26). + +Before making a security-level claim, record an expert-reviewed configuration +with concrete classical/quantum attack estimates, estimator version and cost +models, the exact secret/error distributions, sample exposure and the +assumptions needed by the evaluation keys. A modern estimator was not run in +this review. Its rough and full estimators can use different cost models; +one convenient output number is insufficient evidence. +[Lattice estimator documentation](https://github.com/malb/lattice-estimator). + +Correctness also needs a circuit-specific bound on decryption failure, +including compaction, maximum dimensions/merge depth, message patterns, +key distributions and repeated queries. The tests cover fresh randomness, +extreme patterns and exact arithmetic boundaries. They cannot establish a +negligible failure probability. The benchmark's 26-bit noise budget measures +remaining noise margin on those results; it is neither 26-bit nor 128-bit +cryptographic security, and positive noise diagnostics cannot rule out an +earlier wraparound. + +### BFV-04: private operations and key storage + +Key generation, ephemeral/error sampling, encryption and decryption use +`secrets` and Python/GMP. The review found no benchmark PRNG feeding production +encryption: public uniform coefficients use `secrets.randbelow(Q)`, ternary +coefficients use `randbelow(3)`, and each binomial sample subtracts the popcounts +of two independent `randbits(eta)` results. Tests check mapping and bounds and +verify that an entropy-source exception aborts key generation/encryption. +They do not constitute an entropy audit or statistical security proof. + +Python objects, GMP multiplication/packing, comparisons, allocation and error +paths have no constant-time contract. We did not measure an exploitable timing +channel, but cannot claim its absence. Moving **public server** evaluation to +constant-width RNS does not harden the private key operations. The server +kernels receive no secret key; tests enforce that public-only evaluation never +calls private phase computation. + +`stringify_sk` is an explicit plaintext export. Secret tuples, GMP limbs and +JSON copies have no guaranteed erasure, locked-memory protection or encrypted +storage. Key custody, crash dumps, logs, process isolation and backup handling +need a deployment design. Do not claim that a Python zeroization helper would +erase every copy. Private arithmetic requires independent implementation and +side-channel review appropriate to the intended platform. + +### BFV-05: input handling fixes and remaining resource policy + +The review found that the original wire readers called `int.from_bytes` on +every field before checking size. Arbitrarily long high-order zero padding +could therefore consume conversion memory while representing a small valid +number. Public-key coefficients and native modulus text were similarly passed +to GMP before their bit limits were established. Key JSON had no pre-parse +size cap, silently accepted duplicate object fields, and accepted boolean +version values and multiple textual encodings of some integers. + +The hardening adds: + +- A shared Python wire reader with per-field byte and integer limits checked + before big-integer conversion/unpacking. Negative values, bools and floats + are rejected. C++ still checks every imported coefficient is strictly below + Q before any arithmetic relies on the convolution bound. +- Public-key and secret-key JSON limits of 128 Mi and 1 Mi **characters**, + respectively, checked before `json.loads`. These are local resource policy, + not security parameters. `BFV.deserialize_*` accepts an explicit `max_chars`; + `BFVClient.load_stringified_keys` exposes `max_public_key_chars` for larger + trusted experiments. Existing default public bundles (~86 MB ASCII) fit. +- Duplicate-field rejection, bounded canonical lowercase coefficient hex, + canonical decimal Galois exponents, strict key fields/version types, and + normalization of excessive JSON nesting into `ValueError`. +- Native modulus-text bounds before GMP parsing and a modulus-type check on + direct Python ciphertext objects. Failed key imports leave the client and + existing evaluator context unchanged. + +Previously emitted v1 keys and minimally encoded ciphertexts remain readable; +the serialization version and cryptographic values are unchanged. Noncanonical +hand-written JSON, excessive zero padding and over-limit bundles are now +rejected intentionally. JSON whitespace within the overall cap remains valid. + +These caps do not bound a service's total memory or CPU use. JSON parsing still +allocates the entire permitted object tree. A character cap is not a byte cap +for Unicode; a service must restrict transport bytes before buffering or +decoding, authenticate and allowlist setup parameters, limit index/request +sizes and concurrent evaluations, and impose process/resource quotas. Direct +Python object APIs and private C++ capsules assume trusted application code; +they are not sandbox boundaries. GMP out-of-memory behavior was not fault +injected, and the native extension's exception handling is not an OOM survival +guarantee. + +### BFV-06: authenticate the whole setup bundle + +`_key_id` hashes serialized parameters plus encryption polynomials `(b,a)`. +It is useful for key mixup detection. It is unkeyed, and omits relinearization +and Galois keys. A test changes a relinearization coefficient, successfully +loads the bundle with the same key ID, and confirms the whole-bundle digest +changed. An attacker able to replace all of `(b,a)` can also recompute a new +fingerprint; neither equality check establishes ownership. + +Public import cannot verify the secret-dependent algebraic relationship of +evaluation keys from their shape alone. Secret import checks ternary +coefficients and consistency with the public RLWE sample, but does not certify +key generation or authenticate the public setup. Provision and authenticate +the complete bundle, configuration and intended owner from a trusted source. +An unkeyed digest is meaningful only when its expected value is authenticated +out of band. No incompatible fingerprint change was made in this review. + +### BFV-07: application leakage and authorization + +This protocol returns all candidate distances in packed ciphertexts and lets +the client compute top-k. It does not reveal only three distances. A party with +chosen-query access to all Hamming distances can recover a binary database +vector v: querying the zero vector gives its weight w; querying unit vector +e_i gives `w+1-2*v_i`. These D+1 queries determine every bit. For a data owner +searching their own data this is expected; it is not an access-control scheme +for mutually distrustful users sharing the key or query capability. + +The server also observes vector count, parameters, encrypted-index identity, +message sizes and query timing. Subsequent top-k document fetches may reveal +access patterns. Ciphertext randomization does not hide these metadata. No +circuit-privacy, multi-tenant isolation, forward secrecy or private retrieval +guarantee is established here. Define which leakage is acceptable and enforce +owner/key authorization in the application. + +## Evidence and limits of testing + +The new `tests/x_vec/test_bfv_security.py` contains **63 cases**. Coverage +includes the four explicit protocol-limit demonstrations, randomness handling, +public-only operation, parser limits/ambiguity, response shape before private +decryption, atomic key-load failure, 450 deterministic wire mutations across +kernel levels, and 100 native coefficient-buffer cases. Mutation tests compare +accepted attacker-modified ciphertexts with the GMP reference and reject +malformed framing; they do not assume ciphertext tampering must fail. + +A default-size stress test uses two fresh N=8192, Q=180 RNS key sets, six query +patterns (zeros, ones, alternating bits, first/last one-hot and random bits), +and a 33-vector index with a partial tile and merge. Every distance is checked +against plaintext Hamming distance. Fixed PRNGs generate only test vectors and +mutation positions, never cryptographic randomness. + +The standalone C++ suite checks old/fused NTT spectra through N=32768, +schoolbook products, certified CRT boundaries, gadget widths through the +supported maximum, and exact signed RNS scaling at all eight Q widths. The +review extended the scaling oracle to plaintext multipliers immediately below +2^60 where Q permits. Fixed-word corrections are exercised at ambiguous +rounding boundaries, not merely random points. + +The source review checked the 60-bit-prime, 64/128-bit accumulator and +eight-/18-word storage bounds, the signed tensor quotient bound, immutable +server/key ownership while the GIL is released, byte-buffer lifetime, canonical +imports and guarded exports. No memory corruption was found in the exercised +paths. This is not formal verification, an exhaustive fuzzer campaign or an +independent proof of the RNS construction. + +Validation on this branch: + +- **263 offline tests pass**, including **181 BFV tests** and the existing + Paillier/SEAL coverage. HTTP integration and unavailable GPU tests were excluded. +- The C++ arithmetic suite passes AddressSanitizer and UndefinedBehaviorSanitizer. +- **96 Python-binding/security tests pass** against an ASan/UBSan extension. + This includes malformed inputs and the real-default stress case. +- Ruff and mypy pass for the SDK and BFV benchmark code. CI runs the new + security tests and includes them in the sanitized binding run. + +Reproduce the principal security checks: + +```bash +make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" +.venv/bin/python -m pytest tests/x_vec/test_bfv_security.py -q +``` + +The exact standalone and binding sanitizer commands are in +`.github/workflows/ci.yml`. Sanitizer success covers the executed cases on this +compiler/platform; it does not establish constant-time behavior or memory +erasure. Test output contains no generated secret key material. + +The final performance check after hardening used the same-key interleaved +1,024-vector × 512-dimension workload as the optimization report. Warm medians +were 2.7874 s for the previous residue kernels, 2.4236 s for the new kernels +and 2.4850 s for the preserved SEAL prototype. Native ciphertexts were identical +and all distances were correct; response size stayed 102,496 bytes. This +retains a 13.1% reduction in server time. Raw samples and final source/binary +hashes are in `benchmarks/results/native_bfv_security_1024.json`. The +[performance report](native-bfv-fused-rns.md) explains the different SEAL +parameters, timing boundary and absence of an equal-security comparison. + +## Production gates + +1. Have an independent HE cryptographer review the construction, exact RNS + scaling proof, distributions, parameter estimates, evaluation-key assumptions + and a complete circuit-specific decryption-failure analysis. Freeze and + enforce a reviewed configuration separately from algebra-test parameters. +2. Specify the end-to-end threat model and every output channel. For an active + server, provide a reviewed integrity/verifiability design and demonstrate + that response handling, errors, retries and document retrieval do not expose + a harmful decryption oracle. Authenticate setup and bind sessions, queries, + index versions, IDs and layouts. +3. Review private arithmetic and key custody for the deployment platform. + Establish appropriate side-channel controls, logging/storage policy, + isolation, rotation and incident handling. +4. Apply service-level resource limits and authorization. Run sustained + adversarial fuzzing, platform/compiler coverage, dependency review and load + tests on the actual endpoint before rollout. + +Passing the current tests closes specific regression risks. It does not close +these gates or make the homemade BFV interchangeable with an independently +reviewed production cryptosystem. diff --git a/docs/research/native-bfv.md b/docs/research/native-bfv.md index 8ce2361..c70bef5 100644 --- a/docs/research/native-bfv.md +++ b/docs/research/native-bfv.md @@ -6,8 +6,10 @@ existing `gmpy2` dependency; it does not import SEAL or TenSEAL. The earlier This is a **leveled BFV research implementation** for understanding the scheme and developing faster kernels. It has no bootstrapping, is not constant time, and has -not undergone a security audit. The parameter settings below are arithmetic -defaults, not a claim of a particular security level. +not undergone an independent cryptographic audit. The +[internal security review](native-bfv-security.md) records hardening, tests and +open production blockers. The parameter settings below are arithmetic defaults, +not a claim of a particular security level. ## Code map diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_client.py index 3a247d7..1d1bd6a 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_client.py +++ b/src/xtrace_sdk/x_vec/crypto/bfv_client.py @@ -11,7 +11,12 @@ from typing import Any from xtrace_sdk.x_vec.crypto.device import DeviceMode -from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _parameter_dict +from xtrace_sdk.x_vec.crypto.encryption.bfv import ( + BFV, + MAX_PUBLIC_KEY_CHARS, + _coefficient_modulus, + _parameter_dict, +) from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVEvaluator, BFVServerBackend from xtrace_sdk.x_vec.crypto.encryption.bfv_native import BFVNativeServer from xtrace_sdk.x_vec.crypto.hamming_client_base import HammingClientBase @@ -32,6 +37,9 @@ class BFVClient(HammingClientBase): The explicit ``*_packed`` methods amortize each ciphertext over several database vectors and pack up to N distances into each response ciphertext. All homomorphic evaluation uses public keys only; top-k stays on the client. + This experimental client has no result-authenticity or security-level claim. + Keep decrypted results, errors and diagnostics private; see + docs/research/native-bfv-security.md before designing a production protocol. :param embed_len: Binary embedding dimension, at most N/2 and less than t. :param poly_modulus_degree: Ring degree N, a power of two. @@ -199,13 +207,21 @@ def load_config(self, config: dict[str, Any]) -> None: self.response_modulus_bits = candidate.response_modulus_bits self._configure_layout() - def load_stringified_keys(self, pk: str, sk: str | None = None) -> None: + def load_stringified_keys( + self, + pk: str, + sk: str | None = None, + *, + max_public_key_chars: int = MAX_PUBLIC_KEY_CHARS, + ) -> None: """Load private client keys, or just public keys for server-side evaluation. :param pk: ``stringify_pk()`` JSON, including the evaluation keys. :param sk: ``stringify_sk()`` JSON. Omit on the server. + :param max_public_key_chars: Local JSON import bound; raise explicitly + for larger trusted experiments. Authenticate the entire bundle first. """ - public_key = BFV.deserialize_public_key(pk) + public_key = BFV.deserialize_public_key(pk, max_chars=max_public_key_chars) if public_key["params"] != self.params: raise ValueError("Loaded BFV key parameters do not match the client configuration") n = self.params.poly_modulus_degree @@ -362,9 +378,12 @@ def _validate_count(count: int, ciphertexts: int, capacity: int) -> None: raise ValueError("vector_count does not match the packed ciphertext count") def _decode_slots(self, cipher: Sequence[int | bytes]) -> list[int]: - return BFV.batch_decode( - BFV.decrypt(BFV.ciphertext_from_ints(cipher, self._pk()), self._keys()), self.params - ) + pk = self._pk() + ciphertext = BFV.ciphertext_from_ints(cipher, pk) + target = min(_coefficient_modulus(self.response_modulus_bits), pk["q"]) + if len(ciphertext.components) != 2 or ciphertext.modulus not in (pk["q"], target): + raise ValueError("Unexpected BFV response modulus or component count") + return BFV.batch_decode(BFV.decrypt(ciphertext, self._keys()), self.params) def _check_distances(self, distances: list[int]) -> list[int]: if any(not 0 <= value <= self.embed_len for value in distances): diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp index 6abdb59..f822e99 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp @@ -104,18 +104,31 @@ template PyObject* checked(Function&& function) { return nullptr; } +mpz_class read_modulus(const char* text, Py_ssize_t length, std::size_t max_digits = 128) { + // Bound text before GMP parses it. Leading zero padding, signs, whitespace + // and oversized moduli must not turn a tiny context into a large allocation. + if (length < 1 || static_cast(length) > max_digits || + (length > 1 && text[0] == '0') || + !std::all_of(text, text + length, [](char c) { + return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F'); + })) throw std::invalid_argument("Invalid or oversized native modulus encoding"); + mpz_class value; + if (value.set_str(text, 16) != 0) throw std::invalid_argument("Invalid native modulus"); + return value; +} + PyObject* create_ring(PyObject*, PyObject* args) { return checked([&]() -> PyObject* { PyObject *n_object, *bits_object, *level_object = nullptr; const char* q_hex; + Py_ssize_t q_length; int fast = 0, residue = 0; - if (!PyArg_ParseTuple(args, "OsO|ppO", &n_object, &q_hex, &bits_object, &fast, &residue, &level_object)) return nullptr; + if (!PyArg_ParseTuple(args, "Os#O|ppO", &n_object, &q_hex, &q_length, &bits_object, &fast, &residue, &level_object)) return nullptr; auto n = PyLong_AsUnsignedLongLong(n_object), bits = PyLong_AsUnsignedLongLong(bits_object); auto level = level_object ? PyLong_AsUnsignedLongLong(level_object) : 0; if (PyErr_Occurred()) return nullptr; if (n > 32768 || bits > 512 || level > 2) throw std::invalid_argument("Invalid native BFV ring parameters or kernel level"); - mpz_class q; - if (q.set_str(q_hex, 16) != 0) throw std::invalid_argument("Invalid ciphertext modulus"); + auto q = read_modulus(q_hex, q_length); RingPtr ring; { WithoutGIL release; ring = std::make_shared(n, q, bits, fast, residue, level); } return capsule(std::move(ring), ring_name); @@ -266,7 +279,8 @@ PyObject* create_server(PyObject*, PyObject* args) { return checked([&]() -> PyObject* { PyObject *relin_object, *keys_object, *padded_object, *t_object; const char* target_hex; - if (!PyArg_ParseTuple(args, "OOOOs", &relin_object, &keys_object, &padded_object, &t_object, &target_hex)) return nullptr; + Py_ssize_t target_length; + if (!PyArg_ParseTuple(args, "OOOOs#", &relin_object, &keys_object, &padded_object, &t_object, &target_hex, &target_length)) return nullptr; auto relin = get_capsule(relin_object, key_name); const auto& ring = *relin->ring; auto padded = PyLong_AsUnsignedLongLong(padded_object); @@ -276,7 +290,8 @@ PyObject* create_server(PyObject*, PyObject* args) { if (!padded || padded > ring.n / 2 || (padded & (padded - 1)) || t % (2 * ring.n) != 1 || !mpz_probab_prime_p(plain.get_mpz_t(), 32)) throw std::invalid_argument("Invalid native server layout or plaintext modulus"); - if (target.set_str(target_hex, 16) != 0 || target <= plain || target > ring.q) + target = read_modulus(target_hex, target_length, (mpz_sizeinbase(ring.q.get_mpz_t(), 2) + 3) / 4); + if (target <= plain || target > ring.q) throw std::invalid_argument("Invalid native response modulus"); if (!PyTuple_Check(keys_object)) throw std::invalid_argument("Rotation keys must be a tuple"); std::map keys; diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py index a55f730..a89b2af 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv.py @@ -8,6 +8,7 @@ import hashlib import json +import re import secrets from collections.abc import Sequence from dataclasses import asdict @@ -30,6 +31,71 @@ EncryptedVector, ) +# Local import limits, not cryptographic security parameters. The default +# public Hamming bundle is about 86 MB; larger trusted experiments may override +# this explicitly. A transport must bound bytes before buffering/decoding JSON. +MAX_PUBLIC_KEY_CHARS = 128 * 1024 * 1024 +MAX_SECRET_KEY_CHARS = 1024 * 1024 +_HEX_COEFFICIENT = re.compile(r"(?:0|[1-9a-f][0-9a-f]*)\Z") + + +def _load_key_json(value: str, max_chars: int) -> dict[str, Any]: + if type(max_chars) is not int or max_chars < 1: + raise ValueError("max_chars must be a positive integer") + if not isinstance(value, str) or len(value) > max_chars: + raise ValueError("BFV key JSON exceeds the character limit or is not a string") + + def unique_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for name, item in pairs: + if name in result: + raise ValueError("Duplicate BFV key JSON field") + result[name] = item + return result + + try: + data = json.loads(value, object_pairs_hook=unique_object) + except RecursionError as exc: + raise ValueError("BFV key JSON is too deeply nested") from exc + if not isinstance(data, dict): + raise ValueError("BFV key JSON must be an object") + return data + + +def _wire_integer(value: int | bytes, max_bits: int) -> int: + # Check byte lengths BEFORE allocating a large Python integer, including + # numerically small values hidden behind arbitrarily long zero padding. + if isinstance(value, bytes): + if len(value) > (max_bits + 7) // 8: + raise ValueError("BFV ciphertext entry exceeds its byte limit") + result = int.from_bytes(value, "little") + elif isinstance(value, Integral) and not isinstance(value, bool): + result = int(value) + else: + raise ValueError("Ciphertext entries must be nonnegative integers or bytes") + if result < 0 or result.bit_length() > max_bits: + raise ValueError("BFV ciphertext entry exceeds its integer bounds") + return result + + +def _read_ciphertext_wire(values: Sequence[int | bytes], pk: BFVPublicKey) -> list[int]: + """Bound and validate framing shared by the Python and complete C++ paths.""" + if len(values) not in (8, 9): + raise ValueError("Invalid BFV ciphertext framing") + n, max_q = pk["params"].poly_modulus_degree, pk["q"] + bounds = (32, 1, n.bit_length(), max_q.bit_length(), 256, 2) + data = [_wire_integer(v, bits) for v, bits in zip(values[:6], bounds, strict=True)] + if ( + data[:3] != [0x58424656, 1, n] + or data[4] != int(pk["key_id"], 16) + or data[5] != len(values) - 6 + ): + raise ValueError("Incompatible BFV ciphertext header or key") + if not pk["params"].plain_modulus < data[3] <= max_q: + raise ValueError("Invalid ciphertext modulus") + data.extend(_wire_integer(v, n * data[3].bit_length()) for v in values[6:]) + return data + def _round_div(value: mpz, divisor: mpz) -> mpz: """Nearest integer, with ties toward +infinity; works for negative inputs.""" @@ -343,6 +409,8 @@ def batch_decode(plaintext: BFVPolynomial, params: BFVParameters) -> list[int]: def validate_ciphertext(ciphertext: BFVCiphertext, pk: BFVPublicKey) -> None: """Reject mixed keys, incompatible shapes, and noncanonical coefficients.""" n, q = pk["params"].poly_modulus_degree, ciphertext.modulus + if not isinstance(ciphertext.modulus, Integral) or isinstance(ciphertext.modulus, bool): + raise ValueError("Ciphertext modulus must be an integer") if ciphertext.key_id != pk["key_id"]: raise ValueError("Ciphertext belongs to a different BFV key") if not pk["params"].plain_modulus < q <= pk["q"]: @@ -395,6 +463,9 @@ def decrypt(ciphertext: BFVCiphertext, keys: BFVKeyPair) -> BFVPolynomial: """Recover round(t * (c0 + c1*s + ...) / q) modulo t. Decryption does not authenticate ciphertexts or detect every noise failure. + Keep plaintexts and success/failure feedback inside the trusted client: + attacker-chosen ciphertexts can expose the secret through a decryption + oracle. See docs/research/native-bfv-security.md before protocol use. """ t = keys["pk"]["params"].plain_modulus return tuple( @@ -619,26 +690,10 @@ def ciphertext_to_ints(ciphertext: BFVCiphertext, pk: BFVPublicKey) -> Encrypted @staticmethod def ciphertext_from_ints(values: Sequence[int | bytes], pk: BFVPublicKey) -> BFVCiphertext: """Load native wire v1; byte entries use the same little endian convention as Paillier.""" - if len(values) not in (8, 9): - raise ValueError("Invalid BFV ciphertext framing") - if any(not isinstance(v, (Integral, bytes)) for v in values): - raise ValueError("Ciphertext entries must be nonnegative integers or bytes") - data = [int.from_bytes(v, "little") if isinstance(v, bytes) else int(v) for v in values] - if any(v < 0 for v in data): - raise ValueError("Ciphertext entries must be nonnegative") + data = _read_ciphertext_wire(values, pk) n, q = pk["params"].poly_modulus_degree, mpz(data[3]) - if ( - data[:3] != [0x58424656, 1, n] - or data[4] != int(pk["key_id"], 16) - or data[5] != len(data) - 6 - ): - raise ValueError("Incompatible BFV ciphertext header or key") - if not pk["params"].plain_modulus < q <= pk["q"]: - raise ValueError("Invalid ciphertext modulus") components = [] for packed in data[6:]: - if packed.bit_length() > n * q.bit_length(): - raise ValueError("Packed polynomial exceeds N coefficients") coefficients = gmpy2.unpack(mpz(packed), q.bit_length()) coefficients.extend([mpz(0)] * (n - len(coefficients))) components.append(tuple(coefficients)) @@ -670,15 +725,21 @@ def switch(values: BFVSwitchKey) -> list[list[list[str]]]: ) @staticmethod - def deserialize_public_key(value: str) -> BFVPublicKey: + def deserialize_public_key( + value: str, *, max_chars: int = MAX_PUBLIC_KEY_CHARS + ) -> BFVPublicKey: """Load public JSON, checking parameters, polynomial shapes and key identity. - Serialization is intended for trusted local storage/experiments. Its - fingerprint detects mixups, not malicious modification or resource abuse. + Authenticate the complete bundle before loading it. The fingerprint + covers parameters and (b,a), not evaluation keys, and is not a MAC. + max_chars bounds JSON text before parsing; callers must separately bound + transport buffers and concurrent work. Raising it is a local policy choice. """ try: - data = json.loads(value) - if data["version"] != 1: + data = _load_key_json(value, max_chars) + if set(data) != {"version", "params", "key_id", "b", "a", "relin_key", "galois_keys"}: + raise ValueError("Invalid public key fields") + if type(data["version"]) is not int or data["version"] != 1: raise ValueError("Unsupported public key version") params = BFVParameters(**data["params"]) BFV.validate_parameters(params) @@ -691,7 +752,12 @@ def poly(values: Any) -> BFVPolynomial: if ( not isinstance(values, list) or len(values) != n - or any(not isinstance(v, str) for v in values) + or any( + not isinstance(v, str) + or len(v) > (params.coeff_modulus_bits + 3) // 4 + or _HEX_COEFFICIENT.fullmatch(v) is None + for v in values + ) ): raise ValueError("Invalid key polynomial") result = tuple(mpz(c, 16) for c in values) @@ -710,9 +776,19 @@ def switch(values: Any) -> BFVSwitchKey: identity = _key_id(params, b, a) if data["key_id"] != identity: raise ValueError("Public key fingerprint does not match") - galois_keys = {int(g): switch(key) for g, key in data["galois_keys"].items()} - if any(not 1 < g < 2 * n or g % 2 == 0 for g in galois_keys): - raise ValueError("Invalid Galois exponent") + encoded_galois = data["galois_keys"] + if not isinstance(encoded_galois, dict) or len(encoded_galois) > n - 1: + raise ValueError("Invalid Galois key collection") + galois_keys = {} + for name, key in encoded_galois.items(): + if len(name) > len(str(2 * n)) or not name.isascii() or not name.isdecimal(): + raise ValueError("Invalid Galois exponent") + g = int(name) + if str(g) != name or not 1 < g < 2 * n or g % 2 == 0: + raise ValueError("Invalid Galois exponent") + galois_keys[g] = switch(key) + if not isinstance(data["relin_key"], list): + raise ValueError("Invalid relinearization key") return { "params": params, "q": q, @@ -734,11 +810,19 @@ def serialize_secret_key(sk: BFVSecretKey) -> str: ) @staticmethod - def deserialize_secret_key(value: str, pk: BFVPublicKey) -> BFVSecretKey: + def deserialize_secret_key( + value: str, pk: BFVPublicKey, *, max_chars: int = MAX_SECRET_KEY_CHARS + ) -> BFVSecretKey: """Load and check a ternary secret against its public RLWE sample.""" try: - data = json.loads(value) - if data["version"] != 1 or data["key_id"] != pk["key_id"]: + data = _load_key_json(value, max_chars) + if set(data) != {"version", "key_id", "s"}: + raise ValueError("Invalid secret key fields") + if ( + type(data["version"]) is not int + or data["version"] != 1 + or data["key_id"] != pk["key_id"] + ): raise ValueError("Secret key version or identity mismatch") values = data["s"] if not isinstance(values, list) or len(values) != pk["params"].poly_modulus_degree: diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py index 2c866de..1388cf6 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py @@ -6,6 +6,7 @@ """ from dataclasses import dataclass +from numbers import Integral import sys from typing import Any, Literal @@ -140,6 +141,8 @@ def cache_info(self) -> dict[str, Any]: def _validate(self, ciphertext: BFVCiphertext) -> None: pk = self._pk n, q = pk["params"].poly_modulus_degree, ciphertext.modulus + if not isinstance(ciphertext.modulus, Integral) or isinstance(ciphertext.modulus, bool): + raise ValueError("Ciphertext modulus must be an integer") if ciphertext.key_id != pk["key_id"]: raise ValueError("Ciphertext belongs to a different BFV key") if not pk["params"].plain_modulus < q <= pk["q"]: diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py index 798b8d7..b1b6910 100644 --- a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py @@ -6,10 +6,9 @@ """ from collections.abc import Sequence -from numbers import Integral from typing import Any -from xtrace_sdk.x_vec.crypto.encryption.bfv import _coefficient_modulus +from xtrace_sdk.x_vec.crypto.encryption.bfv import _coefficient_modulus, _read_ciphertext_wire from xtrace_sdk.x_vec.crypto.encryption.bfv_rns import BFVRNSArithmetic from xtrace_sdk.x_vec.utils.xtrace_types import EncryptedVector @@ -59,11 +58,7 @@ def __init__( def _wire(self, values: Sequence[int | bytes]) -> tuple[bytes, bytes]: if len(values) != 8: raise ValueError("Native server requires two-component BFV ciphertext framing") - if any(not isinstance(v, (Integral, bytes)) for v in values): - raise ValueError("Ciphertext entries must be nonnegative integers or bytes") - data = [int.from_bytes(v, "little") if isinstance(v, bytes) else int(v) for v in values] - if any(v < 0 for v in data): - raise ValueError("Ciphertext entries must be nonnegative") + data = _read_ciphertext_wire(values, self.arithmetic._pk) expected = [ 0x58424656, 1, @@ -74,8 +69,6 @@ def _wire(self, values: Sequence[int | bytes]) -> tuple[bytes, bytes]: ] if data[:6] != expected: raise ValueError("Incompatible BFV ciphertext header, modulus or key") - if any(v.bit_length() > self._packed_bytes * 8 for v in data[6:]): - raise ValueError("Packed polynomial exceeds N coefficients") # Coefficient bounds are checked in C++ while importing the bit-packed # buffers. Do not construct or validate N Python integers per component. return data[6].to_bytes(self._packed_bytes, "little"), data[7].to_bytes( diff --git a/tests/x_vec/native/run_binding_sanitizers.py b/tests/x_vec/native/run_binding_sanitizers.py index bca110d..fdcce1c 100644 --- a/tests/x_vec/native/run_binding_sanitizers.py +++ b/tests/x_vec/native/run_binding_sanitizers.py @@ -23,5 +23,12 @@ import pytest # noqa: E402 raise SystemExit( - pytest.main(["tests/x_vec/test_bfv_native.py", "tests/x_vec/test_bfv_residue.py", "-q"]) + pytest.main( + [ + "tests/x_vec/test_bfv_native.py", + "tests/x_vec/test_bfv_residue.py", + "tests/x_vec/test_bfv_security.py", + "-q", + ] + ) ) diff --git a/tests/x_vec/native/test_bfv_rns.cpp b/tests/x_vec/native/test_bfv_rns.cpp index a73b1f8..501a914 100644 --- a/tests/x_vec/native/test_bfv_rns.cpp +++ b/tests/x_vec/native/test_bfv_rns.cpp @@ -209,7 +209,7 @@ int main() { ResidueArithmetic arithmetic(ring); mpz_class bound = 2 * ring.n * (q - 1) * (q - 1); auto tensor_count = ring.prime_count(bound); - for (Word t : {Word(2), Word(97), Word(65537), (Word(1) << 59) - 1}) { + for (Word t : {Word(2), Word(97), Word(65537), (Word(1) << (count == 1 ? 59 : 60)) - 1}) { RNSScale scaler(ring, t); NativeProfile profile; for (int sample = 0; sample < 100; ++sample) { diff --git a/tests/x_vec/test_bfv_security.py b/tests/x_vec/test_bfv_security.py new file mode 100644 index 0000000..e08ed71 --- /dev/null +++ b/tests/x_vec/test_bfv_security.py @@ -0,0 +1,316 @@ +"""Security boundaries and malformed-input regressions for experimental BFV. + +Some tests deliberately demonstrate attacks against hypothetical integrations. +Their success records a limitation; it does NOT mean the attack was prevented. +All keys/data are generated locally for the tests. No service is contacted. +""" + +import hashlib +import json +import random +from dataclasses import replace + +import gmpy2 +import pytest +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.encryption import bfv +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVEvaluator +from xtrace_sdk.x_vec.utils.xtrace_types import BFVCiphertext + + +@pytest.fixture(scope="module") +def client() -> BFVClient: + # Insecure small ring for fast, reproducible algebra/security-contract tests. + return BFVClient(3, 16, 97, 120, 15, response_modulus_bits=40, rns_modulus=True) + + +def public_server(client: BFVClient, backend="residue", level=None) -> BFVClient: + server = BFVClient(skip_key_gen=True, server_backend=backend) + server.load_config(json.loads(client.stringify_config())) + server.load_stringified_keys(client.stringify_pk()) + if level is not None: + server._server_evaluator = BFVEvaluator(server._pk(), backend, kernel_level=level) + server._evaluator_public_key = server._pk() + assert server.keys is None + return server + + +def test_decryption_oracle_would_reveal_secret_key(client: BFVClient) -> None: + """Releasing one raw chosen decryption can disclose every ternary coefficient.""" + pk, keys = client._pk(), client._keys() + n, t = client.params.poly_modulus_degree, client.params.plain_modulus + delta = pk["q"] // t + forged = BFVCiphertext(((mpz(0),) * n, (delta,) + (mpz(0),) * (n - 1)), pk["q"], pk["key_id"]) + wire = BFV.ciphertext_to_ints(forged, pk) + # Syntax validation cannot distinguish this from an encryption's origin. + observed = BFV.decrypt(BFV.ciphertext_from_ints(wire, pk), keys) + recovered = tuple(c if c <= t // 2 else c - t for c in observed) + assert recovered == keys["sk"]["s"] + + +def test_valid_distance_range_does_not_authenticate_answer(client: BFVClient) -> None: + pk, n = client._pk(), client.params.poly_modulus_degree + zero = BFVCiphertext(((mpz(0),) * n,) * 2, pk["q"], pk["key_id"]) + assert client.decode_hamming_client_one(BFV.ciphertext_to_ints(zero, pk)) == 0 + # Anyone with the public key can also forge a nontransparent plausible + # response. Rejecting all-zero ciphertexts would not solve authenticity. + forged = BFV.encrypt(BFV.batch_encode([1] * n, client.params), pk) + assert client.decode_hamming_client_one(BFV.ciphertext_to_ints(forged, pk)) == 1 + + +def test_response_is_not_bound_to_query(client: BFVClient) -> None: + index = client.encrypt_vec_one([0, 0, 0]) + old = client.encode_hamming_server(client.encrypt_vec_one([0, 0, 0]), index) + current = client.encode_hamming_server(client.encrypt_vec_one([1, 1, 1]), index) + assert client.decode_hamming_client_one(current) == 3 + assert client.decode_hamming_client_one(old) == 0 # Replay is still accepted. + + +def test_key_fingerprint_does_not_authenticate_evaluation_keys(client: BFVClient) -> None: + original = client.stringify_pk() + data = json.loads(original) + data["relin_key"][0][0][0] = format( + (mpz(data["relin_key"][0][0][0], 16) + client._pk()["q"] // 2) % client._pk()["q"], "x" + ) + modified = json.dumps(data) + loaded = BFV.deserialize_public_key(modified) + assert loaded["key_id"] == client._pk()["key_id"] + assert loaded["relin_key"] != client._pk()["relin_key"] + assert hashlib.sha256(original.encode()).digest() != hashlib.sha256(modified.encode()).digest() + + +def test_randomized_encryption_and_public_only_evaluation(client: BFVClient, monkeypatch) -> None: + query = client.encrypt_vec_one([0, 1, 0]) + encrypted = [client.encrypt_vec_one([1, 1, 1]) for _ in range(12)] + assert len({tuple(value) for value in encrypted}) == len(encrypted) + server = public_server(client, "optimized") + + def private_operation_forbidden(*args, **kwargs): + raise AssertionError("Server touched secret-key arithmetic") + + with monkeypatch.context() as guard: + guard.setattr(BFV, "_phase", private_operation_forbidden) + responses = [server.encode_hamming_server(query, value) for value in encrypted] + assert client.decode_hamming_client_batch(responses) == [2] * len(responses) + with pytest.raises(RuntimeError, match="public-only"): + server.decode_hamming_client_one(responses[0]) + + +def test_randomness_failure_does_not_fall_back_to_predictable_values(client, monkeypatch) -> None: + def unavailable(*args): + raise OSError("Synthetic OS entropy failure") + + monkeypatch.setattr(bfv.secrets, "randbelow", unavailable) + with pytest.raises(OSError, match="entropy"): + BFV.key_gen(16, 97, 100, 15) + with pytest.raises(OSError, match="entropy"): + client.encrypt_vec_one([0, 1, 0]) + + +def test_sampler_bounds_and_signed_binomial_mapping(monkeypatch) -> None: + bits = iter([0, 7, 7, 0, 7, 7, 0, 0]) + monkeypatch.setattr(bfv.secrets, "randbits", lambda eta: next(bits)) + assert bfv._small_poly(4, 3, mpz(97)) == (94, 3, 0, 0) + ternary = iter([0, 1, 2]) + monkeypatch.setattr(bfv.secrets, "randbelow", lambda bound: next(ternary)) + assert bfv._ternary_poly(3, mpz(97)) == (96, 0, 1) + + +@pytest.mark.parametrize("entry", range(8)) +def test_wire_zero_padding_is_bounded_before_integer_conversion(client, entry) -> None: + wire = client.encrypt_vec_one([0, 1, 0]) + n, bits = client.params.poly_modulus_degree, client._pk()["q"].bit_length() + limits = [4, 1, (n.bit_length() + 7) // 8, (bits + 7) // 8, 32, 1, n * bits // 8, n * bits // 8] + wire[entry] = wire[entry].to_bytes(limits[entry], "little") + b"\0" + with pytest.raises(ValueError, match="byte limit"): + BFV.ciphertext_from_ints(wire, client._pk()) + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + with pytest.raises(ValueError, match="byte limit"): + public_server(client)._native()._wire(wire) + + +@pytest.mark.parametrize("value", [True, 1.0, bytearray(b"\1"), -1, 1 << 256]) +def test_wire_rejects_ambiguous_or_out_of_bounds_values(client, value) -> None: + wire = client.encrypt_vec_one([0, 1, 0]) + wire[1] = value + with pytest.raises(ValueError): + BFV.ciphertext_from_ints(wire, client._pk()) + + +@pytest.mark.parametrize("quadratic", [False, True]) +def test_response_shape_checked_before_decryption(client, monkeypatch, quadratic) -> None: + pk = client._pk() + ct = BFV.ciphertext_from_ints(client.encrypt_vec_one([0, 1, 0]), pk) + if quadratic: + ct = replace(ct, components=(*ct.components, (mpz(0),) * 16)) + else: + ct = BFV.modulus_switch(ct, 35, pk) # Valid raw BFV, wrong negotiated response modulus. + wire = BFV.ciphertext_to_ints(ct, pk) + monkeypatch.setattr(BFV, "decrypt", lambda *args: pytest.fail("Unexpected private operation")) + with pytest.raises(ValueError, match="Unexpected BFV response"): + client.decode_hamming_client_one(wire) + + +@pytest.mark.parametrize("private", [False, True]) +def test_key_json_size_limit_checked_before_parser(client, monkeypatch, private) -> None: + encoded = client.stringify_sk() if private else client.stringify_pk() + loader = ( + (lambda s, **kw: BFV.deserialize_secret_key(s, client._pk(), **kw)) + if private + else BFV.deserialize_public_key + ) + with monkeypatch.context() as guard: + guard.setattr( + bfv.json, "loads", lambda *args, **kwargs: pytest.fail("Oversized JSON was parsed") + ) + with pytest.raises(ValueError, match="character limit"): + loader(encoded, max_chars=len(encoded) - 1) + assert loader(encoded, max_chars=len(encoded)) + + +@pytest.mark.parametrize("private", [False, True]) +@pytest.mark.parametrize( + "bad", ['{"version":1,"version":1}', "[" * 1100 + "0" + "]" * 1100, "null", "[]"] +) +def test_key_json_duplicate_fields_and_nesting_rejected(client, private, bad) -> None: + with pytest.raises(ValueError): + if private: + BFV.deserialize_secret_key(bad, client._pk()) + else: + BFV.deserialize_public_key(bad) + + +@pytest.mark.parametrize("private", [False, True]) +def test_boolean_key_version_rejected(client, private) -> None: + data = json.loads(client.stringify_sk() if private else client.stringify_pk()) + data["version"] = True + with pytest.raises(ValueError, match="version"): + if private: + BFV.deserialize_secret_key(json.dumps(data), client._pk()) + else: + BFV.deserialize_public_key(json.dumps(data)) + + +@pytest.mark.parametrize( + "coefficient", ["", "00", "+1", "-0", "0x1", " 1", "1_0", "F", "0" * 10000] +) +def test_key_coefficients_have_bounded_canonical_hex(client, coefficient) -> None: + data = json.loads(client.stringify_pk()) + data["a"][0] = coefficient + with pytest.raises(ValueError, match="key polynomial"): + BFV.deserialize_public_key(json.dumps(data)) + + +@pytest.mark.parametrize("exponent", ["03", "+3", "3_0", "٣", "0" * 10000, "2", "33"]) +def test_galois_names_do_not_alias_or_escape_ring(client, exponent) -> None: + data = json.loads(client.stringify_pk()) + data["galois_keys"] = {exponent: next(iter(data["galois_keys"].values()))} + with pytest.raises(ValueError, match="Galois"): + BFV.deserialize_public_key(json.dumps(data)) + + +def test_failed_key_load_preserves_client_state(client) -> None: + fresh = public_server(client, "optimized") + before = fresh._pk() + with pytest.raises(ValueError): + fresh.load_stringified_keys(client.stringify_pk(), '{"s":[]}') + assert fresh._pk() is before and fresh.keys is None + + +@pytest.mark.parametrize("text", ["0" * 10000 + "3", "f" * 129, "+3", " 3", "3\0", "0x3"]) +def test_native_modulus_text_bounded_before_gmp(client, text) -> None: + native = pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + with pytest.raises(ValueError, match="modulus"): + native.create_ring(16, text, 15) + server = public_server(client)._native() + with pytest.raises(ValueError, match="modulus"): + native.create_server( + server.arithmetic._prepare_key(0, client._pk()["relin_key"]), (), 4, 97, text + ) + + +@pytest.mark.parametrize("level", [0, 1, 2]) +def test_mutated_wire_corpus_agrees_with_reference(client, level) -> None: + """Bounded mutation fuzzing, including arbitrary valid attacker ciphertexts.""" + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + server = public_server(client, level=level) + reference = public_server(client, "reference") + wire = client.encrypt_vec_one([0, 1, 0]) + rng = random.Random(20260910) + for sample in range(150): + candidate = wire[:] + at = rng.randrange(8) + candidate[at] ^= 1 << rng.randrange(max(1, candidate[at].bit_length() + 2)) + if sample % 3 == 0: + candidate[at] = candidate[at].to_bytes((candidate[at].bit_length() + 7) // 8, "little") + try: + expected = reference.encode_hamming_server(candidate, wire) + except ValueError: + with pytest.raises(ValueError): + server.encode_hamming_server(candidate, wire) + else: + assert server.encode_hamming_server(candidate, wire) == expected + + +def test_native_coefficient_bounds_before_kernels(client) -> None: + native = pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + server = public_server(client)._native() + arithmetic, q = server.arithmetic, client._pk()["q"] + zero = bytes(arithmetic.n * arithmetic.width) + rng = random.Random(6421) + for sample in range(100): + coefficients = [mpz(rng.randrange(int(q))) for _ in range(arithmetic.n)] + if sample % 2: + coefficients[sample % arithmetic.n] = q # Must not be silently reduced. + data = gmpy2.pack(coefficients, 8 * arithmetic.width).to_bytes(len(zero), "little") + if sample % 2: + with pytest.raises(ValueError, match="Noncanonical"): + native.ring_product(arithmetic._ring, data, zero) + else: + assert native.ring_product(arithmetic._ring, data, zero) == zero + + +def test_direct_ciphertext_rejects_noninteger_modulus(client) -> None: + pk = client._pk() + ct = BFV.ciphertext_from_ints(client.encrypt_vec_one([0, 1, 0]), pk) + ct = replace(ct, modulus=float(pk["q"] // 2)) + with pytest.raises(ValueError, match="integer"): + BFV.decrypt(ct, client._keys()) + with pytest.raises(ValueError, match="integer"): + BFVEvaluator(pk).add(ct, ct) + + +def test_default_rns_circuit_structured_inputs_across_fresh_keys() -> None: + """Exercise real defaults and extreme plaintext patterns; no failure-rate claim.""" + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + rng = random.Random(314159) + patterns = [ + [0] * 512, + [1] * 512, + [0, 1] * 256, + [1] + [0] * 511, + [0] * 511 + [1], + [rng.randrange(2) for _ in range(512)], + ] + vectors = (patterns * 6)[:33] # Full tiles plus a partial tile and merge. + for _ in range(2): + private = BFVClient(rns_modulus=True) + server = BFVClient(skip_key_gen=True, rns_modulus=True, server_backend="residue") + server.public_key = private._pk() # Trusted generated PK; contains no secret. + index = private.encrypt_vec_packed(vectors) + for query in patterns: + response = server.encode_hamming_server_packed( + private.encrypt_vec_one(query), index, len(vectors) + ) + expected = [sum(a != b for a, b in zip(query, v, strict=True)) for v in vectors] + assert private.decode_hamming_client_packed(response, len(vectors)) == expected + assert ( + BFV.noise_budget( + BFV.ciphertext_from_ints(response[0], private._pk()), private._keys() + ) + > 0 + ) + assert server.keys is None From 0cd7b41454533ab27082667cb9cb2f3c8040e823 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Fri, 11 Sep 2026 00:30:30 -0400 Subject: [PATCH 09/12] Add authenticated BFV sessions, private result checks and protected exports --- .github/workflows/ci.yml | 2 +- benchmarks/bfv_verified.py | 243 +++++++++ .../results/native_bfv_verified_1024.json | 269 ++++++++++ .../results/native_bfv_verified_32.json | 269 ++++++++++ docs/index.rst | 1 + docs/research/native-bfv-security.md | 9 + docs/research/native-bfv-verified-sessions.md | 348 +++++++++++++ docs/research/native-bfv.md | 5 + src/xtrace_sdk/x_vec/crypto/__init__.py | 4 + src/xtrace_sdk/x_vec/crypto/bfv_security.py | 238 +++++++++ .../x_vec/crypto/bfv_verified_client.py | 465 ++++++++++++++++++ tests/x_vec/native/run_binding_sanitizers.py | 1 + tests/x_vec/test_bfv_verified_client.py | 461 +++++++++++++++++ 13 files changed, 2314 insertions(+), 1 deletion(-) create mode 100644 benchmarks/bfv_verified.py create mode 100644 benchmarks/results/native_bfv_verified_1024.json create mode 100644 benchmarks/results/native_bfv_verified_32.json create mode 100644 docs/research/native-bfv-verified-sessions.md create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_security.py create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py create mode 100644 tests/x_vec/test_bfv_verified_client.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8ccbe04..5a7e740 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -68,7 +68,7 @@ jobs: sudo apt-get install -y libgmp-dev make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 4' - uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py tests/x_vec/test_bfv_security.py + uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py tests/x_vec/test_bfv_security.py tests/x_vec/test_bfv_verified_client.py - name: BFV native arithmetic sanitizers run: | diff --git a/benchmarks/bfv_verified.py b/benchmarks/bfv_verified.py new file mode 100644 index 0000000..6eb2cb5 --- /dev/null +++ b/benchmarks/bfv_verified.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +"""Measure authenticated/result-checked BFV sessions against the same raw inputs. + +Both servers use the unchanged residue kernels and identical keys/ciphertexts. +Run without concurrent benchmarks or tests. Only synthetic plaintext is seeded. +No key, ciphertext, verification seed or private fingerprint is written to JSON. +""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.metadata +import json +import platform +import secrets +import statistics +import subprocess +import sys +from datetime import UTC, datetime +from functools import partial +from pathlib import Path +from typing import Any + +import gmpy2 + +from bfv_client_matrix import REPO_ROOT, make_data, packet, time_call +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + _authenticated_body, + _decode_ciphertexts, + _encode_ciphertexts, + _unpack, +) +from xtrace_sdk.x_vec.crypto.bfv_verified_client import ( + BFVVerifiedClient, + BFVVerifiedServer, + _HammingSummary, +) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--num-vectors", type=int, default=1024) + parser.add_argument("--embed-len", type=int, default=512) + parser.add_argument("--seed", type=int, default=1337) + parser.add_argument("--repeats", type=int, default=6, help="Includes one cold search") + parser.add_argument("--json-out", type=Path) + args = parser.parse_args() + if args.num_vectors < 1 or args.embed_len < 1 or args.repeats < 2: + parser.error("positive dimensions and at least two repeats are required") + policy = BFVExecutionPolicy(embed_len=args.embed_len) + if args.num_vectors > policy.max_vectors or args.repeats > policy.max_queries: + parser.error("workload exceeds the local session policy") + vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) + timings: dict[str, list[float]] = { + name: [] for name in ("raw_server", "verified_server", "raw_client", "verified_client") + } + setup_times: dict[str, float] = {} + print("Preparing shared keys, encrypted index and private summary", file=sys.stderr, flush=True) + setup_times["key_generation_s"], private = time_call(lambda: BFVClient(**policy.config())) + auth = secrets.token_bytes(32) + client = BFVVerifiedClient(private, auth, policy=policy) + setup_times["encrypt_index_summary_export_auth_s"], setup = time_call( + lambda: client.prepare_index(vectors) + ) + setup_times["authenticate_import_setup_s"], verified = time_call( + lambda: BFVVerifiedServer(setup, auth, policy=policy) + ) + raw = BFVClient(**policy.config(), skip_key_gen=True, server_backend="residue") + # Trusted benchmark-only sharing of the same parsed PUBLIC key/index. Both + # server instances retain independent arithmetic contexts and lazy caches. + raw.public_key = verified._client._pk() + assert raw.keys is None and verified._client.keys is None + pk, index = raw._pk(), verified._index + setup_times["raw_arithmetic_context_s"], _ = time_call(raw._evaluator) + setup_times["verified_arithmetic_context_s"], _ = time_call(verified._client._evaluator) + setup_times["summary_only_preprocess_s"], _ = time_call( + lambda: _HammingSummary(vectors, args.embed_len, secrets.token_bytes(16)) + ) + summary_samples, runs, query_samples = [], [], [] + raw_response = b"" + response = b"" + request = b"" + query_wire = b"" + count = ( + len(vectors) + policy.params.poly_modulus_degree - 1 + ) // policy.params.poly_modulus_degree + + def raw_search(wire: bytes) -> bytes: + q = _decode_ciphertexts(wire, pk, 1, policy.max_query_bytes)[0] + result = raw.encode_hamming_server_packed(q, index, len(vectors)) + return _encode_ciphertexts(result, pk) + + def raw_finish(wire: bytes) -> list[int]: + cts = _decode_ciphertexts(wire, pk, count, policy.max_response_bytes) + return private.decode_hamming_client_packed(cts, len(vectors)) + + for repeat in range(args.repeats): + elapsed, request = time_call(lambda: client.begin_query(query)) + query_samples.append(elapsed) + query_wire = _unpack( + _authenticated_body(request, auth, 2, policy.max_query_bytes), + limit=policy.max_query_bytes, + )[2] + + order = ("raw", "verified") if repeat % 2 == 0 else ("verified", "raw") + for variant in order: + elapsed, result = time_call( + partial(raw_search, query_wire) + if variant == "raw" + else partial(verified.search, request) + ) + timings[f"{variant}_server"].append(elapsed) + runs.append({"repeat": repeat, "phase": f"{variant}_server", "seconds": elapsed}) + if variant == "raw": + raw_response = result + else: + response = result + print(f"{variant} server: {elapsed:.6f} s", file=sys.stderr, flush=True) + response_wire = _unpack( + _authenticated_body(response, auth, 3, policy.max_response_bytes), + limit=policy.max_response_bytes, + )[3] + assert response_wire == raw_response, "Wrapper changed the evaluated ciphertexts" + for variant in reversed(order): + elapsed, distances = time_call( + partial(raw_finish, raw_response) + if variant == "raw" + else partial(client.finish_query, response) + ) + assert distances == expected, "Incorrect Hamming distances" + timings[f"{variant}_client"].append(elapsed) + runs.append({"repeat": repeat, "phase": f"{variant}_client", "seconds": elapsed}) + assert client._summary is not None + elapsed, valid = time_call( + lambda: ( + client._summary is not None + and client._summary.matches(expected, client._summary.expected(query)) + ) + ) + assert valid + summary_samples.append(elapsed) + + summary = { + name: { + "first_s": samples[0], + "warm_samples_s": samples[1:], + "warm_median_s": statistics.median(samples[1:]), + } + for name, samples in timings.items() + } + setup_fields = _unpack( + _authenticated_body(setup, auth, 1, policy.max_setup_bytes), + limit=policy.max_setup_bytes, + array_limit=policy.max_vectors, + ) + public_bytes, index_bytes = len(setup_fields[2]), len(setup_fields[5]) + query_cts = _decode_ciphertexts(query_wire, pk, 1, policy.max_query_bytes) + response_cts = _decode_ciphertexts(raw_response, pk, count, policy.max_response_bytes) + protected_s, protected = time_call(lambda: client.protect_state(secrets.token_bytes(32))) + sources = [ + Path(__file__).resolve(), + REPO_ROOT / "benchmarks/bfv_client_matrix.py", + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto").glob("bfv*.py"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption").glob("bfv*.py"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.cpp"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.h"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.so"), + REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile", + REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", + ] + output: dict[str, Any] = { + "environment": { + "utc": datetime.now(UTC).isoformat(), + "python": platform.python_version(), + "platform": platform.platform(), + "gmpy2": gmpy2.version(), + "gmp": gmpy2.mp_version(), + "pycryptodome": importlib.metadata.version("pycryptodome"), + "msgpack": importlib.metadata.version("msgpack"), + "revision": subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip(), + "working_tree_dirty": bool( + subprocess.run( + ["git", "status", "--porcelain"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip() + ), + }, + "command": sys.argv, + "config": policy.config(), + "seed": args.seed, + "vectors": len(vectors), + "measurement_notes": "Identical keys and ciphertexts, independent public-only residue server contexts. Both server timers include MessagePack ciphertext decode/encode and native arithmetic; verified additionally includes authentication, session checks and outer framing. Index import, arithmetic context creation, encryption and private decryption are excluded from server times. First search has cold key/mask caches; five warm samples by default, alternating order. Client timers include ciphertext decode/decrypt/slot decode; verified adds authentication, ticket handling and private result checking. Query encryption is separately timed outside these phases. No network, remote endpoint, SEAL, security-level claim or constant-time claim.", + "setup_timings": setup_times, + "runs": runs, + "summary": summary, + "query_encrypt_and_bind_samples_s": query_samples, + "summary_only_check_samples_s": summary_samples, + "protect_private_state_s": protected_s, + "sizes": { + "public_keys_bytes": public_bytes, + "index_ciphertext_array_bytes": index_bytes, + "authenticated_setup_bytes": len(setup), + "raw_query_array_bytes": len(query_wire), + "raw_response_array_bytes": len(raw_response), + "authenticated_query_bytes": len(request), + "authenticated_response_bytes": len(response), + "authenticated_query_plus_response_bytes": len(request) + len(response), + "query_authentication_and_framing_overhead_bytes": len(request) - len(query_wire), + "response_authentication_and_framing_overhead_bytes": len(response) - len(raw_response), + "private_verification_payload_bytes": client.verification_state_bytes, + "protected_private_state_bytes": len(protected), + "previous_benchmark_encrypted_index_bytes": len(packet(index, len(vectors))), + "previous_benchmark_query_bytes": len(packet(query_cts, 1)), + "previous_benchmark_response_bytes": len(packet(response_cts, len(vectors))), + }, + "identical_ciphertexts": True, + "all_distances_correct": True, + "source_sha256": { + str(p.relative_to(REPO_ROOT)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in sources + }, + } + encoded = json.dumps(output, indent=2) + print(encoded) + if args.json_out: + args.json_out.parent.mkdir(parents=True, exist_ok=True) + args.json_out.write_text(encoded + "\n") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/results/native_bfv_verified_1024.json b/benchmarks/results/native_bfv_verified_1024.json new file mode 100644 index 0000000..3075bde --- /dev/null +++ b/benchmarks/results/native_bfv_verified_1024.json @@ -0,0 +1,269 @@ +{ + "environment": { + "utc": "2026-09-11T04:24:09.304787+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "pycryptodome": "3.23.0", + "msgpack": "1.1.2", + "revision": "b7841e45333772d1e470939a1c2351fea17a639c", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_verified.py", + "--num-vectors", + "1024", + "--repeats", + "6", + "--json-out", + "benchmarks/results/native_bfv_verified_1024.json" + ], + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "seed": 1337, + "vectors": 1024, + "measurement_notes": "Identical keys and ciphertexts, independent public-only residue server contexts. Both server timers include MessagePack ciphertext decode/encode and native arithmetic; verified additionally includes authentication, session checks and outer framing. Index import, arithmetic context creation, encryption and private decryption are excluded from server times. First search has cold key/mask caches; five warm samples by default, alternating order. Client timers include ciphertext decode/decrypt/slot decode; verified adds authentication, ticket handling and private result checking. Query encryption is separately timed outside these phases. No network, remote endpoint, SEAL, security-level claim or constant-time claim.", + "setup_timings": { + "key_generation_s": 4.339769265148789, + "encrypt_index_summary_export_auth_s": 8.079525091918185, + "authenticate_import_setup_s": 1.7375480628106743, + "raw_arithmetic_context_s": 0.0016010950785130262, + "verified_arithmetic_context_s": 0.0009621409699320793, + "summary_only_preprocess_s": 0.025153255090117455 + }, + "runs": [ + { + "repeat": 0, + "phase": "raw_server", + "seconds": 2.6571030700579286 + }, + { + "repeat": 0, + "phase": "verified_server", + "seconds": 2.6737288739532232 + }, + { + "repeat": 0, + "phase": "verified_client", + "seconds": 0.041729690972715616 + }, + { + "repeat": 0, + "phase": "raw_client", + "seconds": 0.03954386292025447 + }, + { + "repeat": 1, + "phase": "verified_server", + "seconds": 2.3735971769783646 + }, + { + "repeat": 1, + "phase": "raw_server", + "seconds": 2.371556321159005 + }, + { + "repeat": 1, + "phase": "raw_client", + "seconds": 0.03905453206971288 + }, + { + "repeat": 1, + "phase": "verified_client", + "seconds": 0.04158204700797796 + }, + { + "repeat": 2, + "phase": "raw_server", + "seconds": 2.3573059749323875 + }, + { + "repeat": 2, + "phase": "verified_server", + "seconds": 2.3584984519984573 + }, + { + "repeat": 2, + "phase": "verified_client", + "seconds": 0.04113910696469247 + }, + { + "repeat": 2, + "phase": "raw_client", + "seconds": 0.03929260699078441 + }, + { + "repeat": 3, + "phase": "verified_server", + "seconds": 2.3811275989282876 + }, + { + "repeat": 3, + "phase": "raw_server", + "seconds": 2.3726596529595554 + }, + { + "repeat": 3, + "phase": "raw_client", + "seconds": 0.03970337612554431 + }, + { + "repeat": 3, + "phase": "verified_client", + "seconds": 0.04124582908116281 + }, + { + "repeat": 4, + "phase": "raw_server", + "seconds": 2.3679882378783077 + }, + { + "repeat": 4, + "phase": "verified_server", + "seconds": 2.3805392258800566 + }, + { + "repeat": 4, + "phase": "verified_client", + "seconds": 0.041710648918524384 + }, + { + "repeat": 4, + "phase": "raw_client", + "seconds": 0.03959750197827816 + }, + { + "repeat": 5, + "phase": "verified_server", + "seconds": 2.373485399177298 + }, + { + "repeat": 5, + "phase": "raw_server", + "seconds": 2.367954927030951 + }, + { + "repeat": 5, + "phase": "raw_client", + "seconds": 0.03955245087854564 + }, + { + "repeat": 5, + "phase": "verified_client", + "seconds": 0.04190468299202621 + } + ], + "summary": { + "raw_server": { + "first_s": 2.6571030700579286, + "warm_samples_s": [ + 2.371556321159005, + 2.3573059749323875, + 2.3726596529595554, + 2.3679882378783077, + 2.367954927030951 + ], + "warm_median_s": 2.3679882378783077 + }, + "verified_server": { + "first_s": 2.6737288739532232, + "warm_samples_s": [ + 2.3735971769783646, + 2.3584984519984573, + 2.3811275989282876, + 2.3805392258800566, + 2.373485399177298 + ], + "warm_median_s": 2.3735971769783646 + }, + "raw_client": { + "first_s": 0.03954386292025447, + "warm_samples_s": [ + 0.03905453206971288, + 0.03929260699078441, + 0.03970337612554431, + 0.03959750197827816, + 0.03955245087854564 + ], + "warm_median_s": 0.03955245087854564 + }, + "verified_client": { + "first_s": 0.041729690972715616, + "warm_samples_s": [ + 0.04158204700797796, + 0.04113910696469247, + 0.04124582908116281, + 0.041710648918524384, + 0.04190468299202621 + ], + "warm_median_s": 0.04158204700797796 + } + }, + "query_encrypt_and_bind_samples_s": [ + 0.10001991898752749, + 0.09990808507427573, + 0.09966578800231218, + 0.09928345982916653, + 0.10008095717057586, + 0.09987170086242259 + ], + "summary_only_check_samples_s": [ + 0.002144736936315894, + 0.0021136258728802204, + 0.0021337070502340794, + 0.0021290569566190243, + 0.002179748145863414, + 0.002148417057469487 + ], + "protect_private_state_s": 0.0023410909343510866, + "sizes": { + "public_keys_bytes": 85603219, + "index_ciphertext_array_bytes": 23598467, + "authenticated_setup_bytes": 109205980, + "raw_query_array_bytes": 368727, + "raw_response_array_bytes": 102467, + "authenticated_query_bytes": 368840, + "authenticated_response_bytes": 102614, + "authenticated_query_plus_response_bytes": 471454, + "query_authentication_and_framing_overhead_bytes": 113, + "response_authentication_and_framing_overhead_bytes": 147, + "private_verification_payload_bytes": 16448, + "protected_private_state_bytes": 36729, + "previous_benchmark_encrypted_index_bytes": 23598496, + "previous_benchmark_query_bytes": 368754, + "previous_benchmark_response_bytes": 102496 + }, + "identical_ciphertexts": true, + "all_distances_correct": true, + "source_sha256": { + "benchmarks/bfv_verified.py": "1578cf1c019159a9856bc31d0f3561499ce10b5d024eb2d24682f14a79b6884e", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "e781be6172b9f1ad676fa4ac5f706be7035563bc230e285a56b90129bf18b654", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "05e30fc3b8191e2fd0f0620ac7dbbc35ccea0a43913fc62f66575b0d44716aeb", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962" + } +} diff --git a/benchmarks/results/native_bfv_verified_32.json b/benchmarks/results/native_bfv_verified_32.json new file mode 100644 index 0000000..84e93e7 --- /dev/null +++ b/benchmarks/results/native_bfv_verified_32.json @@ -0,0 +1,269 @@ +{ + "environment": { + "utc": "2026-09-11T04:24:37.184175+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "pycryptodome": "3.23.0", + "msgpack": "1.1.2", + "revision": "b7841e45333772d1e470939a1c2351fea17a639c", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_verified.py", + "--num-vectors", + "32", + "--repeats", + "6", + "--json-out", + "benchmarks/results/native_bfv_verified_32.json" + ], + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "seed": 1337, + "vectors": 32, + "measurement_notes": "Identical keys and ciphertexts, independent public-only residue server contexts. Both server timers include MessagePack ciphertext decode/encode and native arithmetic; verified additionally includes authentication, session checks and outer framing. Index import, arithmetic context creation, encryption and private decryption are excluded from server times. First search has cold key/mask caches; five warm samples by default, alternating order. Client timers include ciphertext decode/decrypt/slot decode; verified adds authentication, ticket handling and private result checking. Query encryption is separately timed outside these phases. No network, remote endpoint, SEAL, security-level claim or constant-time claim.", + "setup_timings": { + "key_generation_s": 4.338192637078464, + "encrypt_index_summary_export_auth_s": 1.3449218550231308, + "authenticate_import_setup_s": 1.6677678199484944, + "raw_arithmetic_context_s": 0.0015940847806632519, + "verified_arithmetic_context_s": 0.0009516510181128979, + "summary_only_preprocess_s": 0.0008359679486602545 + }, + "runs": [ + { + "repeat": 0, + "phase": "raw_server", + "seconds": 0.3808726631104946 + }, + { + "repeat": 0, + "phase": "verified_server", + "seconds": 0.38237067707814276 + }, + { + "repeat": 0, + "phase": "verified_client", + "seconds": 0.03770374902524054 + }, + { + "repeat": 0, + "phase": "raw_client", + "seconds": 0.03787448303773999 + }, + { + "repeat": 1, + "phase": "verified_server", + "seconds": 0.07546799909323454 + }, + { + "repeat": 1, + "phase": "raw_server", + "seconds": 0.07493602787144482 + }, + { + "repeat": 1, + "phase": "raw_client", + "seconds": 0.037949763936921954 + }, + { + "repeat": 1, + "phase": "verified_client", + "seconds": 0.03785317204892635 + }, + { + "repeat": 2, + "phase": "raw_server", + "seconds": 0.07507768110372126 + }, + { + "repeat": 2, + "phase": "verified_server", + "seconds": 0.07534101605415344 + }, + { + "repeat": 2, + "phase": "verified_client", + "seconds": 0.03754223510622978 + }, + { + "repeat": 2, + "phase": "raw_client", + "seconds": 0.037920583970844746 + }, + { + "repeat": 3, + "phase": "verified_server", + "seconds": 0.07540737697854638 + }, + { + "repeat": 3, + "phase": "raw_server", + "seconds": 0.07481546490453184 + }, + { + "repeat": 3, + "phase": "raw_client", + "seconds": 0.037693017860874534 + }, + { + "repeat": 3, + "phase": "verified_client", + "seconds": 0.03777332999743521 + }, + { + "repeat": 4, + "phase": "raw_server", + "seconds": 0.07513878191821277 + }, + { + "repeat": 4, + "phase": "verified_server", + "seconds": 0.07539898692630231 + }, + { + "repeat": 4, + "phase": "verified_client", + "seconds": 0.03783954097889364 + }, + { + "repeat": 4, + "phase": "raw_client", + "seconds": 0.03770379885099828 + }, + { + "repeat": 5, + "phase": "verified_server", + "seconds": 0.07531027519144118 + }, + { + "repeat": 5, + "phase": "raw_server", + "seconds": 0.07499737804755569 + }, + { + "repeat": 5, + "phase": "raw_client", + "seconds": 0.03793373401276767 + }, + { + "repeat": 5, + "phase": "verified_client", + "seconds": 0.03781877108849585 + } + ], + "summary": { + "raw_server": { + "first_s": 0.3808726631104946, + "warm_samples_s": [ + 0.07493602787144482, + 0.07507768110372126, + 0.07481546490453184, + 0.07513878191821277, + 0.07499737804755569 + ], + "warm_median_s": 0.07499737804755569 + }, + "verified_server": { + "first_s": 0.38237067707814276, + "warm_samples_s": [ + 0.07546799909323454, + 0.07534101605415344, + 0.07540737697854638, + 0.07539898692630231, + 0.07531027519144118 + ], + "warm_median_s": 0.07539898692630231 + }, + "raw_client": { + "first_s": 0.03787448303773999, + "warm_samples_s": [ + 0.037949763936921954, + 0.037920583970844746, + 0.037693017860874534, + 0.03770379885099828, + 0.03793373401276767 + ], + "warm_median_s": 0.037920583970844746 + }, + "verified_client": { + "first_s": 0.03770374902524054, + "warm_samples_s": [ + 0.03785317204892635, + 0.03754223510622978, + 0.03777332999743521, + 0.03783954097889364, + 0.03781877108849585 + ], + "warm_median_s": 0.03781877108849585 + } + }, + "query_encrypt_and_bind_samples_s": [ + 0.10028813499957323, + 0.0998352849856019, + 0.0996188500430435, + 0.10010351985692978, + 0.10015947092324495, + 0.09966332116164267 + ], + "summary_only_check_samples_s": [ + 0.0003099560271948576, + 0.00028055603615939617, + 0.00035280780866742134, + 0.00031211692839860916, + 0.00031296699307858944, + 0.00032019708305597305 + ], + "protect_private_state_s": 0.0022762599401175976, + "sizes": { + "public_keys_bytes": 85603214, + "index_ciphertext_array_bytes": 737453, + "authenticated_setup_bytes": 86341035, + "raw_query_array_bytes": 368727, + "raw_response_array_bytes": 102467, + "authenticated_query_bytes": 368840, + "authenticated_response_bytes": 102614, + "authenticated_query_plus_response_bytes": 471454, + "query_authentication_and_framing_overhead_bytes": 113, + "response_authentication_and_framing_overhead_bytes": 147, + "private_verification_payload_bytes": 16448, + "protected_private_state_bytes": 36713, + "previous_benchmark_encrypted_index_bytes": 737480, + "previous_benchmark_query_bytes": 368754, + "previous_benchmark_response_bytes": 102494 + }, + "identical_ciphertexts": true, + "all_distances_correct": true, + "source_sha256": { + "benchmarks/bfv_verified.py": "1578cf1c019159a9856bc31d0f3561499ce10b5d024eb2d24682f14a79b6884e", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "e781be6172b9f1ad676fa4ac5f706be7035563bc230e285a56b90129bf18b654", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "05e30fc3b8191e2fd0f0620ac7dbbc35ccea0a43913fc62f66575b0d44716aeb", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962" + } +} diff --git a/docs/index.rst b/docs/index.rst index 0aeabe7..2b902aa 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -87,6 +87,7 @@ Choose your path: research/native-bfv-residue research/native-bfv-fused-rns research/native-bfv-security + research/native-bfv-verified-sessions research/bfv-packed-hamming .. toctree:: diff --git a/docs/research/native-bfv-security.md b/docs/research/native-bfv-security.md index aee54ea..fda596c 100644 --- a/docs/research/native-bfv-security.md +++ b/docs/research/native-bfv-security.md @@ -7,6 +7,15 @@ implementation assurance remain open. This is an internal, AI-assisted source review and regression exercise, not an independent cryptographic audit, penetration test of a deployed service, or proof of security. +**2026-09-11 follow-up:** the optional +[authenticated, result-checked session layer](native-bfv-verified-sessions.md) +adds whole-setup authentication, one-use requests, a private Hamming result +check, encrypted private exports and bounded admission. It also demonstrates +that observable acceptance/rejection can still recover the secret key even +with those checks. The findings below describe the original raw BFV APIs; +the follow-up records the new mitigations, their performance and the remaining +production blockers. + The optimization baseline reviewed is commit `06d8d17`, following `e5c0751`. This report and the accompanying hardening are on `research/bfv-packed-hamming`. No production endpoint or `main` change is part diff --git a/docs/research/native-bfv-verified-sessions.md b/docs/research/native-bfv-verified-sessions.md new file mode 100644 index 0000000..c0c722e --- /dev/null +++ b/docs/research/native-bfv-verified-sessions.md @@ -0,0 +1,348 @@ +# BFV authenticated sessions and remaining security blockers + +Date: 2026-09-11. Branch: `research/bfv-packed-hamming`. + +**Several audit risks now have concrete SDK protections, with the existing +RNS evaluator preserved. All findings are not closed, and this implementation +remains experimental.** In particular, a new regression demonstrates secret-key +recovery from observable accept/reject feedback even through the new result +checker. This follow-up is internal, AI-assisted engineering and testing, not +an independent cryptographic audit or production certification. + +This extends the [original audit](native-bfv-security.md). The raw BFV, +Paillier and SEAL paths remain available. Existing `BFVClient` calls do not +automatically acquire these protections; applications must explicitly use the +new local session API and provide trusted key provisioning. + +## Code and usage + +| Path relative to the repository | Purpose | +| --- | --- | +| `src/xtrace_sdk/x_vec/crypto/bfv_security.py` | Locally pinned parameters/limits, bounded authenticated framing, AES-GCM private exports | +| `src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py` | Private result summary, client request lifecycle and public-only server wrapper | +| `tests/x_vec/test_bfv_verified_client.py` | Tampering/replay/storage/resource regressions and the residual oracle demonstration | +| `benchmarks/bfv_verified.py` | Same-key, same-ciphertext comparison against raw residue evaluation | + +The following is a local experiment. It deliberately contains no network +callbacks: distances, errors and result-dependent behavior must remain inside +the trusted client. Provision the authentication key to the intended evaluator +over a trusted authenticated channel; keep the wrapping key exclusively in +separate trusted client storage or a suitable key-management service. + +```python +import secrets + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_security import BFVExecutionPolicy +from xtrace_sdk.x_vec.crypto.bfv_verified_client import ( + BFVVerifiedClient, + BFVVerifiedServer, +) + +# The default policy pins the measured N=8192, Q=180-bit RNS configuration. +# This is an arithmetic/resource allowlist, not a 128-bit security claim. +policy = BFVExecutionPolicy() +authentication_key = secrets.token_bytes(32) # Shared with this evaluator. +wrapping_key = secrets.token_bytes(32) # Private; never sent to it. +raw_client = BFVClient(**policy.config()) +client = BFVVerifiedClient(raw_client, authentication_key, policy=policy) +setup = client.prepare_index( + [[0] * 512, [1] * 512], vector_ids=["document-a", "document-b"] +) + +# Separate public-only instance, using the existing C++ residue implementation. +server = BFVVerifiedServer(setup, authentication_key, policy=policy) +request = client.begin_query([0] * 512) +response = server.search(request) +distances = client.finish_query(response) # [0, 512], checked before return. +top3 = sorted(zip(client.vector_ids, distances), key=lambda item: (item[1], item[0]))[:3] + +# Caller persists these bytes securely; no file is written by the SDK. +protected = client.protect_state(wrapping_key) +restored = BFVVerifiedClient.restore( + setup, protected, authentication_key, wrapping_key, policy=policy +) +``` + +The compiled backend is built with the existing command: + +```bash +make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" +``` + +For CPU-only fallback, explicitly select `backend="optimized"` on +`BFVVerifiedServer`. The wrapper snapshots the private client's key dictionaries; +reloading the original `BFVClient` cannot silently change the active session. +An index is immutable within a session. Rebuild setup and its fresh private +summary for an updated index. + +## What each protection does + +**Whole-setup and message authentication.** A versioned, domain-separated +HMAC-SHA256 envelope covers the exact setup bytes: parameters, layout, +encryption keys, all evaluation keys, index ciphertexts, ordered unique IDs, +vector count and a fresh session identifier. Each request binds that setup's +SHA-256 digest and a fresh 256-bit ticket; each response additionally binds +the exact authenticated request's digest. Incoming byte limits and HMAC are +checked before MessagePack or key JSON parsing. The unkeyed BFV key ID remains +a compatibility check, not an ownership credential. + +The caller supplies an independent, uniformly generated 32-byte authentication +key. Use distinct authorization/key domains for distinct data owners. The +server knows this key and can authenticate a false answer; HMAC protects +against outsiders and mixups, not dishonesty by the authenticated evaluator. +Setup authentication cannot certify that a malicious key generator sampled +valid keys or that an authorized client submits only permitted queries. + +**One-use requests and bounded admission.** The client records at most eight +pending requests by default. An authenticated response bound to a pending +request consumes its ticket before any ciphertext decryption, including on a +subsequent rejection. A concurrent duplicate cannot trigger a second private +decryption. An unauthenticated or unrelated packet cannot consume a legitimate +ticket. Each client/server instance admits at most 65,536 queries; cancellation +does not reset the client count. The server defaults to one concurrent search +and rejects work immediately when busy. Its replay set is bounded by the query +budget. + +Default caps are 65,536 vectors, 384 MiB setup, 128 MiB public JSON, 256 MiB +encoded index, 1 MiB query and 16 MiB response. These are independent upper +bounds, not a promise that every combination fits. The wrapper requires the +exact configured terminal response modulus and expected ciphertext count, +then inherits canonical coefficient/shape/range checks from the raw BFV code. +An explicit local policy can select tiny test parameters; received setup +cannot change the policy. A policy is not a security-level certificate. + +Budgets and replay state are per object and in memory. Protected client state +preserves the count of started requests, but discards pending tickets on +restore. It has **no anti-rollback mechanism**. A service still needs durable +authorization/replay state or a fresh client-authorized setup after restart, +limits before transport buffering, aggregate memory/CPU quotas and admission +across tenants. Reinstantiating an object must not reset service quotas. + +**Encrypted private storage.** AES-256-GCM uses a fresh 96-bit nonce and a +128-bit tag, authenticating before plaintext parsing. `protect_state` encrypts +the BFV secret key, verification summary and used query count, with the entire +authenticated setup digest as associated context. The wrapping key must +differ from the server authentication key. For a raw `BFVClient`, +`protect_bfv_secret_key` and `restore_bfv_client` provide a separate encrypted +export bound to the exact full public JSON, including every evaluation key. +This uses the existing PyCryptodome dependency's authenticated encryption API; +it is not a new block cipher or password-based key derivation scheme. +[PyCryptodome authenticated-mode documentation](https://www.pycryptodome.org/src/cipher/modern). + +These APIs protect stored bytes. Existing explicit `stringify_sk()` exports +remain plaintext, and Python/GMP memory, temporary copies, crash dumps, +wrapping-key custody and local timing/cache behavior remain separate concerns. +No zeroization or constant-time guarantee is added. + +## Private Hamming result check + +For binary database rows `x_i` and query `q` of dimension D, Hamming distance +is `d_i = |x_i| + |q| - 2 * sum_j(x_ij*q_j)`. At index creation the client +generates an independent private seed and derives a pseudorandom weight `r_i` +for each row in the prime field `p = 2^255 - 19`. It retains: + +```text +R = sum_i r_i mod p +S_j = sum_i r_i * x_ij mod p + +expected(q) = sum_j S_j + R*|q| - 2*sum_j S_j*q_j mod p +actual(d) = sum_i r_i*d_i mod p +``` + +The client checks equality only after decrypting and range-checking the full +answer, and returns no unchecked or partial distance vector. Swapping unequal +distances or replacing them with plausible constant results changes this +fingerprint except for a collision. The seed and summary never go to the +evaluator; disclosing the columns can also disclose information about the +database, so they are encrypted alongside the secret key when persisted. + +This adapts private preprocessing for verifying a fixed linear computation, +as described in Slalom's Lemma 3.1 and Appendix B. The Hamming identity above +is our derivation for this application; the Slalom paper does **not** prove +security for this BFV protocol or make it chosen-ciphertext secure. +[Tramèr and Boneh, Slalom, ICLR 2019](https://arxiv.org/pdf/1806.03287). + +With independent uniform field weights, a fixed nonzero result error passes +one linear check with probability `1/p`. Finite repeated checks require a +query-bound analysis; using HMAC as a pseudorandom function adds that +computational assumption. HMAC-SHA256 is keyed with the private seed and binds +the session, row index and a rejection-sampling counter. Sampling rejects +values outside the field, with a bounded retry count that fails closed. +This reasoning motivates the experimental check, not a reviewed security +theorem for its implementation, key reuse or the complete application. The +255-bit field is unrelated to BFV's plaintext modulus and does not establish +255-bit or 128-bit BFV security. + +Preprocessing is O(MD); online checking is O(M+D), with no additional server +arithmetic or verification ciphertext. The binary summary payload is +`32*(D+2)` bytes: seed, R and D columns. At D=512 this is **16,448 bytes**, +excluding IDs, session metadata and Python object overhead. The client need +not retain the original binary vectors after setup. + +## The oracle blocker still exists + +`test_acceptance_feedback_still_allows_key_recovery` intentionally demonstrates +an attack on the new wrapper when the evaluator can observe whether a response +was accepted. It uses fresh toy keys and a known zero-distance answer; there +is no external target. It forges authenticated packets as a malicious server +that already knows the transport authentication key. + +At the negotiated terminal modulus q, choose the constant coefficient of c0 +as `floor(q/(2*t)) + offset`, where `offset` is 0 or 1. Choose c1 as a monomial +that places one signed secret coefficient at position zero in `c1*s`. +The tested decryption rounding makes every decoded slot either zero or one. +Both are plausible distances. For the known zero answer, the result check's +acceptance tells whether this chosen phase crossed the rounding boundary. +Two observed responses per coefficient recover the ternary secret in the +test, using fresh legitimate query tickets each time. Terminal-modulus checks, +HMAC, replay rejection and the private result fingerprint all remain enabled. +This is an algebraic demonstration, not a brute-force attack on small N; the +test does not time a full attack at the benchmark parameters. + +Consequently, a uniform exception message does not solve the problem. Neither +does returning only correct plaintexts: **accept/reject, timing, retries, +telemetry or subsequent document fetches can themselves convey a decryption +predicate.** The class name means result-checked, not a public proof of valid +evaluation before decryption. SEAL likewise requires private handling of +decryptor outputs and noise diagnostics and distinguishes ciphertext loading +from authenticity. +[SEAL security guidance](https://github.com/microsoft/SEAL/blob/main/SECURITY.md). + +Closing BFV-01 for an active server with observable client behavior needs a +reviewed protocol boundary. Possibilities to evaluate include verifying a +proof of the prescribed public computation **before decryption**, or a trusted +evaluation environment with an appropriate attestation and side-channel +model. Merely signing the result using a key held by an untrusted evaluator +does not suffice. Neither mechanism is implemented here, and its performance +cannot be inferred from this wrapper benchmark. Removing all harmful feedback +requires analysis of the actual application, not just a local Python API. + +## Audit status after this change + +| Finding | Implemented mitigation | Still open | +| --- | --- | --- | +| BFV-01: chosen decryption | Authenticated framing, exact response shape/modulus, one attempt per request, no partial plaintext | Confirmed accept/reject key-recovery oracle if feedback reaches the server; no CCA-secure or pre-decryption verified protocol | +| BFV-02: forged/replayed results | Private Hamming fingerprint; authenticated query/index/ID binding; in-memory one-use tickets | Experimental verifier requires independent review; no durable replay or arbitrary-circuit proof; BFV-01 remains | +| BFV-03: parameter assurance | Measured parameter profile pinned by trusted local policy | Independent attack estimates, evaluation-key assumptions and circuit-specific failure bounds | +| BFV-04: private implementation/storage | Authenticated encryption for exported keys and summary, bound to complete setup | Variable-time Python/GMP, secret memory erasure, platform isolation and wrapping-key custody | +| BFV-05: resource exhaustion | Auth-before-parse, packet/index caps, query budgets and concurrent-search admission, retaining earlier parser hardening | Transport/aggregate/process quotas, deployment fuzzing and durable service policy | +| BFV-06: unauthenticated bundle | Keyed authentication covers configuration and every public/evaluation key, index and ordered IDs | Trusted key provisioning and ownership authorization in the application; malicious key-generation assurance | +| BFV-07: leakage/authorization | Explicit owner/session boundary and authenticated ordered IDs | All-distances/chosen-query leakage, sizes/counts/timing, follow-up access patterns, tenant authorization and private retrieval | + +The BFV-07 leakage is inherent in the current functionality: a query holder +that can obtain all distances for zero and unit-vector queries can reconstruct +a binary row. A data owner searching their own index may accept that capability; +mutually distrustful query holders need a different access-control/protocol +design. This session wrapper does not implement private top-k selection, PIR, +ORAM, circuit privacy or multi-tenant key isolation. + +## Performance and validation + +The benchmark compares the wrapper with raw evaluation using the **same BFV +keys, index ciphertexts, query ciphertexts and unchanged residue kernels**. +Both server measurements include MessagePack ciphertext decoding/encoding and +native evaluation. The wrapper also includes authentication and request +bookkeeping. Separate client measurements include decryption/slot decoding; +the wrapper adds authentication and the private result check. Setup/import, +arithmetic-context construction and query encryption are outside server +timings. One cold search is followed by five warm searches per variant, +alternating order in a single process without concurrent tests. + +On the Ryzen 7 5800X CPU, Python 3.12.3 and GMP 6.3.0, using 512-bit vectors, +N=8192, t=65537, a 180-bit RNS Q and a 50-bit terminal modulus: + +| Warm median | Raw BFV path | Authenticated/result-checked path | +| --- | ---: | ---: | +| Server, 1,024 vectors | 2.36799 s | 2.37360 s | +| Client decode, 1,024 vectors | 39.55 ms | 41.58 ms | +| Server, 32 vectors | 75.00 ms | 75.40 ms | + +At 1,024 vectors, the measured server difference is **+0.24%**, smaller than +the spread of the warm samples; this is not evidence of a material slowdown. +Client overhead is **2.03 ms**. The isolated private check takes 2.13 ms and +its one-time summary preprocessing takes 25.15 ms. Index encryption, summary +construction, public export and setup authentication together take 8.08 s; +authenticated setup import takes 1.74 s. These setup measurements are reported +separately rather than charged to each search. Cold search times, including +lazy evaluation-key/mask preparation, are 2.657 s raw and 2.674 s wrapped. +All measured ciphertext outputs are exactly identical and every distance +matches an independent plaintext Hamming calculation. + +The same 1,024-vector run gives the following serialized sizes (decimal bytes): + +| Item | Size | +| --- | ---: | +| Public BFV setup keys, unchanged JSON encoding | 85,603,219 | +| Encrypted-index ciphertext array | 23,598,467 | +| Complete authenticated setup, including keys, index, IDs and metadata | 109,205,980 | +| Authenticated query | 368,840 | +| Authenticated response | 102,614 | +| Authenticated query + response | 471,454 | +| Private summary payload, never sent to server | 16,448 | +| AES-GCM private export, including secret key and summary | 36,729 | + +Authentication/session framing adds **113 query bytes + 147 response bytes** +over the bare ciphertext arrays used by both variants in this benchmark. +Using the earlier benchmark's dictionary framing on these exact ciphertexts +gives 368,754 query bytes and 102,496 response bytes: 471,250 combined. The +authenticated combined transfer is therefore only **204 bytes (0.043%) larger +than that earlier framing**. This distinction avoids calling the 260-byte +overhead a difference from an envelope that already contained 56 bytes of +other metadata. Encryption, compaction and public-key serialization are +unchanged; fresh random keys can cause small JSON-size variations. + +Raw samples and source hashes are saved in +`benchmarks/results/native_bfv_verified_1024.json` and +`benchmarks/results/native_bfv_verified_32.json`. The recorded base revision +is `b7841e4` with the new files present in a dirty working tree; the recorded +hashes identify the measured source. These are local measurements, not a +network/load benchmark, and they do not include any future proof or isolated +private-arithmetic implementation needed to close the remaining blockers. + +Reproduce the comparison and regression suite: + +```bash +.venv/bin/python benchmarks/bfv_verified.py --num-vectors 1024 --repeats 6 \ + --json-out /tmp/bfv-verified.json +.venv/bin/python -m pytest tests/x_vec/test_bfv_verified_client.py -q +``` + +Benchmark JSON records raw timings, environment and source/binary hashes, not +keys, ciphertexts or private summary values. The wire-size table separates +the new array framing from the previous benchmark's dictionary framing; the +same ciphertexts can therefore be compared without attributing envelope bytes +to a cryptographic change. No security equivalence with SEAL is asserted. + +Validation after the change: + +- **302 offline tests pass**, including **220 native BFV cases** and the existing + Paillier/SEAL coverage. The service HTTP integrations and unavailable GPU + tests are excluded, as in the original audit. +- The new session test file has **39 cases**: correct results across tile/row/ + response boundaries, public-only evaluation, plausible authenticated + forgeries, cross-query/session replay, tampering before parsing/decryption, + bounded MessagePack, terminal-modulus enforcement, concurrent duplicate + consumption, parameter/resource policy, entropy failures, NumPy/GMP integer + compatibility and encrypted-state binding/restoration. One case intentionally + demonstrates the remaining accept/reject oracle; its passing status means + the attack succeeds under that forbidden integration. +- **135 binding/security cases pass** against a freshly compiled extension + with AddressSanitizer and UndefinedBehaviorSanitizer. Leak detection is + disabled for the Python runtime, as in the existing CI command. No C++ + arithmetic or ABI change is part of this patch. +- Ruff and mypy pass for the SDK and BFV benchmark scripts; the new Python + files pass the scoped formatter check. Existing repository-wide formatting + differences were not reformatted. +- A built wheel imports both new modules and ABI 4 from a separate extracted + package and passes public-only residue evaluation, checked decoding and + encrypted-state restore. No SEAL implementation is imported by this API. +- CI includes the new session tests in both the regular BFV and sanitized + binding runs. The documentation build is checked separately; the original + research page's pre-existing `experiments/bfv/README.md` cross-reference + warning is unrelated to this change. + +Tests demonstrate the covered behavior. They neither establish a negligible +BFV failure probability nor prove absence of side channels or oracle leakage +in a deployed application. The open entries in the status table remain +production blockers wherever their threat model applies. diff --git a/docs/research/native-bfv.md b/docs/research/native-bfv.md index c70bef5..c7fa042 100644 --- a/docs/research/native-bfv.md +++ b/docs/research/native-bfv.md @@ -11,6 +11,11 @@ not undergone an independent cryptographic audit. The open production blockers. The parameter settings below are arithmetic defaults, not a claim of a particular security level. +The optional [authenticated session layer](native-bfv-verified-sessions.md) +adds request/index binding, private Hamming result checks, encrypted private +exports and admission limits. It preserves the server arithmetic and raw APIs; +the follow-up report measures its overhead and records the remaining blockers. + ## Code map All paths are relative to the repository root. diff --git a/src/xtrace_sdk/x_vec/crypto/__init__.py b/src/xtrace_sdk/x_vec/crypto/__init__.py index 3a386fc..98d59b6 100644 --- a/src/xtrace_sdk/x_vec/crypto/__init__.py +++ b/src/xtrace_sdk/x_vec/crypto/__init__.py @@ -12,4 +12,8 @@ ``bfv_client`` provides a native GMP BFV implementation of that interface, with additional packed-index and packed-response operations (experimental). + + ``bfv_security`` and ``bfv_verified_client`` add optional authenticated + sessions, private result checks and encrypted private exports. These remain + experimental and do not make observable decryption feedback safe. """ diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_security.py b/src/xtrace_sdk/x_vec/crypto/bfv_security.py new file mode 100644 index 0000000..d01cca9 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_security.py @@ -0,0 +1,238 @@ +"""Bounded authentication and private storage for the experimental BFV protocol. + +The transport key authenticates a peer, not its computation. The private +verification seed must never be shared with the evaluator. No security level +or constant-time guarantee for BFV is implied by these wrappers. +""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import secrets +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any + +import msgpack +from Crypto.Cipher import AES + +from xtrace_sdk.x_vec.crypto.encryption.bfv import ( + MAX_PUBLIC_KEY_CHARS, + _load_key_json, + _read_ciphertext_wire, +) +from xtrace_sdk.x_vec.utils.xtrace_types import BFVParameters, BFVPublicKey, EncryptedVector + +if TYPE_CHECKING: + from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + + +class BFVProtocolError(ValueError): + """Local protocol rejection. Do not transmit decryption-dependent failures.""" + + +@dataclass(frozen=True) +class BFVExecutionPolicy: + """Locally pinned arithmetic and resource policy, not a security certificate. + + Defaults pin the measured RNS configuration. Override parameters only through + trusted application configuration (e.g. explicit tiny-ring tests), never + from a received setup packet. Limits apply per protocol instance; deployments + still need aggregate quotas and transport limits before buffering packets. + """ + + params: BFVParameters = BFVParameters(rns_modulus=True) + embed_len: int = 512 + response_modulus_bits: int = 50 + max_vectors: int = 65536 + max_setup_bytes: int = 384 * 1024 * 1024 + max_public_key_bytes: int = MAX_PUBLIC_KEY_CHARS + max_index_bytes: int = 256 * 1024 * 1024 + max_query_bytes: int = 1024 * 1024 + max_response_bytes: int = 16 * 1024 * 1024 + max_pending_queries: int = 8 + max_queries: int = 65536 + max_concurrent_searches: int = 1 + + def __post_init__(self) -> None: + from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + + BFVClient(**self.config(), skip_key_gen=True) + for name, value in vars(self).items(): + if name.startswith("max_") and (type(value) is not int or value < 1): + raise ValueError("BFV execution limits must be positive integers") + if self.max_vectors >= 1 << 64 or self.max_queries >= 1 << 64: + raise ValueError("BFV execution counts must fit in 64 bits") + + def config(self) -> dict[str, Any]: + from xtrace_sdk.x_vec.crypto.encryption.bfv import _parameter_dict + + return { + "embed_len": self.embed_len, + **_parameter_dict(self.params), + "response_modulus_bits": self.response_modulus_bits, + "device": "cpu", + } + + def check_client(self, client: BFVClient) -> None: + if json.loads(client.stringify_config()) != self.config(): + raise BFVProtocolError("BFV parameters do not match the locally pinned policy") + + +def _key(key: bytes) -> bytes: + if type(key) is not bytes or len(key) != 32: + raise ValueError("Use an independently generated 32-byte key") + return key + + +_MAGIC = b"XBFVA1" +_HEADER_BYTES = len(_MAGIC) + 1 + 32 + + +def _authenticate(body: bytes, key: bytes, kind: int, limit: int) -> bytes: + prefix = _MAGIC + bytes([kind]) + if len(body) + _HEADER_BYTES > limit: + raise BFVProtocolError("BFV packet exceeds the local size limit") + mac = hmac.new(_key(key), prefix, hashlib.sha256) + mac.update(body) + return prefix + mac.digest() + body + + +def _authenticated_body(packet: bytes, key: bytes, kind: int, limit: int) -> bytes: + if type(packet) is not bytes or not _HEADER_BYTES <= len(packet) <= limit: + raise BFVProtocolError("BFV packet rejected") + prefix = _MAGIC + bytes([kind]) + if packet[: len(prefix)] != prefix: + raise BFVProtocolError("BFV packet rejected") + mac = hmac.new(_key(key), prefix, hashlib.sha256) + mac.update(memoryview(packet)[_HEADER_BYTES:]) + if not hmac.compare_digest(mac.digest(), packet[len(prefix) : _HEADER_BYTES]): + raise BFVProtocolError("BFV packet rejected") + # Parse only after authentication. The peer still needs structural checks. + return packet[_HEADER_BYTES:] + + +def _pack(value: Any) -> bytes: + return msgpack.packb(value, use_bin_type=True) + + +def _unpack(data: bytes, *, limit: int, array_limit: int = 8) -> Any: + if type(data) is not bytes or len(data) > limit: + raise BFVProtocolError("BFV encoded object exceeds its size limit") + try: + return msgpack.unpackb( + data, + raw=False, + max_bin_len=limit, + max_str_len=256, + max_array_len=max(8, array_limit), + max_map_len=0, + max_ext_len=0, + ) + except (ValueError, TypeError, msgpack.UnpackException) as exc: + raise BFVProtocolError("BFV encoded object rejected") from exc + + +def _record(value: Any, length: int) -> list[Any]: + if type(value) is not list or len(value) != length: + raise BFVProtocolError("Invalid BFV record") + return value + + +def _bytes(value: Any, length: int) -> bytes: + if type(value) is not bytes or len(value) != length: + raise BFVProtocolError("Invalid BFV identifier") + return value + + +def _encode_ciphertexts(values: list[EncryptedVector], pk: BFVPublicKey) -> bytes: + return _pack( + [ + [v.to_bytes((v.bit_length() + 7) // 8, "little") for v in _read_ciphertext_wire(ct, pk)] + for ct in values + ] + ) + + +def _decode_ciphertexts( + data: bytes, pk: BFVPublicKey, count: int, limit: int +) -> list[EncryptedVector]: + rows = _unpack(data, limit=limit, array_limit=count) + if type(rows) is not list or len(rows) != count: + raise BFVProtocolError("Unexpected BFV ciphertext count") + result = [] + for row in rows: + _record(row, 8) + if any(type(v) is not bytes for v in row): + raise BFVProtocolError("BFV wire fields must be byte strings") + result.append(_read_ciphertext_wire(row, pk)) + return result + + +_STORAGE_MAGIC = b"XBFVK1" + + +def protect_bfv_state(plaintext: bytes, wrapping_key: bytes, binding: bytes) -> bytes: + """AES-256-GCM private export bound to an authenticated public context. + + Supply a wrapping key from separate trusted storage/KMS. This protects the + stored bytes, not Python/GMP memory or copies made by the caller. + """ + _key(wrapping_key) + _bytes(binding, 32) + if type(plaintext) is not bytes or len(plaintext) > 1024 * 1024: + raise ValueError("BFV private state exceeds the 1 MiB limit") + nonce = secrets.token_bytes(12) + cipher = AES.new(wrapping_key, AES.MODE_GCM, nonce=nonce, mac_len=16) + cipher.update(_STORAGE_MAGIC + binding) + ciphertext, tag = cipher.encrypt_and_digest(plaintext) + return _STORAGE_MAGIC + nonce + tag + ciphertext + + +def unprotect_bfv_state(packet: bytes, wrapping_key: bytes, binding: bytes) -> bytes: + """Authenticate private storage before returning any plaintext or parsing it.""" + _key(wrapping_key) + _bytes(binding, 32) + header = len(_STORAGE_MAGIC) + 28 + if ( + type(packet) is not bytes + or not header <= len(packet) <= header + 1024 * 1024 + or packet[: len(_STORAGE_MAGIC)] != _STORAGE_MAGIC + ): + raise BFVProtocolError("BFV private state rejected") + offset = len(_STORAGE_MAGIC) + cipher = AES.new(wrapping_key, AES.MODE_GCM, nonce=packet[offset : offset + 12], mac_len=16) + cipher.update(_STORAGE_MAGIC + binding) + try: + return cipher.decrypt_and_verify(packet[header:], packet[offset + 12 : header]) + except ValueError: + raise BFVProtocolError("BFV private state rejected") from None + + +def protect_bfv_secret_key(client: BFVClient, wrapping_key: bytes) -> bytes: + """Protect secret/configuration; the binding covers the complete public JSON.""" + public = client.stringify_pk().encode() + private = _pack([client.stringify_config().encode(), client.stringify_sk().encode()]) + return protect_bfv_state(private, wrapping_key, hashlib.sha256(public).digest()) + + +def restore_bfv_client(public_json: str, protected: bytes, wrapping_key: bytes) -> BFVClient: + """Restore against exactly the saved public bundle, including evaluation keys.""" + from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + + if ( + type(public_json) is not str + or len(public_json) > MAX_PUBLIC_KEY_CHARS + or not public_json.isascii() + ): + raise BFVProtocolError("BFV public key exceeds its import limit") + plaintext = unprotect_bfv_state( + protected, wrapping_key, hashlib.sha256(public_json.encode()).digest() + ) + config, secret = _record(_unpack(plaintext, limit=1024 * 1024), 2) + if type(config) is not bytes or type(secret) is not bytes: + raise BFVProtocolError("Invalid BFV private state") + candidate = BFVClient(**_load_key_json(config.decode(), 4096), skip_key_gen=True) + candidate.load_stringified_keys(public_json, secret.decode()) + return candidate diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py new file mode 100644 index 0000000..eee5f00 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py @@ -0,0 +1,465 @@ +"""Authenticated, result-checked BFV sessions for a trusted data-owner client. + +A private Freivalds-style summary checks the binary Hamming result in O(M+D) +client work, without adding server arithmetic. This protocol is experimental: +post-decryption checking is NOT chosen-ciphertext security. Keep outcomes and +timing private, and obtain independent review before deployment. +""" + +from __future__ import annotations + +import hashlib +import hmac +import secrets +import threading +from collections.abc import Sequence +from dataclasses import dataclass + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + BFVProtocolError, + _authenticate, + _authenticated_body, + _bytes, + _decode_ciphertexts, + _encode_ciphertexts, + _key, + _pack, + _record, + _unpack, + protect_bfv_state, + unprotect_bfv_state, +) +from xtrace_sdk.x_vec.crypto.encryption.bfv import _coefficient_modulus, _load_key_json +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVServerBackend + + +_FIELD = (1 << 255) - 19 +_SETUP, _QUERY, _RESPONSE = 1, 2, 3 + + +class _HammingSummary: + """Private linear fingerprint; neither seed nor columns may reach the server. + + For weights r_i, store R=sum(r_i) and S_j=sum(r_i*x_ij). Then + sum(r_i*d_i)=sum(S_j)+R*|q|-2*sum(q_j*S_j) modulo the verification field. + HMAC expands an independent secret seed into pseudorandom field weights. + Rejection sampling avoids modulo bias. This field is separate from BFV's t. + """ + + def __init__(self, vectors: Sequence[Sequence[int]], dimension: int, session: bytes) -> None: + self.seed = secrets.token_bytes(32) + self.session = session + self.count = len(vectors) + self.columns = [0] * dimension + self.total = 0 + for i, row in enumerate(vectors): + weight = self.weight(i) + self.total += weight + for j, bit in enumerate(row): + if bit: + self.columns[j] += weight + self.total %= _FIELD + self.columns = [value % _FIELD for value in self.columns] + + def weight(self, index: int) -> int: + prefix = b"XTRACE-BFV-HAMMING-CHECK-v1\0" + self.session + index.to_bytes(8, "little") + for retry in range(16): + digest = hmac.digest(self.seed, prefix + bytes([retry]), "sha256") + candidate = int.from_bytes(digest, "little") & ((1 << 255) - 1) + if candidate < _FIELD: + return candidate + raise RuntimeError("BFV verification weight generation failed") + + def expected(self, query: Sequence[int]) -> int: + # BFVClient also accepts numpy/GMP Integral scalars. Convert before + # mixing them with 255-bit values so fixed-width scalar coercion cannot + # overflow or silently truncate the private fingerprint arithmetic. + bits = [int(bit) for bit in query] + return ( + sum(self.columns) + + self.total * sum(bits) + - 2 * sum(c * bit for c, bit in zip(self.columns, bits, strict=True)) + ) % _FIELD + + def matches(self, distances: Sequence[int], expected: int) -> bool: + if len(distances) != self.count: + return False + actual = sum(self.weight(i) * value for i, value in enumerate(distances)) % _FIELD + return hmac.compare_digest(actual.to_bytes(32, "little"), expected.to_bytes(32, "little")) + + def export(self) -> list[bytes]: + return [ + self.seed, + self.total.to_bytes(32, "little"), + *(c.to_bytes(32, "little") for c in self.columns), + ] + + @classmethod + def restore( + cls, data: list[bytes], dimension: int, count: int, session: bytes + ) -> _HammingSummary: + _record(data, dimension + 2) + seed = _bytes(data[0], 32) + values = [int.from_bytes(_bytes(value, 32), "little") for value in data[1:]] + if any(value >= _FIELD for value in values): + raise BFVProtocolError("Invalid BFV verification summary") + result = cls.__new__(cls) + result.seed, result.session, result.count = seed, session, count + result.total, result.columns = values[0], values[1:] + return result + + +@dataclass(frozen=True) +class _PendingQuery: + request_digest: bytes + expected: int + + +def _ids(values: Sequence[str] | None, count: int) -> tuple[str, ...]: + result = tuple(str(i) for i in range(count)) if values is None else tuple(values) + if len(result) != count or any( + type(v) is not str or not 1 <= len(v.encode()) <= 256 for v in result + ): + raise BFVProtocolError("Expected one bounded identifier per vector") + if len(set(result)) != count: + raise BFVProtocolError("BFV vector identifiers must be unique") + return result + + +def _setup_fields( + setup: bytes, authentication_key: bytes, policy: BFVExecutionPolicy +) -> tuple[bytes, str, str, int, tuple[str, ...], bytes]: + body = _authenticated_body(setup, authentication_key, _SETUP, policy.max_setup_bytes) + session, config, public, count, ids, index = _record( + _unpack(body, limit=policy.max_setup_bytes, array_limit=policy.max_vectors), 6 + ) + session = _bytes(session, 16) + if type(config) is not bytes or len(config) > 4096: + raise BFVProtocolError("Invalid BFV configuration") + if type(public) is not bytes or len(public) > policy.max_public_key_bytes: + raise BFVProtocolError("BFV public key exceeds its import limit") + if type(count) is not int or not 1 <= count <= policy.max_vectors: + raise BFVProtocolError("BFV vector count exceeds the local policy") + if type(index) is not bytes or len(index) > policy.max_index_bytes: + raise BFVProtocolError("BFV index exceeds its import limit") + if type(ids) is not list: + raise BFVProtocolError("Invalid BFV vector identifiers") + config_text, public_text = config.decode(), public.decode() + if _load_key_json(config_text, 4096) != policy.config(): + raise BFVProtocolError("BFV setup does not match the locally pinned policy") + return session, config_text, public_text, count, _ids(ids, count), index + + +class BFVVerifiedClient: + """One immutable index/session with bounded, one-use query tickets. + + Retains a private O(D) summary rather than the original binary index. + The authentication key is shared with this evaluator over a trusted channel; + it must differ from the wrapping key and BFV private verification material. + This object has no network, telemetry, noise-budget or server callback API. + Return values and exceptions must stay inside the trusted client. + """ + + def __init__( + self, + client: BFVClient, + authentication_key: bytes, + *, + policy: BFVExecutionPolicy | None = None, + ) -> None: + self.policy = policy or BFVExecutionPolicy() + self.policy.check_client(client) + keys = client._keys() + # Snapshot key dictionaries, sharing immutable polynomial tuples. Loading + # different keys into the original BFVClient cannot retarget this session. + self._client = BFVClient(**self.policy.config(), skip_key_gen=True) + self._client.public_key = {**keys["pk"], "galois_keys": dict(keys["pk"]["galois_keys"])} + self._client.keys = { + "pk": self._client.public_key, + "sk": {"s": keys["sk"]["s"], "key_id": keys["sk"]["key_id"]}, + } + self._authentication_key = _key(authentication_key) + self._session = secrets.token_bytes(16) + self._setup_digest: bytes | None = None + self._summary: _HammingSummary | None = None + self._ids: tuple[str, ...] = () + self._pending: dict[bytes, _PendingQuery] = {} + self._started = 0 + self._lock = threading.Lock() + + @property + def vector_ids(self) -> tuple[str, ...]: + return self._ids + + @property + def verification_state_bytes(self) -> int: + """Encoded summary payload only; excludes identifiers and Python objects.""" + return 0 if self._summary is None else (self.policy.embed_len + 2) * 32 + + def prepare_index( + self, vectors: list[list[int]], *, vector_ids: Sequence[str] | None = None + ) -> bytes: + """Encrypt and authenticate setup once; build the private result summary.""" + with self._lock: + if self._setup_digest is not None: + raise BFVProtocolError("Create a new verified client for a new index/session") + if not 1 <= len(vectors) <= self.policy.max_vectors: + raise BFVProtocolError("BFV vector count exceeds the local policy") + ids = _ids(vector_ids, len(vectors)) + for vector in vectors: + self._client._validate_vector(vector) + pk = self._client._pk() + tiles = ( + len(vectors) + self._client.vectors_per_ciphertext - 1 + ) // self._client.vectors_per_ciphertext + minimum_bytes = ( + tiles * 2 * (self.policy.params.poly_modulus_degree * pk["q"].bit_length() // 8) + ) + if minimum_bytes > self.policy.max_index_bytes: + raise BFVProtocolError("BFV index exceeds the local size policy") + summary = _HammingSummary(vectors, self.policy.embed_len, self._session) + index = _encode_ciphertexts(self._client.encrypt_vec_packed(vectors), pk) + public = self._client.stringify_pk().encode() + if ( + len(index) > self.policy.max_index_bytes + or len(public) > self.policy.max_public_key_bytes + ): + raise BFVProtocolError("BFV setup exceeds the local size policy") + body = _pack( + [ + self._session, + self._client.stringify_config().encode(), + public, + len(vectors), + ids, + index, + ] + ) + setup = _authenticate( + body, self._authentication_key, _SETUP, self.policy.max_setup_bytes + ) + self._setup_digest = hashlib.sha256(setup).digest() + self._summary, self._ids = summary, ids + return setup + + def begin_query(self, query: list[int]) -> bytes: + """Encrypt a query and bind its one-use ticket to this exact setup.""" + with self._lock: + if self._setup_digest is None or self._summary is None: + raise BFVProtocolError("Prepare a BFV index first") + if ( + self._started >= self.policy.max_queries + or len(self._pending) >= self.policy.max_pending_queries + ): + raise BFVProtocolError("BFV query budget exhausted") + self._client._validate_vector(query) + ticket = secrets.token_bytes(32) + if ticket in self._pending: + raise BFVProtocolError("BFV query ticket collision") + ciphertext = _encode_ciphertexts( + [self._client.encrypt_vec_one(query)], self._client._pk() + ) + body = _pack([self._setup_digest, ticket, ciphertext]) + packet = _authenticate( + body, self._authentication_key, _QUERY, self.policy.max_query_bytes + ) + self._pending[ticket] = _PendingQuery( + hashlib.sha256(packet).digest(), self._summary.expected(query) + ) + self._started += 1 + return packet + + def finish_query(self, response: bytes) -> list[int]: + """Authenticate, consume the ticket, then decrypt and check all distances. + + Never returns partial or unchecked plaintext. A post-decryption rejection + is still secret-dependent: generic text is not an oracle/side-channel fix. + """ + try: + body = _authenticated_body( + response, self._authentication_key, _RESPONSE, self.policy.max_response_bytes + ) + setup, ticket, request, encoded = _record( + _unpack(body, limit=self.policy.max_response_bytes), 4 + ) + _bytes(setup, 32) + _bytes(ticket, 32) + _bytes(request, 32) + with self._lock: + pending = self._pending.get(ticket) + if ( + setup != self._setup_digest + or pending is None + or request != pending.request_digest + ): + raise BFVProtocolError("BFV response rejected") + # A validly authenticated attempt consumes the ticket even if + # its ciphertext or result later fails. Concurrent duplicates + # can never trigger two private decryptions for the same ticket. + del self._pending[ticket] + if self._summary is None: + raise BFVProtocolError("BFV response rejected") + count = ( + len(self._ids) + self.policy.params.poly_modulus_degree - 1 + ) // self.policy.params.poly_modulus_degree + ciphertexts = _decode_ciphertexts( + encoded, self._client._pk(), count, self.policy.max_response_bytes + ) + target = min( + _coefficient_modulus(self.policy.response_modulus_bits), self._client._pk()["q"] + ) + if any(ct[3] != target for ct in ciphertexts): + raise BFVProtocolError("BFV response rejected") + distances = self._client.decode_hamming_client_packed(ciphertexts, len(self._ids)) + if not self._summary.matches(distances, pending.expected): + raise BFVProtocolError("BFV response rejected") + return distances + except (ValueError, TypeError, OverflowError): + raise BFVProtocolError("BFV response rejected") from None + + def cancel_pending_queries(self) -> None: + """Locally discard tickets; does not reset the total query budget.""" + with self._lock: + self._pending.clear() + + def protect_state(self, wrapping_key: bytes) -> bytes: + """Encrypt keys and summary. Pending requests are deliberately not persisted. + + Protect against rollback in the application's storage. Restoring an old + copy does not make in-memory budgets/replay state durable across restarts. + """ + if hmac.compare_digest(_key(wrapping_key), self._authentication_key): + raise ValueError( + "The private wrapping key must differ from the server authentication key" + ) + with self._lock: + if self._setup_digest is None or self._summary is None: + raise BFVProtocolError("Prepare a BFV index first") + body = _pack( + [self._client.stringify_sk().encode(), self._summary.export(), self._started] + ) + return protect_bfv_state(body, wrapping_key, self._setup_digest) + + @classmethod + def restore( + cls, + setup: bytes, + protected: bytes, + authentication_key: bytes, + wrapping_key: bytes, + *, + policy: BFVExecutionPolicy | None = None, + ) -> BFVVerifiedClient: + """Restore only with the exact authenticated setup and a private wrapping key.""" + policy = policy or BFVExecutionPolicy() + if hmac.compare_digest(_key(wrapping_key), _key(authentication_key)): + raise ValueError("The wrapping and server authentication keys must differ") + # Check setup length/authentication and private-state binding before + # parsing the public JSON or restoring the secret. + _authenticated_body(setup, authentication_key, _SETUP, policy.max_setup_bytes) + digest = hashlib.sha256(setup).digest() + plaintext = unprotect_bfv_state(protected, wrapping_key, digest) + session, config, public, count, ids, _ = _setup_fields(setup, authentication_key, policy) + secret, summary, started = _record( + _unpack(plaintext, limit=1024 * 1024, array_limit=policy.embed_len + 2), 3 + ) + if ( + type(secret) is not bytes + or type(started) is not int + or not 0 <= started <= policy.max_queries + ): + raise BFVProtocolError("Invalid BFV private state") + client = BFVClient(**_load_key_json(config, 4096), skip_key_gen=True) + client.load_stringified_keys( + public, secret.decode(), max_public_key_chars=policy.max_public_key_bytes + ) + result = cls(client, authentication_key, policy=policy) + result._session, result._setup_digest, result._ids, result._started = ( + session, + digest, + ids, + started, + ) + result._summary = _HammingSummary.restore(summary, policy.embed_len, count, session) + return result + + +class BFVVerifiedServer: + """One authenticated public-only index with admission and replay limits. + + Pin policy/authentication_key in trusted service configuration. The setup + hash covers every evaluation key, parameter, ordered ID and encrypted tile. + Replay/admission state is in-memory; use durable service state or a fresh + client-authorized setup after restart. A malicious server can ignore these + checks, which is why the client separately verifies decrypted results. + """ + + def __init__( + self, + setup: bytes, + authentication_key: bytes, + *, + policy: BFVExecutionPolicy | None = None, + backend: BFVServerBackend = "residue", + ) -> None: + self.policy = policy or BFVExecutionPolicy() + self._authentication_key = _key(authentication_key) + _, config, public, count, ids, encoded = _setup_fields( + setup, authentication_key, self.policy + ) + self._client = BFVClient( + **_load_key_json(config, 4096), skip_key_gen=True, server_backend=backend + ) + self._client.load_stringified_keys( + public, max_public_key_chars=self.policy.max_public_key_bytes + ) + self._setup_digest = hashlib.sha256(setup).digest() + self._count, self.vector_ids = count, ids + tiles = ( + count + self._client.vectors_per_ciphertext - 1 + ) // self._client.vectors_per_ciphertext + self._index = _decode_ciphertexts( + encoded, self._client._pk(), tiles, self.policy.max_index_bytes + ) + self._seen: set[bytes] = set() + self._lock = threading.Lock() + self._admission = threading.BoundedSemaphore(self.policy.max_concurrent_searches) + + def search(self, query: bytes) -> bytes: + """Evaluate an authenticated one-use request with the existing BFV kernels.""" + if not self._admission.acquire(blocking=False): + raise BFVProtocolError("BFV evaluator is at its concurrency limit") + try: + body = _authenticated_body( + query, self._authentication_key, _QUERY, self.policy.max_query_bytes + ) + setup, ticket, encoded = _record(_unpack(body, limit=self.policy.max_query_bytes), 3) + _bytes(setup, 32) + _bytes(ticket, 32) + if setup != self._setup_digest: + raise BFVProtocolError("BFV request belongs to another setup") + with self._lock: + if ticket in self._seen or len(self._seen) >= self.policy.max_queries: + raise BFVProtocolError("BFV request replayed or query budget exhausted") + self._seen.add(ticket) + ciphertext = _decode_ciphertexts( + encoded, self._client._pk(), 1, self.policy.max_query_bytes + )[0] + response = self._client.encode_hamming_server_packed( + ciphertext, self._index, self._count + ) + body = _pack( + [ + self._setup_digest, + ticket, + hashlib.sha256(query).digest(), + _encode_ciphertexts(response, self._client._pk()), + ] + ) + return _authenticate( + body, self._authentication_key, _RESPONSE, self.policy.max_response_bytes + ) + finally: + self._admission.release() diff --git a/tests/x_vec/native/run_binding_sanitizers.py b/tests/x_vec/native/run_binding_sanitizers.py index fdcce1c..fea6be8 100644 --- a/tests/x_vec/native/run_binding_sanitizers.py +++ b/tests/x_vec/native/run_binding_sanitizers.py @@ -28,6 +28,7 @@ "tests/x_vec/test_bfv_native.py", "tests/x_vec/test_bfv_residue.py", "tests/x_vec/test_bfv_security.py", + "tests/x_vec/test_bfv_verified_client.py", "-q", ] ) diff --git a/tests/x_vec/test_bfv_verified_client.py b/tests/x_vec/test_bfv_verified_client.py new file mode 100644 index 0000000..216b6fc --- /dev/null +++ b/tests/x_vec/test_bfv_verified_client.py @@ -0,0 +1,461 @@ +"""Authenticated-session regressions and an explicit residual oracle attack. + +The oracle demonstration intentionally succeeds if accept/reject feedback is +exposed. These tests must not be described as proving chosen-ciphertext security. +""" + +import hashlib +import json +import random +import secrets +import threading +from concurrent.futures import ThreadPoolExecutor +from dataclasses import replace + +import pytest +import numpy as np +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + BFVProtocolError, + _authenticate, + _authenticated_body, + _decode_ciphertexts, + _encode_ciphertexts, + _pack, + _unpack, + protect_bfv_secret_key, + restore_bfv_client, +) +from xtrace_sdk.x_vec.crypto.bfv_verified_client import ( + BFVVerifiedClient, + BFVVerifiedServer, + _FIELD, + _HammingSummary, +) +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _coefficient_modulus +from xtrace_sdk.x_vec.utils.xtrace_types import BFVCiphertext + + +@pytest.fixture(scope="module") +def private(): + return BFVClient(3, 16, 97, 120, 15, response_modulus_bits=40, rns_modulus=True) + + +def policy(private, **limits): + # Explicit insecure algebra-test policy. The default wrapper rejects this ring. + return BFVExecutionPolicy( + params=private.params, + embed_len=private.embed_len, + response_modulus_bits=private.response_modulus_bits, + **limits, + ) + + +def session(private, vectors=None, **limits): + auth = secrets.token_bytes(32) + p = policy(private, **limits) + client = BFVVerifiedClient(private, auth, policy=p) + vectors = [[0, 1, 0], [1, 0, 1], [0, 0, 0]] if vectors is None else vectors + setup = client.prepare_index(vectors) + return client, BFVVerifiedServer(setup, auth, policy=p, backend="optimized"), setup, auth + + +def forged_response(client, request, authentication_key, ciphertexts): + setup, ticket, _ = _unpack( + _authenticated_body(request, authentication_key, 2, client.policy.max_query_bytes), + limit=client.policy.max_query_bytes, + ) + body = _pack( + [ + setup, + ticket, + hashlib.sha256(request).digest(), + _encode_ciphertexts(ciphertexts, client._client._pk()), + ] + ) + return _authenticate(body, authentication_key, 3, client.policy.max_response_bytes) + + +def fake_distances(private, distances): + slots = [0] * private.params.poly_modulus_degree + for i, value in enumerate(distances): + tile, lane = divmod(i, private.vectors_per_ciphertext) + row, col = divmod(lane, private.lanes_per_row) + slots[row * (len(slots) // 2) + tile * private.lanes_per_row + col] = value + ct = BFV.encrypt(BFV.batch_encode(slots, private.params), private._pk()) + return [ + BFV.ciphertext_to_ints( + BFV.modulus_switch(ct, private.response_modulus_bits, private._pk()), private._pk() + ) + ] + + +@pytest.mark.parametrize("count", [1, 5, 39]) +@pytest.mark.parametrize("backend", ["optimized", "residue"]) +def test_verified_distances_ids_and_public_only_server(private, count, backend, monkeypatch): + if backend == "residue": + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns") + vectors = ([[0, 1, 0], [1, 0, 1], [0, 0, 0]] * 13)[:count] + auth = secrets.token_bytes(32) + client = BFVVerifiedClient(private, auth, policy=policy(private)) + ids = [f"document-{i}" for i in range(count)] + setup = client.prepare_index(vectors, vector_ids=ids) + server = BFVVerifiedServer(setup, auth, policy=policy(private), backend=backend) + assert client.vector_ids == server.vector_ids == tuple(ids) + assert server._client.keys is None + assert client.verification_state_bytes == 32 * (3 + 2) + request = client.begin_query([0, 1, 0]) + with monkeypatch.context() as guard: + guard.setattr( + BFV, "_phase", lambda *args: pytest.fail("Server accessed private arithmetic") + ) + response = server.search(request) + expected = [sum(a != b for a, b in zip([0, 1, 0], v, strict=True)) for v in vectors] + assert client.finish_query(response) == expected + + +def test_private_summary_matches_independent_hamming_oracle(): + rng = random.Random(718) + vectors = [[rng.randrange(2) for _ in range(17)] for _ in range(31)] + summary = _HammingSummary(vectors, 17, secrets.token_bytes(16)) + assert _FIELD.bit_length() == 255 + for _ in range(30): + query = [rng.randrange(2) for _ in range(17)] + distances = [sum(a != b for a, b in zip(query, row, strict=True)) for row in vectors] + assert summary.matches(distances, summary.expected(query)) + distances[0] += 1 + assert not summary.matches(distances, summary.expected(query)) + assert not summary.matches([], 0) + + +@pytest.mark.parametrize("scalar", [np.int64, mpz, bool]) +def test_binary_integral_scalars_preserve_full_precision(private, scalar): + client, server, _, _ = session(private) + request = client.begin_query([scalar(0), scalar(1), scalar(0)]) + assert client.finish_query(server.search(request)) == [0, 3, 1] + + +@pytest.mark.parametrize("distances", [[0, 0, 0], [1, 1, 1], [3, 0, 1], [0, 3, 2]]) +def test_authenticated_plausible_forgery_rejected(private, distances): + client, _, _, auth = session(private) + request = client.begin_query([0, 1, 0]) # Correct distances are [0,3,1]. + response = forged_response(client, request, auth, fake_distances(private, distances)) + with pytest.raises(BFVProtocolError, match="response rejected"): + client.finish_query(response) + assert not client._pending # Failed authenticated attempts are still consumed. + + +def test_authenticated_wrong_result_cannot_be_resubmitted(private): + client, server, _, auth = session(private) + request = client.begin_query([0, 1, 0]) + honest = server.search(request) + forged = forged_response(client, request, auth, fake_distances(private, [1, 1, 1])) + with pytest.raises(BFVProtocolError): + client.finish_query(forged) + with pytest.raises(BFVProtocolError): + client.finish_query(honest) + + +def test_replay_and_cross_query_rebinding(private): + client, server, _, auth = session(private) + request = client.begin_query([0, 1, 0]) + response = server.search(request) + with pytest.raises(BFVProtocolError, match="replayed"): + server.search(request) + assert client.finish_query(response) == [0, 3, 1] + with pytest.raises(BFVProtocolError): + client.finish_query(response) + request2 = client.begin_query([1, 0, 1]) + fields = _unpack( + _authenticated_body(response, auth, 3, client.policy.max_response_bytes), + limit=client.policy.max_response_bytes, + ) + old = _decode_ciphertexts(fields[3], private._pk(), 1, client.policy.max_response_bytes) + with pytest.raises(BFVProtocolError): + client.finish_query(forged_response(client, request2, auth, old)) + + +def test_same_key_different_index_session_rejected(private): + client, server, _, auth = session(private) + other = BFVVerifiedClient(private, auth, policy=policy(private)) + other.prepare_index([[0, 1, 0]]) + with pytest.raises(BFVProtocolError, match="another setup"): + server.search(other.begin_query([0, 1, 0])) + response = server.search(client.begin_query([0, 1, 0])) + with pytest.raises(BFVProtocolError): + other.finish_query(response) + + +def test_authentication_checked_before_parser_or_decryption(private, monkeypatch): + client, server, setup, auth = session(private) + response = server.search(client.begin_query([0, 1, 0])) + import xtrace_sdk.x_vec.crypto.bfv_security as security + + with monkeypatch.context() as guard: + guard.setattr( + security.msgpack, "unpackb", lambda *a, **kw: pytest.fail("Unauthenticated body parsed") + ) + guard.setattr( + BFV, "decrypt", lambda *a: pytest.fail("Unauthenticated ciphertext decrypted") + ) + with pytest.raises(BFVProtocolError): + client.finish_query(response[:-1] + bytes([response[-1] ^ 1])) + with pytest.raises(BFVProtocolError): + BFVVerifiedServer(setup[:-1] + bytes([setup[-1] ^ 1]), auth, policy=policy(private)) + # An outsider's bad MAC does not consume the legitimate ticket. + assert client.finish_query(response) == [0, 3, 1] + + +def test_setup_auth_covers_evaluation_keys_ids_and_configuration(private): + _, _, setup, auth = session(private) + header = 6 + 1 + 32 + fields = _unpack(setup[header:], limit=len(setup), array_limit=65536) + for position in (1, 2, 4, 5): + changed = fields[:] + changed[position] = ["other"] if position == 4 else changed[position] + b" " + tampered = setup[:header] + _pack(changed) + with pytest.raises(BFVProtocolError): + BFVVerifiedServer(tampered, auth, policy=policy(private)) + + +def test_authenticated_malformed_response_has_no_partial_plaintext(private, monkeypatch): + client, _, _, auth = session(private) + request = client.begin_query([0, 1, 0]) + response = forged_response(client, request, auth, []) + monkeypatch.setattr(BFV, "decrypt", lambda *a: pytest.fail("Wrong count reached decryption")) + with pytest.raises(BFVProtocolError): + client.finish_query(response) + + +def test_response_requires_negotiated_terminal_modulus(private, monkeypatch): + client, _, _, auth = session(private) + request = client.begin_query([0, 1, 0]) + response = forged_response(client, request, auth, [private.encrypt_vec_one([0, 0, 0])]) + monkeypatch.setattr(BFV, "decrypt", lambda *a: pytest.fail("Wrong modulus reached decryption")) + with pytest.raises(BFVProtocolError): + client.finish_query(response) + + +def test_query_budgets_are_bounded_and_cancellation_does_not_reset(private): + client, server, _, _ = session(private, max_pending_queries=1, max_queries=2) + request = client.begin_query([0, 1, 0]) + with pytest.raises(BFVProtocolError, match="budget"): + client.begin_query([0, 1, 0]) + client.cancel_pending_queries() + with pytest.raises(BFVProtocolError): + client.finish_query(server.search(request)) + assert client.finish_query(server.search(client.begin_query([0, 1, 0]))) == [0, 3, 1] + with pytest.raises(BFVProtocolError, match="budget"): + client.begin_query([0, 1, 0]) + + +def test_size_limits_and_pinned_parameters(private, monkeypatch): + with pytest.raises(BFVProtocolError, match="pinned"): + BFVVerifiedClient(private, secrets.token_bytes(32)) + with pytest.raises(ValueError): + policy(private, max_vectors=0) + client = BFVVerifiedClient( + private, secrets.token_bytes(32), policy=policy(private, max_index_bytes=1) + ) + monkeypatch.setattr( + BFVClient, "encrypt_vec_packed", lambda *a: pytest.fail("Oversized index encrypted") + ) + with pytest.raises(BFVProtocolError, match="size policy"): + client.prepare_index([[0, 1, 0]]) + + +def test_packet_limits_precede_authentication_and_parsing(private, monkeypatch): + client, server, setup, auth = session(private) + request = client.begin_query([0, 1, 0]) + response = server.search(request) + import xtrace_sdk.x_vec.crypto.bfv_security as security + + monkeypatch.setattr( + security.hmac, "new", lambda *a, **kw: pytest.fail("Oversized packet hashed") + ) + for packet, kind in ((setup, 1), (request, 2), (response, 3)): + with pytest.raises(BFVProtocolError): + _authenticated_body(packet, auth, kind, len(packet) - 1) + + +@pytest.mark.parametrize("encoded", [b"\xc1", b"\x81\xa1x\xc0", b"\xdd\xff\xff\xff\xff"]) +def test_authenticated_invalid_messagepack_never_decrypts(private, encoded, monkeypatch): + client, _, _, auth = session(private) + monkeypatch.setattr(BFV, "decrypt", lambda *a: pytest.fail("Malformed body decrypted")) + with pytest.raises(BFVProtocolError): + client.finish_query(_authenticate(encoded, auth, 3, client.policy.max_response_bytes)) + + +def test_setup_policy_checked_before_public_key_import(private, monkeypatch): + _, _, setup, auth = session(private) + p = policy(private) + fields = _unpack( + _authenticated_body(setup, auth, 1, p.max_setup_bytes), limit=p.max_setup_bytes + ) + fields[1] = json.dumps({**p.config(), "embed_len": 2}).encode() + changed = _authenticate(_pack(fields), auth, 1, p.max_setup_bytes) + monkeypatch.setattr( + BFV, "deserialize_public_key", lambda *a, **kw: pytest.fail("Unpinned setup imported") + ) + with pytest.raises(BFVProtocolError, match="pinned"): + BFVVerifiedServer(changed, auth, policy=p) + with pytest.raises(BFVProtocolError, match="import limit"): + BFVVerifiedServer(setup, auth, policy=replace(p, max_public_key_bytes=1)) + + +def test_summary_randomness_failure_aborts_preparation(private, monkeypatch): + client = BFVVerifiedClient(private, secrets.token_bytes(32), policy=policy(private)) + monkeypatch.setattr( + secrets, "token_bytes", lambda *a: (_ for _ in ()).throw(OSError("entropy unavailable")) + ) + with pytest.raises(OSError, match="entropy"): + client.prepare_index([[0, 1, 0]]) + assert client._summary is None and client._setup_digest is None + + +def test_summary_rejection_sampling_is_bounded(monkeypatch): + import xtrace_sdk.x_vec.crypto.bfv_verified_client as verified + + monkeypatch.setattr(verified.hmac, "digest", lambda *a: b"\xff" * 32) + with pytest.raises(RuntimeError, match="weight generation failed"): + _HammingSummary([[0, 1, 0]], 3, secrets.token_bytes(16)) + + +def test_original_client_key_replacement_does_not_retarget_session(private): + client, server, _, _ = session(private) + original = client._client._pk() + # The caller can reuse their raw BFVClient; the verified session owns its snapshot. + saved = private.public_key + try: + private.public_key = None + assert client.finish_query(server.search(client.begin_query([0, 1, 0]))) == [0, 3, 1] + assert client._client._pk() is original + finally: + private.public_key = saved + + +@pytest.mark.parametrize("kind", [1, 2, 3]) +def test_packet_types_cannot_be_confused(private, kind): + client, server, setup, auth = session(private) + query = client.begin_query([0, 1, 0]) + response = server.search(query) + packet = (setup, query, response)[kind - 1] + with pytest.raises(BFVProtocolError): + _authenticated_body(packet, auth, kind % 3 + 1, len(packet)) + + +def test_concurrent_duplicate_causes_only_one_private_decryption(private, monkeypatch): + client, server, _, _ = session(private) + response = server.search(client.begin_query([0, 1, 0])) + decrypt, calls, lock = BFV.decrypt, [], threading.Lock() + + def counted(*args): + with lock: + calls.append(1) + return decrypt(*args) + + monkeypatch.setattr(BFV, "decrypt", counted) + + def finish(): + try: + return client.finish_query(response) + except BFVProtocolError: + return None + + with ThreadPoolExecutor(max_workers=4) as pool: + results = list(pool.map(lambda _: finish(), range(4))) + assert results.count([0, 3, 1]) == 1 and len(calls) == 1 + + +def test_server_admission_releases_after_failure(private): + client, server, _, _ = session(private) + assert server._admission.acquire(blocking=False) + with pytest.raises(BFVProtocolError, match="concurrency"): + server.search(b"bad") + server._admission.release() + with pytest.raises(BFVProtocolError): + server.search(b"bad") + assert client.finish_query(server.search(client.begin_query([0, 1, 0]))) == [0, 3, 1] + + +def test_protected_private_state_roundtrip_and_pending_tickets_discarded(private): + client, server, setup, auth = session(private, max_queries=3) + wrap = secrets.token_bytes(32) + old_response = server.search(client.begin_query([0, 1, 0])) + protected = client.protect_state(wrap) + restored = BFVVerifiedClient.restore(setup, protected, auth, wrap, policy=client.policy) + with pytest.raises(BFVProtocolError): + restored.finish_query(old_response) + assert restored._started == 1 and not restored._pending + assert restored.finish_query(server.search(restored.begin_query([0, 1, 0]))) == [0, 3, 1] + assert restored.vector_ids == client.vector_ids + with pytest.raises(ValueError, match="differ"): + client.protect_state(auth) + with pytest.raises(BFVProtocolError): + BFVVerifiedClient.restore( + setup, protected, auth, secrets.token_bytes(32), policy=client.policy + ) + with pytest.raises(BFVProtocolError): + BFVVerifiedClient.restore( + setup, protected[:-1] + bytes([protected[-1] ^ 1]), auth, wrap, policy=client.policy + ) + + +def test_protected_key_binds_entire_public_bundle_before_loading(private, monkeypatch): + key = secrets.token_bytes(32) + protected = protect_bfv_secret_key(private, key) + original = private.stringify_pk() + assert restore_bfv_client(original, protected, key)._keys() == private._keys() + data = json.loads(original) + data["relin_key"][0][0][0] = format( + (mpz(data["relin_key"][0][0][0], 16) + 1) % private._pk()["q"], "x" + ) + changed = json.dumps(data) + monkeypatch.setattr( + BFV, + "deserialize_public_key", + lambda *a, **kw: pytest.fail("Public key parsed before storage authentication"), + ) + for public, blob, secret in [ + (changed, protected, key), + (original, protected, secrets.token_bytes(32)), + (original, protected[:-1], key), + ]: + with pytest.raises(BFVProtocolError): + restore_bfv_client(public, blob, secret) + + +def test_acceptance_feedback_still_allows_key_recovery(private): + """Residual blocker: perfect plaintext-result checks are still a predicate oracle. + + For a known zero answer, c0=floor(q/(2t))+offset, c1=X^k probes + whether a signed secret coefficient crosses the decryption boundary. + All returned slots are 0 or 1, so range checks do not remove this oracle. + Two accepted/rejected responses per coefficient recover the ternary secret. + """ + client, _, _, auth = session(private, vectors=[[0, 0, 0]]) + n, t = private.params.poly_modulus_degree, private.params.plain_modulus + q = _coefficient_modulus(private.response_modulus_bits) + recovered = [] + for index in range(n): + observed = [] + for offset in (0, 1): + c0 = (q // (2 * t) + offset,) + (mpz(0),) * (n - 1) + c1 = [mpz(0)] * n + c1[0 if index == 0 else n - index] = mpz(1) + forged = BFVCiphertext((c0, tuple(c1)), q, private._pk()["key_id"]) + request = client.begin_query([0, 0, 0]) + response = forged_response( + client, request, auth, [BFV.ciphertext_to_ints(forged, private._pk())] + ) + try: + assert client.finish_query(response) == [0] + observed.append(True) + except BFVProtocolError: + observed.append(False) + signed = -1 if observed[1] else 0 if observed[0] else 1 + recovered.append(signed if index == 0 else -signed) + assert tuple(recovered) == private._keys()["sk"]["s"] From 46b637b604e1c2835653140610b7b2396f10b284 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Fri, 11 Sep 2026 09:50:46 -0400 Subject: [PATCH 10/12] Add pre-decryption BFV approval and private arithmetic assurance --- .github/workflows/ci.yml | 26 +- benchmarks/bfv_guarded.py | 237 ++++++++ benchmarks/bfv_parameter_audit.py | 186 +++++++ .../native_bfv_guarded_original_1024.json | 191 +++++++ .../native_bfv_guarded_review_1024.json | 191 +++++++ .../native_bfv_parameter_estimates.json | 521 ++++++++++++++++++ ...native_bfv_review_parameter_estimates.json | 521 ++++++++++++++++++ docs/index.rst | 1 + docs/research/native-bfv-predecryption.md | 466 ++++++++++++++++ docs/research/native-bfv-security.md | 6 + docs/research/native-bfv-verified-sessions.md | 6 + docs/research/native-bfv.md | 6 + src/xtrace_sdk/x_vec/crypto/bfv_assurance.py | 129 +++++ .../x_vec/crypto/bfv_cpu_ext/Makefile | 10 +- .../x_vec/crypto/bfv_cpu_ext/_bfv_private.pyi | 7 + .../crypto/bfv_cpu_ext/private_bindings.cpp | 104 ++++ .../crypto/bfv_cpu_ext/private_decoder.h | 245 ++++++++ .../x_vec/crypto/bfv_guarded_client.py | 259 +++++++++ .../x_vec/crypto/bfv_verified_client.py | 16 +- .../x_vec/crypto/encryption/bfv_private.py | 98 ++++ tests/x_vec/native/run_binding_sanitizers.py | 25 +- tests/x_vec/native/test_bfv_private.cpp | 114 ++++ tests/x_vec/test_bfv_assurance.py | 63 +++ tests/x_vec/test_bfv_guarded_client.py | 264 +++++++++ tests/x_vec/test_bfv_private.py | 188 +++++++ 25 files changed, 3862 insertions(+), 18 deletions(-) create mode 100644 benchmarks/bfv_guarded.py create mode 100644 benchmarks/bfv_parameter_audit.py create mode 100644 benchmarks/results/native_bfv_guarded_original_1024.json create mode 100644 benchmarks/results/native_bfv_guarded_review_1024.json create mode 100644 benchmarks/results/native_bfv_parameter_estimates.json create mode 100644 benchmarks/results/native_bfv_review_parameter_estimates.json create mode 100644 docs/research/native-bfv-predecryption.md create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_assurance.py create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.pyi create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py create mode 100644 src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py create mode 100644 tests/x_vec/native/test_bfv_private.cpp create mode 100644 tests/x_vec/test_bfv_assurance.py create mode 100644 tests/x_vec/test_bfv_guarded_client.py create mode 100644 tests/x_vec/test_bfv_private.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5a7e740..2946df3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -65,23 +65,41 @@ jobs: - name: Build and test native BFV CPU kernels run: | sudo apt-get update - sudo apt-get install -y libgmp-dev + sudo apt-get install -y libgmp-dev valgrind clang make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" - uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 4' - uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py tests/x_vec/test_bfv_security.py tests/x_vec/test_bfv_verified_client.py + uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns, _bfv_private; assert _bfv_rns.ABI_VERSION == 4; assert _bfv_private.ABI_VERSION == 1' + uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py tests/x_vec/test_bfv_security.py tests/x_vec/test_bfv_verified_client.py tests/x_vec/test_bfv_private.py tests/x_vec/test_bfv_guarded_client.py tests/x_vec/test_bfv_assurance.py - name: BFV native arithmetic sanitizers run: | g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext tests/x_vec/native/test_bfv_rns.cpp -lgmpxx -lgmp -o /tmp/test_bfv_rns /tmp/test_bfv_rns + g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext tests/x_vec/native/test_bfv_private.cpp -lgmpxx -lgmp -o /tmp/test_bfv_private + /tmp/test_bfv_private + + - name: BFV private kernel secret-taint regression + run: | + bfv_gcc_install=$(dirname "$(g++ -print-libgcc-file-name)") + for bfv_cxx in g++ clang++; do + bfv_toolchain=() + if [ "$bfv_cxx" = clang++ ]; then + bfv_toolchain=("--gcc-install-dir=$bfv_gcc_install") + fi + "$bfv_cxx" "${bfv_toolchain[@]}" -std=c++17 -O3 -g -fPIC -fstack-protector-strong -D_FORTIFY_SOURCE=3 -DXTRACE_BFV_CTGRIND -I src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext tests/x_vec/native/test_bfv_private.cpp -lgmpxx -lgmp -o /tmp/test_bfv_private_taint + valgrind --tool=memcheck --error-exitcode=1 --track-origins=yes /tmp/test_bfv_private_taint + if valgrind --tool=memcheck --error-exitcode=1 /tmp/test_bfv_private_taint --negative-control; then + exit 1 + fi + done - name: BFV Python binding sanitizers run: | bfv_py_include=$(.venv/bin/python -c 'import sysconfig; print(sysconfig.get_paths()["include"])') g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -fPIC -shared -I"$bfv_py_include" src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp -lgmpxx -lgmp -o /tmp/bfv_sanitized.so + g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -fPIC -shared -I"$bfv_py_include" src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp -lgmpxx -lgmp -o /tmp/bfv_private_sanitized.so bfv_asan_runtime=$(g++ -print-file-name=libasan.so) bfv_cpp_runtime=$(g++ -print-file-name=libstdc++.so) - ASAN_OPTIONS=detect_leaks=0 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 LD_PRELOAD="$bfv_asan_runtime:$bfv_cpp_runtime" .venv/bin/python tests/x_vec/native/run_binding_sanitizers.py /tmp/bfv_sanitized.so + ASAN_OPTIONS=detect_leaks=0 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 LD_PRELOAD="$bfv_asan_runtime:$bfv_cpp_runtime" .venv/bin/python tests/x_vec/native/run_binding_sanitizers.py /tmp/bfv_sanitized.so /tmp/bfv_private_sanitized.so # ── Integration tests (require XTrace API credentials) ────────── # Triggers: diff --git a/benchmarks/bfv_guarded.py b/benchmarks/bfv_guarded.py new file mode 100644 index 0000000..71b772e --- /dev/null +++ b/benchmarks/bfv_guarded.py @@ -0,0 +1,237 @@ +#!/usr/bin/env python3 +"""Measure public recomputation and private decoding on identical BFV ciphertexts. + +Run the original and review profiles separately, without concurrent CPU work. +Only synthetic plaintext is seeded. JSON contains measurements and public +parameters, never keys, ciphertexts, signing seeds or private fingerprints. +""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.metadata +import json +import platform +import resource +import secrets +import statistics +import subprocess +import sys +from dataclasses import replace +from datetime import UTC, datetime +from functools import partial +from pathlib import Path +from typing import Any + +import gmpy2 + +from bfv_client_matrix import REPO_ROOT, make_data, time_call +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy, hamming_noise_bound +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_guarded_client import ( + BFVGuardedClient, + BFVPublicVerifier, + bfv_verifier_public_key, +) +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + _authenticated_body, + _decode_ciphertexts, + _unpack, +) +from xtrace_sdk.x_vec.crypto.bfv_verified_client import BFVVerifiedServer + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--profile", choices=("original", "review"), default="review") + parser.add_argument("--num-vectors", type=int, default=1024) + parser.add_argument("--embed-len", type=int, default=512) + parser.add_argument("--seed", type=int, default=1337) + parser.add_argument("--repeats", type=int, default=4, help="One cold and three warm trials") + parser.add_argument("--json-out", type=Path) + args = parser.parse_args() + if min(args.num_vectors, args.embed_len) < 1 or args.repeats < 2: + parser.error("positive dimensions and at least two repeats are required") + policy = replace( + BFVExecutionPolicy() if args.profile == "original" else bfv_review_policy(), + embed_len=args.embed_len, + ) + if args.num_vectors > policy.max_vectors or args.repeats > policy.max_queries: + parser.error("workload exceeds the local session policy") + vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) + setup_times: dict[str, float] = {} + timings: dict[str, list[float]] = { + name: [] + for name in ( + "server", + "verifier", + "python_raw_client", + "native_raw_client", + "guarded_client", + ) + } + print(f"Preparing {args.profile} profile", file=sys.stderr, flush=True) + setup_times["key_generation_s"], raw = time_call(lambda: BFVClient(**policy.config())) + auth, signing_seed = secrets.token_bytes(32), secrets.token_bytes(32) + setup_times["client_key_snapshot_native_import_s"], client = time_call( + lambda: BFVGuardedClient(raw, auth, bfv_verifier_public_key(signing_seed), policy=policy) + ) + setup_times["encrypt_index_summary_export_auth_s"], setup = time_call( + lambda: client.prepare_index(vectors) + ) + setup_times["server_setup_import_s"], server = time_call( + lambda: BFVVerifiedServer(setup, auth, policy=policy) + ) + setup_times["verifier_setup_import_s"], verifier = time_call( + lambda: BFVPublicVerifier( + setup, auth, signing_seed, expected_setup_digest=client.setup_digest, policy=policy + ) + ) + assert server._client.keys is None and verifier._server._client.keys is None + setup_times["server_arithmetic_context_s"], _ = time_call(server._client._evaluator) + setup_times["verifier_arithmetic_context_s"], _ = time_call(verifier._server._client._evaluator) + native = client._session._private_decoder + assert native is not None + pk = raw._pk() + count = ( + len(vectors) + policy.params.poly_modulus_degree - 1 + ) // policy.params.poly_modulus_degree + query_times, runs = [], [] + request = response = receipt = b"" + + def raw_finish(wire: bytes, use_native: bool) -> list[int]: + # Benchmark-only direct private primitive access: applications must use + # finish_query(response, receipt). Both reference timers include parsing. + cts = _decode_ciphertexts(wire, pk, count, policy.max_response_bytes) + if use_native: + return native.decode_packed(cts, len(vectors)) + return raw.decode_hamming_client_packed(cts, len(vectors)) + + for repeat in range(args.repeats): + elapsed, request = time_call(lambda: client.begin_query(query)) + query_times.append(elapsed) + elapsed, response = time_call(partial(server.search, request)) + timings["server"].append(elapsed) + print(f"server: {elapsed:.6f} s", file=sys.stderr, flush=True) + elapsed, receipt = time_call(partial(verifier.approve, request, response)) + timings["verifier"].append(elapsed) + print(f"verifier: {elapsed:.6f} s", file=sys.stderr, flush=True) + wire = _unpack( + _authenticated_body(response, auth, 3, policy.max_response_bytes), + limit=policy.max_response_bytes, + )[3] + variants = ("python_raw_client", "native_raw_client", "guarded_client") + for name in variants if repeat % 2 == 0 else reversed(variants): + elapsed, distances = time_call( + partial(client.finish_query, response, receipt) + if name == "guarded_client" + else partial(raw_finish, wire, name == "native_raw_client") + ) + assert distances == expected, "Incorrect Hamming distances" + timings[name].append(elapsed) + runs.append({"repeat": repeat, **{name: samples[-1] for name, samples in timings.items()}}) + setup_fields = _unpack( + _authenticated_body(setup, auth, 1, policy.max_setup_bytes), + limit=policy.max_setup_bytes, + array_limit=policy.max_vectors, + ) + crypto = REPO_ROOT / "src/xtrace_sdk/x_vec/crypto" + sources = [ + Path(__file__).resolve(), + REPO_ROOT / "benchmarks/bfv_client_matrix.py", + *crypto.glob("bfv*.py"), + *(crypto / "encryption").glob("bfv*.py"), + *(crypto / "bfv_cpu_ext").glob("*.cpp"), + *(crypto / "bfv_cpu_ext").glob("*.h"), + *(crypto / "bfv_cpu_ext").glob("*.so"), + crypto / "bfv_cpu_ext/Makefile", + REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", + ] + output: dict[str, Any] = { + "environment": { + "utc": datetime.now(UTC).isoformat(), + "python": platform.python_version(), + "platform": platform.platform(), + "cpu": next( + ( + line.split(":", 1)[1].strip() + for line in Path("/proc/cpuinfo").read_text().splitlines() + if line.startswith("model name") + ), + "unknown", + ), + "gmpy2": gmpy2.version(), + "gmp": gmpy2.mp_version(), + "pycryptodome": importlib.metadata.version("pycryptodome"), + "msgpack": importlib.metadata.version("msgpack"), + "compiler": subprocess.run( + ["c++", "--version"], capture_output=True, text=True, check=True + ).stdout.splitlines()[0], + "revision": subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip(), + "working_tree_dirty": bool( + subprocess.run( + ["git", "status", "--porcelain"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip() + ), + }, + "command": sys.argv, + "profile": args.profile, + "config": policy.config(), + "vectors": len(vectors), + "seed": args.seed, + "measurement_notes": "Within each profile all variants use identical keys and evaluated ciphertexts. Server and owner-controlled public-only verifier have independently imported setups and separate residue arithmetic contexts. Server search and verifier approval include authentication, MessagePack and arithmetic; setup import, arithmetic-context creation, query encryption and private decoding are separate. Approval independently recomputes the full result and compares exact bytes before Ed25519 signing. Server then verifier run sequentially; no network or overlap claimed. One cold trial followed by three warm trials by default. Raw Python/native client timers include ciphertext-array parsing and all distance decoding; guarded includes receipt, authentication, replay and private result checks, with alternating client order. No security-level or end-to-end constant-time claim.", + "setup_timings": setup_times, + "runs": runs, + "summary": { + name: { + "first_s": samples[0], + "warm_samples_s": samples[1:], + "warm_median_s": statistics.median(samples[1:]), + } + for name, samples in timings.items() + }, + "query_encrypt_and_bind_samples_s": query_times, + "sizes": { + "public_keys_bytes_per_evaluator": len(setup_fields[2]), + "encrypted_index_bytes_per_evaluator": len(setup_fields[5]), + "authenticated_setup_bytes_per_evaluator": len(setup), + "query_bytes": len(request), + "response_bytes": len(response), + "receipt_bytes": len(receipt), + "client_response_and_receipt_bytes": len(response) + len(receipt), + "client_query_response_receipt_bytes": len(request) + len(response) + len(receipt), + "remote_verifier_additional_query_response_input_bytes": len(request) + len(response), + "private_verification_payload_bytes": client.verification_state_bytes, + "native_locked_key_bytes": 16 * policy.params.poly_modulus_degree, + "native_locked_scratch_bytes_per_active_decode": 24 * policy.params.poly_modulus_degree, + }, + "whole_process_peak_rss_kib_linux": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, + "noise_bound": hamming_noise_bound(policy, policy.max_vectors).as_dict(), + "identical_ciphertexts_within_profile": True, + "all_distances_correct": True, + "source_sha256": { + str(p.relative_to(REPO_ROOT)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in sources + }, + } + encoded = json.dumps(output, indent=2) + print(encoded) + if args.json_out: + args.json_out.parent.mkdir(parents=True, exist_ok=True) + args.json_out.write_text(encoded + "\n") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/bfv_parameter_audit.py b/benchmarks/bfv_parameter_audit.py new file mode 100644 index 0000000..35f1506 --- /dev/null +++ b/benchmarks/bfv_parameter_audit.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""Export the actual BFV profile and run a pinned, external lattice estimator. + +Use the SDK Python for `manifest`, and a Sage Python for `estimate`. +This reports heuristic attack costs, not a certificate for BFV or its protocol. +It never generates or exports cryptographic keys. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import platform +import subprocess +import sys +import time +from datetime import UTC, datetime +from functools import partial +from pathlib import Path +from typing import Any + +REPO_ROOT = Path(__file__).resolve().parents[1] +ESTIMATOR_REVISION = "53da5982597709ba0fdf94ea37a84d822310fd84" + + +def revision(path: Path) -> str: + return subprocess.run( + ["git", "rev-parse", "HEAD"], cwd=path, check=True, capture_output=True, text=True + ).stdout.strip() + + +def manifest(profile: str) -> dict[str, Any]: + sys.path.insert(0, str(REPO_ROOT / "src")) + from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy, hamming_noise_bound + from xtrace_sdk.x_vec.crypto.bfv_security import BFVExecutionPolicy + from xtrace_sdk.x_vec.crypto.encryption.bfv import _parameter_modulus, _rns_coefficient_primes + + policy = BFVExecutionPolicy() if profile == "original" else bfv_review_policy() + client = BFVClient(**policy.config(), skip_key_gen=True) + params = policy.params + n = params.poly_modulus_degree + rotations = {pow(3, s % (n // 2), 2 * n) for s in client._rotation_steps()} - {1} + digits = ( + params.coeff_modulus_bits + params.decomposition_bits - 1 + ) // params.decomposition_bits + sources = [ + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py", + ] + return { + "schema": 1, + "profile": profile, + "config": policy.config(), + "q": str(_parameter_modulus(params)), + "q_primes": _rns_coefficient_primes(n, params.coeff_modulus_bits), + "secret": {"distribution": "uniform", "inclusive_bounds": [-1, 1]}, + "error": { + "distribution": "centered_binomial", + "eta": params.error_eta, + "variance": params.error_eta / 2, + }, + "ephemeral_u": {"distribution": "uniform", "inclusive_bounds": [-1, 1]}, + "evaluation_keys": { + "galois_exponents": sorted(rotations), + "digits_per_key": digits, + "key_switch_pairs": (1 + len(rotations)) * digits, + "note": "Encryptions of gadget-scaled s^2 and automorphic secrets. Ordinary LWE estimates do not justify these circular/key-dependent-message assumptions.", + }, + "sdk_revision": revision(REPO_ROOT), + "worst_supported_hamming_bound": hamming_noise_bound(policy, policy.max_vectors).as_dict(), + "source_sha256": { + p: hashlib.sha256((REPO_ROOT / p).read_bytes()).hexdigest() for p in sources + }, + } + + +def estimate(args: argparse.Namespace) -> dict[str, Any]: + root = args.estimator.resolve() + actual_revision = revision(root) + if actual_revision != ESTIMATOR_REVISION: + raise ValueError( + f"Expected reviewed estimator revision {ESTIMATOR_REVISION}, got {actual_revision}" + ) + if subprocess.run( + ["git", "diff", "--exit-code", "HEAD"], cwd=root, capture_output=True + ).returncode: + raise ValueError("Estimator tracked sources must match the pinned revision") + data = json.loads(args.manifest.read_text()) + if data["schema"] != 1 or data["secret"] != { + "distribution": "uniform", + "inclusive_bounds": [-1, 1], + }: + raise ValueError("Unsupported parameter manifest") + if data["error"]["distribution"] != "centered_binomial": + raise ValueError("Unsupported error distribution") + sys.path.insert(0, str(root)) + from sage.all import ZZ, log, oo + from sage.env import SAGE_VERSION + from estimator import LWE, ND, RC + from estimator.reduction import ADPS16 + + n, q = data["config"]["poly_modulus_degree"], ZZ(data["q"]) + models = { + "MATZOV-classical": RC.MATZOV, + "ADPS16-classical": ADPS16(), + "ADPS16-quantum": ADPS16(mode="quantum"), + } + attacks = { + "primal_usvp": LWE.primal_usvp, + "primal_bdd": LWE.primal_bdd, + "dual": LWE.dual, + "dual_hybrid": LWE.dual_hybrid, + } + selected_models = args.models.split(",") + selected_attacks = args.attacks.split(",") + if any(m not in models for m in selected_models) or any( + a not in attacks for a in selected_attacks + ): + raise ValueError("Unknown cost model or attack") + runs: list[dict[str, Any]] = [] + output = { + "utc": datetime.now(UTC).isoformat(), + "sage": SAGE_VERSION, + "python": platform.python_version(), + "estimator_revision": actual_revision, + "runner_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + "manifest": data, + "command": sys.argv, + "interpretation": "Heuristic generic LWE attack estimates with Gaussian-shape approximation for centered binomial errors. Reports m=N and unbounded independent samples; RLWE structure, evaluation-key assumptions, decryption failures, implementation leakage and protocol attacks are not certified. ADPS16-quantum is a quantum sieving cost sensitivity model, not a complete quantum security analysis. Unlisted attacks are not evaluated.", + "runs": runs, + "complete": False, + } + for sample_label, samples in (("N", n), ("unbounded", oo)): + params = LWE.Parameters( + n=n, q=q, Xs=ND.Uniform(-1, 1), Xe=ND.CenteredBinomial(data["error"]["eta"]), m=samples + ) + for model_name in selected_models: + for attack_name in selected_attacks: + print( + f"Estimating {sample_label} samples / {model_name} / {attack_name}", flush=True + ) + kwargs = {"red_cost_model": models[model_name]} + if attack_name.startswith("primal"): + kwargs["red_shape_model"] = "gsa" + start = time.perf_counter() + cost = attacks[attack_name](params, **kwargs) + record = { + "samples": sample_label, + "model": model_name, + "attack": attack_name, + "seconds": time.perf_counter() - start, + "log2_rop": float(log(cost["rop"], 2)) if cost["rop"] != oo else None, + "cost": {str(k): str(v) for k, v in cost.items()}, + } + runs.append(record) + print(json.dumps(record), flush=True) + args.json_out.write_text(json.dumps(output, indent=2) + "\n") + output["complete"] = True + return output + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="mode", required=True) + make = sub.add_parser("manifest") + make.add_argument("--json-out", type=Path, required=True) + make.add_argument("--profile", choices=("original", "review"), default="original") + run = sub.add_parser("estimate") + run.add_argument("--manifest", type=Path, required=True) + run.add_argument("--estimator", type=Path, required=True) + run.add_argument("--json-out", type=Path, required=True) + run.add_argument("--models", default="MATZOV-classical,ADPS16-classical,ADPS16-quantum") + run.add_argument("--attacks", default="primal_usvp,primal_bdd,dual,dual_hybrid") + args = parser.parse_args() + args.json_out.parent.mkdir(parents=True, exist_ok=True) + output = manifest(args.profile) if args.mode == "manifest" else estimate(args) + args.json_out.write_text(json.dumps(output, indent=2) + "\n") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/results/native_bfv_guarded_original_1024.json b/benchmarks/results/native_bfv_guarded_original_1024.json new file mode 100644 index 0000000..2f1d75b --- /dev/null +++ b/benchmarks/results/native_bfv_guarded_original_1024.json @@ -0,0 +1,191 @@ +{ + "environment": { + "utc": "2026-09-11T13:41:44.618087+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "cpu": "AMD Ryzen 7 5800X 8-Core Processor", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "pycryptodome": "3.23.0", + "msgpack": "1.1.2", + "compiler": "c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0", + "revision": "0cd7b41454533ab27082667cb9cb2f3c8040e823", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_guarded.py", + "--profile", + "original", + "--json-out", + "benchmarks/results/native_bfv_guarded_original_1024.json" + ], + "profile": "original", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "vectors": 1024, + "seed": 1337, + "measurement_notes": "Within each profile all variants use identical keys and evaluated ciphertexts. Server and owner-controlled public-only verifier have independently imported setups and separate residue arithmetic contexts. Server search and verifier approval include authentication, MessagePack and arithmetic; setup import, arithmetic-context creation, query encryption and private decoding are separate. Approval independently recomputes the full result and compares exact bytes before Ed25519 signing. Server then verifier run sequentially; no network or overlap claimed. One cold trial followed by three warm trials by default. Raw Python/native client timers include ciphertext-array parsing and all distance decoding; guarded includes receipt, authentication, replay and private result checks, with alternating client order. No security-level or end-to-end constant-time claim.", + "setup_timings": { + "key_generation_s": 4.318560369079933, + "client_key_snapshot_native_import_s": 0.004531735088676214, + "encrypt_index_summary_export_auth_s": 8.078310951124877, + "server_setup_import_s": 1.7306867081206292, + "verifier_setup_import_s": 1.7689348999410868, + "server_arithmetic_context_s": 0.0012989682145416737, + "verifier_arithmetic_context_s": 0.0009495900012552738 + }, + "runs": [ + { + "repeat": 0, + "server": 2.669166201958433, + "verifier": 2.656381974928081, + "python_raw_client": 0.0393350338563323, + "native_raw_client": 0.012509942054748535, + "guarded_client": 0.016294020926579833 + }, + { + "repeat": 1, + "server": 2.3515148740261793, + "verifier": 2.3537557921372354, + "python_raw_client": 0.039415847044438124, + "native_raw_client": 0.012399370083585382, + "guarded_client": 0.01612957799807191 + }, + { + "repeat": 2, + "server": 2.353946026880294, + "verifier": 2.3622144910041243, + "python_raw_client": 0.03945690696127713, + "native_raw_client": 0.012485342100262642, + "guarded_client": 0.016256339848041534 + }, + { + "repeat": 3, + "server": 2.3585687349550426, + "verifier": 2.3510658380109817, + "python_raw_client": 0.038974247174337506, + "native_raw_client": 0.012455580988898873, + "guarded_client": 0.016008156118914485 + } + ], + "summary": { + "server": { + "first_s": 2.669166201958433, + "warm_samples_s": [ + 2.3515148740261793, + 2.353946026880294, + 2.3585687349550426 + ], + "warm_median_s": 2.353946026880294 + }, + "verifier": { + "first_s": 2.656381974928081, + "warm_samples_s": [ + 2.3537557921372354, + 2.3622144910041243, + 2.3510658380109817 + ], + "warm_median_s": 2.3537557921372354 + }, + "python_raw_client": { + "first_s": 0.0393350338563323, + "warm_samples_s": [ + 0.039415847044438124, + 0.03945690696127713, + 0.038974247174337506 + ], + "warm_median_s": 0.039415847044438124 + }, + "native_raw_client": { + "first_s": 0.012509942054748535, + "warm_samples_s": [ + 0.012399370083585382, + 0.012485342100262642, + 0.012455580988898873 + ], + "warm_median_s": 0.012455580988898873 + }, + "guarded_client": { + "first_s": 0.016294020926579833, + "warm_samples_s": [ + 0.01612957799807191, + 0.016256339848041534, + 0.016008156118914485 + ], + "warm_median_s": 0.01612957799807191 + } + }, + "query_encrypt_and_bind_samples_s": [ + 0.09995461604557931, + 0.10021196096204221, + 0.10079239308834076, + 0.10059099900536239 + ], + "sizes": { + "public_keys_bytes_per_evaluator": 85602438, + "encrypted_index_bytes_per_evaluator": 23598467, + "authenticated_setup_bytes_per_evaluator": 109205199, + "query_bytes": 368840, + "response_bytes": 102614, + "receipt_bytes": 198, + "client_response_and_receipt_bytes": 102812, + "client_query_response_receipt_bytes": 471652, + "remote_verifier_additional_query_response_input_bytes": 471454, + "private_verification_payload_bytes": 16448, + "native_locked_key_bytes": 131072, + "native_locked_scratch_bytes_per_active_decode": 196608 + }, + "whole_process_peak_rss_kib_linux": 1094104, + "noise_bound": { + "encrypted": 344085, + "difference": 689382, + "squared": 6077954481171225104, + "key_switch": 1108307719766016, + "tile": 1670999460573335352275202771320, + "merged": 855551723813548266710148620588680, + "compacted": 69633, + "original_rounding_error_numerator": 56070293323568512755383010147599750792, + "terminal_rounding_error_numerator": 8856539137, + "original_modulus": 1532495540865518635130821056977027158796330141975560193, + "terminal_modulus": 1125899906842597, + "sufficient": true + }, + "identical_ciphertexts_within_profile": true, + "all_distances_correct": true, + "source_sha256": { + "benchmarks/bfv_guarded.py": "1c82dbe809674bc70d6946b0f4a7827c0d8d2c8ce715dc1f2d25356a7a73aa79", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py": "ba64de6db3f76031ba9c35627a0a3790244c816937a97a3b65c98b9c699bb33d", + "src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py": "2e684a8a7f2364a1134aa0271ee34fc6e2562367f26ae453f61a03fa80e31db3", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "6a384b16c347d84f90e40ccc9903b1ae916ae277e6171bd668181366d56e88a0", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py": "b1291e6d017be84638b054fb84e651734a44b2d54d601eb50dae2e9d2016bdfa", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp": "7b48d92e526147d97d9401e868c8d3e9dbf0b8ba567a67e05047e01f2a371fc0", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h": "66aceec7cc8452dabdde0061d347195e49a9500fba7bcbbe805c967b7171d767", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.cpython-312-x86_64-linux-gnu.so": "c55be18d74e56a1d6bb82c9fa31f55cb9ecf568aecf191a26278a660502b6c72", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "ee3f7371f92ba9fa3ebb8037906fc23c5f7832f96908d5879f1a4326847ffb53", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962" + } +} diff --git a/benchmarks/results/native_bfv_guarded_review_1024.json b/benchmarks/results/native_bfv_guarded_review_1024.json new file mode 100644 index 0000000..6d45f94 --- /dev/null +++ b/benchmarks/results/native_bfv_guarded_review_1024.json @@ -0,0 +1,191 @@ +{ + "environment": { + "utc": "2026-09-11T13:43:35.312233+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "cpu": "AMD Ryzen 7 5800X 8-Core Processor", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "pycryptodome": "3.23.0", + "msgpack": "1.1.2", + "compiler": "c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0", + "revision": "0cd7b41454533ab27082667cb9cb2f3c8040e823", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_guarded.py", + "--profile", + "review", + "--json-out", + "benchmarks/results/native_bfv_guarded_review_1024.json" + ], + "profile": "review", + "config": { + "embed_len": 512, + "poly_modulus_degree": 16384, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "vectors": 1024, + "seed": 1337, + "measurement_notes": "Within each profile all variants use identical keys and evaluated ciphertexts. Server and owner-controlled public-only verifier have independently imported setups and separate residue arithmetic contexts. Server search and verifier approval include authentication, MessagePack and arithmetic; setup import, arithmetic-context creation, query encryption and private decoding are separate. Approval independently recomputes the full result and compares exact bytes before Ed25519 signing. Server then verifier run sequentially; no network or overlap claimed. One cold trial followed by three warm trials by default. Raw Python/native client timers include ciphertext-array parsing and all distance decoding; guarded includes receipt, authentication, replay and private result checks, with alternating client order. No security-level or end-to-end constant-time claim.", + "setup_timings": { + "key_generation_s": 8.871012358926237, + "client_key_snapshot_native_import_s": 0.0067274910397827625, + "encrypt_index_summary_export_auth_s": 9.416916926158592, + "server_setup_import_s": 3.3731832071207464, + "verifier_setup_import_s": 3.446248749969527, + "server_arithmetic_context_s": 0.0023073479533195496, + "verifier_arithmetic_context_s": 0.001975191989913583 + }, + "runs": [ + { + "repeat": 0, + "server": 3.071234895149246, + "verifier": 3.0581927029415965, + "python_raw_client": 0.07951001706533134, + "native_raw_client": 0.024246918968856335, + "guarded_client": 0.02819207194261253 + }, + { + "repeat": 1, + "server": 2.4405403789132833, + "verifier": 2.437672710046172, + "python_raw_client": 0.0788352529052645, + "native_raw_client": 0.024385120952501893, + "guarded_client": 0.028834844008088112 + }, + { + "repeat": 2, + "server": 2.4386640209704638, + "verifier": 2.435292261186987, + "python_raw_client": 0.08160242205485702, + "native_raw_client": 0.025453513953834772, + "guarded_client": 0.028343144105747342 + }, + { + "repeat": 3, + "server": 2.4315330830868334, + "verifier": 2.432053772965446, + "python_raw_client": 0.080289953853935, + "native_raw_client": 0.024648976977914572, + "guarded_client": 0.028425215976312757 + } + ], + "summary": { + "server": { + "first_s": 3.071234895149246, + "warm_samples_s": [ + 2.4405403789132833, + 2.4386640209704638, + 2.4315330830868334 + ], + "warm_median_s": 2.4386640209704638 + }, + "verifier": { + "first_s": 3.0581927029415965, + "warm_samples_s": [ + 2.437672710046172, + 2.435292261186987, + 2.432053772965446 + ], + "warm_median_s": 2.435292261186987 + }, + "python_raw_client": { + "first_s": 0.07951001706533134, + "warm_samples_s": [ + 0.0788352529052645, + 0.08160242205485702, + 0.080289953853935 + ], + "warm_median_s": 0.080289953853935 + }, + "native_raw_client": { + "first_s": 0.024246918968856335, + "warm_samples_s": [ + 0.024385120952501893, + 0.025453513953834772, + 0.024648976977914572 + ], + "warm_median_s": 0.024648976977914572 + }, + "guarded_client": { + "first_s": 0.02819207194261253, + "warm_samples_s": [ + 0.028834844008088112, + 0.028343144105747342, + 0.028425215976312757 + ], + "warm_median_s": 0.028425215976312757 + } + }, + "query_encrypt_and_bind_samples_s": [ + 0.20502086984924972, + 0.2047637349460274, + 0.20417200191877782, + 0.20610628393478692 + ], + "sizes": { + "public_keys_bytes_per_evaluator": 171205156, + "encrypted_index_bytes_per_evaluator": 23595715, + "authenticated_setup_bytes_per_evaluator": 194805166, + "query_bytes": 737480, + "response_bytes": 205018, + "receipt_bytes": 198, + "client_response_and_receipt_bytes": 205216, + "client_query_response_receipt_bytes": 942696, + "remote_verifier_additional_query_response_input_bytes": 942498, + "private_verification_payload_bytes": 16448, + "native_locked_key_bytes": 262144, + "native_locked_scratch_bytes_per_active_decode": 393216 + }, + "whole_process_peak_rss_kib_linux": 1985372, + "noise_bound": { + "encrypted": 688149, + "difference": 1427785, + "squared": 52059769849576678535, + "key_switch": 2216615439532032, + "tile": 28621377361628378284959515398718, + "merged": 14654145209153730814589761537631682, + "compacted": 73729, + "original_rounding_error_numerator": 960388714572308056395769201895141795266, + "terminal_rounding_error_numerator": 9124978689, + "original_modulus": 1532495540865191964058577030327937992314109603578642433, + "terminal_modulus": 1125899906842597, + "sufficient": true + }, + "identical_ciphertexts_within_profile": true, + "all_distances_correct": true, + "source_sha256": { + "benchmarks/bfv_guarded.py": "1c82dbe809674bc70d6946b0f4a7827c0d8d2c8ce715dc1f2d25356a7a73aa79", + "benchmarks/bfv_client_matrix.py": "4962c7cf837a67becb240e48c7af7068def6276f6bf9647d518de3735be58a89", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py": "ba64de6db3f76031ba9c35627a0a3790244c816937a97a3b65c98b9c699bb33d", + "src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py": "2e684a8a7f2364a1134aa0271ee34fc6e2562367f26ae453f61a03fa80e31db3", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "6a384b16c347d84f90e40ccc9903b1ae916ae277e6171bd668181366d56e88a0", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py": "b1291e6d017be84638b054fb84e651734a44b2d54d601eb50dae2e9d2016bdfa", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp": "7b48d92e526147d97d9401e868c8d3e9dbf0b8ba567a67e05047e01f2a371fc0", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h": "66aceec7cc8452dabdde0061d347195e49a9500fba7bcbbe805c967b7171d767", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.cpython-312-x86_64-linux-gnu.so": "c55be18d74e56a1d6bb82c9fa31f55cb9ecf568aecf191a26278a660502b6c72", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "ee3f7371f92ba9fa3ebb8037906fc23c5f7832f96908d5879f1a4326847ffb53", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962" + } +} diff --git a/benchmarks/results/native_bfv_parameter_estimates.json b/benchmarks/results/native_bfv_parameter_estimates.json new file mode 100644 index 0000000..1c7ccd4 --- /dev/null +++ b/benchmarks/results/native_bfv_parameter_estimates.json @@ -0,0 +1,521 @@ +{ + "utc": "2026-09-11T13:06:59.442710+00:00", + "sage": "10.9", + "python": "3.14.6", + "estimator_revision": "53da5982597709ba0fdf94ea37a84d822310fd84", + "runner_sha256": "8161c495e2b6932a5321b19dfa4da56f25b09934ae6df0b98a36d53e64425b7b", + "manifest": { + "schema": 1, + "profile": "original", + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "q": "1532495540865518635130821056977027158796330141975560193", + "q_primes": [ + 1152921504606830593, + 1152921504606748673, + 1152921504606683137 + ], + "secret": { + "distribution": "uniform", + "inclusive_bounds": [ + -1, + 1 + ] + }, + "error": { + "distribution": "centered_binomial", + "eta": 21, + "variance": 10.5 + }, + "ephemeral_u": { + "distribution": "uniform", + "inclusive_bounds": [ + -1, + 1 + ] + }, + "evaluation_keys": { + "galois_exponents": [ + 385, + 769, + 1537, + 2657, + 3073, + 4097, + 5953, + 6145, + 6561, + 8193, + 10241, + 12289, + 13313, + 14529, + 14849, + 15617, + 16001 + ], + "digits_per_key": 6, + "key_switch_pairs": 108, + "note": "Encryptions of gadget-scaled s^2 and automorphic secrets. Ordinary LWE estimates do not justify these circular/key-dependent-message assumptions." + }, + "sdk_revision": "0cd7b41454533ab27082667cb9cb2f3c8040e823", + "worst_supported_hamming_bound": { + "encrypted": 344085, + "difference": 689382, + "squared": 6077954481171225104, + "key_switch": 1108307719766016, + "tile": 1670999460573335352275202771320, + "merged": 855551723813548266710148620588680, + "compacted": 69633, + "original_rounding_error_numerator": 56070293323568512755383010147599750792, + "terminal_rounding_error_numerator": 8856539137, + "original_modulus": 1532495540865518635130821056977027158796330141975560193, + "terminal_modulus": 1125899906842597, + "sufficient": true + }, + "source_sha256": { + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py": "ba64de6db3f76031ba9c35627a0a3790244c816937a97a3b65c98b9c699bb33d", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962" + } + }, + "command": [ + "benchmarks/bfv_parameter_audit.py", + "estimate", + "--manifest", + "/tmp/xtrace-bfv-parameter-manifest.json", + "--estimator", + "/tmp/xtrace-bfv-lattice-estimator", + "--json-out", + "benchmarks/results/native_bfv_parameter_estimates.json" + ], + "interpretation": "Heuristic generic LWE attack estimates with Gaussian-shape approximation for centered binomial errors. Reports m=N and unbounded independent samples; RLWE structure, evaluation-key assumptions, decryption failures, implementation leakage and protocol attacks are not certified. ADPS16-quantum is a quantum sieving cost sensitivity model, not a complete quantum security analysis. Unlisted attacks are not evaluated.", + "runs": [ + { + "samples": "N", + "model": "MATZOV-classical", + "attack": "primal_usvp", + "seconds": 0.052037457935512066, + "log2_rop": 154.8562154453958, + "cost": { + "rop": "4.13395613507757e46", + "red": "4.13395613507757e46", + "delta": "1.00379266173102", + "beta": "429", + "d": "15759", + "tag": "usvp", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)" + } + }, + { + "samples": "N", + "model": "MATZOV-classical", + "attack": "primal_bdd", + "seconds": 2.4931123971473426, + "log2_rop": 154.3993899051486, + "cost": { + "rop": "3.01194941468505e46", + "red": "2.88343174765430e46", + "svp": "1.28517667030745e45", + "beta": "427", + "eta": "460", + "d": "16145", + "tag": "bdd", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)" + } + }, + { + "samples": "N", + "model": "MATZOV-classical", + "attack": "dual", + "seconds": 0.03983351797796786, + "log2_rop": 156.18958860022911, + "cost": { + "rop": "1.04172042412270e47", + "mem": "3729007849560365844932083519906", + "m": "8162", + "beta": "430", + "d": "16354", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "N", + "model": "MATZOV-classical", + "attack": "dual_hybrid", + "seconds": 4.287429159972817, + "log2_rop": 155.04029557642883, + "cost": { + "rop": "4.69655600047778e46", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)", + "red": "4.69655589846026e46", + "guess": "1.02017520797435e39", + "beta": "426", + "p": "3", + "zeta": "0", + "t": "70", + "beta_": "443", + "N": "3.12173363136213e27", + "m": "8192" + } + }, + { + "samples": "N", + "model": "ADPS16-classical", + "attack": "primal_usvp", + "seconds": 0.03931264718994498, + "log2_rop": 125.26799999999999, + "cost": { + "rop": "5.12183397729253e37", + "red": "5.12183397729253e37", + "delta": "1.00379266173102", + "beta": "429", + "d": "15759", + "tag": "usvp", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)" + } + }, + { + "samples": "N", + "model": "ADPS16-classical", + "attack": "primal_bdd", + "seconds": 2.626561006065458, + "log2_rop": 125.28842587069742, + "cost": { + "rop": "5.19486536675701e37", + "red": "5.12183397729253e37", + "svp": "7.30313894644757e35", + "beta": "429", + "eta": "407", + "d": "16231", + "tag": "bdd", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)" + } + }, + { + "samples": "N", + "model": "ADPS16-classical", + "attack": "dual", + "seconds": 0.028203140944242477, + "log2_rop": 125.5600000000002, + "cost": { + "rop": "6.27084773423208e37", + "mem": "311087831586185034754193559976", + "m": "8162", + "beta": "430", + "d": "16354", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "N", + "model": "ADPS16-classical", + "attack": "dual_hybrid", + "seconds": 2.994258353020996, + "log2_rop": 124.69359450053989, + "cost": { + "rop": "3.3836789e37*e^(-5) + 3.41683478187330e37", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)", + "red": "3.41683478187330e37", + "guess": "3.3836789e37*e^(-5)", + "beta": "427", + "p": "3", + "zeta": "10", + "t": "40", + "beta_": "427", + "N": "1.42331887192754e26", + "m": "8192" + } + }, + { + "samples": "N", + "model": "ADPS16-quantum", + "attack": "primal_usvp", + "seconds": 0.039275356102734804, + "log2_rop": 113.685, + "cost": { + "rop": "1.66953318958631e34", + "red": "1.66953318958631e34", + "delta": "1.00379266173102", + "beta": "429", + "d": "15759", + "tag": "usvp", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)" + } + }, + { + "samples": "N", + "model": "ADPS16-quantum", + "attack": "primal_bdd", + "seconds": 2.6875910649541765, + "log2_rop": 113.7151575146729, + "cost": { + "rop": "1.70479975216603e34", + "red": "1.66953318958631e34", + "svp": "3.52665625797147e32", + "beta": "429", + "eta": "407", + "d": "16231", + "tag": "bdd", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)" + } + }, + { + "samples": "N", + "model": "ADPS16-quantum", + "attack": "dual", + "seconds": 0.023456628900021315, + "log2_rop": 113.9500000006802, + "cost": { + "rop": "2.00617126743433e34", + "mem": "311087831586185034754193559976", + "m": "8162", + "beta": "430", + "d": "16354", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "N", + "model": "ADPS16-quantum", + "attack": "dual_hybrid", + "seconds": 1.8661667311098427, + "log2_rop": 113.42000129809128, + "cost": { + "rop": "1.38938465805634e34", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=8192, tag=None)", + "red": "1.38938340793262e34", + "guess": "1.25012371911589e28", + "beta": "428", + "p": "3", + "zeta": "0", + "t": "40", + "beta_": "428", + "N": "1.03040264691278e25", + "m": "8192" + } + }, + { + "samples": "unbounded", + "model": "MATZOV-classical", + "attack": "primal_usvp", + "seconds": 0.03961539291776717, + "log2_rop": 154.8562154453958, + "cost": { + "rop": "4.13395613507757e46", + "red": "4.13395613507757e46", + "delta": "1.00379266173102", + "beta": "429", + "d": "15759", + "tag": "usvp", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "MATZOV-classical", + "attack": "primal_bdd", + "seconds": 2.544985809130594, + "log2_rop": 154.3993899051486, + "cost": { + "rop": "3.01194941468505e46", + "red": "2.88343174765430e46", + "svp": "1.28517667030745e45", + "beta": "427", + "eta": "460", + "d": "16145", + "tag": "bdd", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "MATZOV-classical", + "attack": "dual", + "seconds": 0.037890825886279345, + "log2_rop": 156.18958860022911, + "cost": { + "rop": "1.04172042412270e47", + "mem": "3729007849560365844932083519906", + "m": "8162", + "beta": "430", + "d": "16354", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "unbounded", + "model": "MATZOV-classical", + "attack": "dual_hybrid", + "seconds": 4.301755240187049, + "log2_rop": 155.04029557642883, + "cost": { + "rop": "4.69655600047778e46", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)", + "red": "4.69655589846026e46", + "guess": "1.02017520797435e39", + "beta": "426", + "p": "3", + "zeta": "0", + "t": "70", + "beta_": "443", + "N": "3.12173363136213e27", + "m": "8192" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-classical", + "attack": "primal_usvp", + "seconds": 0.03921360499225557, + "log2_rop": 125.26799999999999, + "cost": { + "rop": "5.12183397729253e37", + "red": "5.12183397729253e37", + "delta": "1.00379266173102", + "beta": "429", + "d": "15759", + "tag": "usvp", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-classical", + "attack": "primal_bdd", + "seconds": 2.647876218194142, + "log2_rop": 125.28842587069742, + "cost": { + "rop": "5.19486536675701e37", + "red": "5.12183397729253e37", + "svp": "7.30313894644757e35", + "beta": "429", + "eta": "407", + "d": "16231", + "tag": "bdd", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-classical", + "attack": "dual", + "seconds": 0.026285489089787006, + "log2_rop": 125.5600000000002, + "cost": { + "rop": "6.27084773423208e37", + "mem": "311087831586185034754193559976", + "m": "8162", + "beta": "430", + "d": "16354", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-classical", + "attack": "dual_hybrid", + "seconds": 2.982937719207257, + "log2_rop": 124.69359450053989, + "cost": { + "rop": "3.3836789e37*e^(-5) + 3.41683478187330e37", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)", + "red": "3.41683478187330e37", + "guess": "3.3836789e37*e^(-5)", + "beta": "427", + "p": "3", + "zeta": "10", + "t": "40", + "beta_": "427", + "N": "1.42331887192754e26", + "m": "8192" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-quantum", + "attack": "primal_usvp", + "seconds": 0.03953855112195015, + "log2_rop": 113.685, + "cost": { + "rop": "1.66953318958631e34", + "red": "1.66953318958631e34", + "delta": "1.00379266173102", + "beta": "429", + "d": "15759", + "tag": "usvp", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-quantum", + "attack": "primal_bdd", + "seconds": 2.6987395398318768, + "log2_rop": 113.7151575146729, + "cost": { + "rop": "1.70479975216603e34", + "red": "1.66953318958631e34", + "svp": "3.52665625797147e32", + "beta": "429", + "eta": "407", + "d": "16231", + "tag": "bdd", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-quantum", + "attack": "dual", + "seconds": 0.02442668005824089, + "log2_rop": 113.9500000006802, + "cost": { + "rop": "2.00617126743433e34", + "mem": "311087831586185034754193559976", + "m": "8162", + "beta": "430", + "d": "16354", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-quantum", + "attack": "dual_hybrid", + "seconds": 1.8888231310993433, + "log2_rop": 113.42000129809128, + "cost": { + "rop": "1.38938465805634e34", + "problem": "LWEParameters(n=8192, q=1532495540865518635130821056977027158796330141975560193, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)", + "red": "1.38938340793262e34", + "guess": "1.25012371911589e28", + "beta": "428", + "p": "3", + "zeta": "0", + "t": "40", + "beta_": "428", + "N": "1.03040264691278e25", + "m": "8192" + } + } + ], + "complete": true +} diff --git a/benchmarks/results/native_bfv_review_parameter_estimates.json b/benchmarks/results/native_bfv_review_parameter_estimates.json new file mode 100644 index 0000000..e64d2d5 --- /dev/null +++ b/benchmarks/results/native_bfv_review_parameter_estimates.json @@ -0,0 +1,521 @@ +{ + "utc": "2026-09-11T13:05:22.031193+00:00", + "sage": "10.9", + "python": "3.14.6", + "estimator_revision": "53da5982597709ba0fdf94ea37a84d822310fd84", + "runner_sha256": "8161c495e2b6932a5321b19dfa4da56f25b09934ae6df0b98a36d53e64425b7b", + "manifest": { + "schema": 1, + "profile": "review", + "config": { + "embed_len": 512, + "poly_modulus_degree": 16384, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "q": "1532495540865191964058577030327937992314109603578642433", + "q_primes": [ + 1152921504606748673, + 1152921504606683137, + 1152921504606584833 + ], + "secret": { + "distribution": "uniform", + "inclusive_bounds": [ + -1, + 1 + ] + }, + "error": { + "distribution": "centered_binomial", + "eta": 21, + "variance": 10.5 + }, + "ephemeral_u": { + "distribution": "uniform", + "inclusive_bounds": [ + -1, + 1 + ] + }, + "evaluation_keys": { + "galois_exponents": [ + 385, + 1537, + 3073, + 6145, + 8193, + 12289, + 14529, + 15617, + 16001, + 16385, + 17153, + 20481, + 22337, + 24577, + 26625, + 29697, + 31233 + ], + "digits_per_key": 6, + "key_switch_pairs": 108, + "note": "Encryptions of gadget-scaled s^2 and automorphic secrets. Ordinary LWE estimates do not justify these circular/key-dependent-message assumptions." + }, + "sdk_revision": "0cd7b41454533ab27082667cb9cb2f3c8040e823", + "worst_supported_hamming_bound": { + "encrypted": 688149, + "difference": 1427785, + "squared": 52059769849576678535, + "key_switch": 2216615439532032, + "tile": 28621377361628378284959515398718, + "merged": 14654145209153730814589761537631682, + "compacted": 73729, + "original_rounding_error_numerator": 960388714572308056395769201895141795266, + "terminal_rounding_error_numerator": 9124978689, + "original_modulus": 1532495540865191964058577030327937992314109603578642433, + "terminal_modulus": 1125899906842597, + "sufficient": true + }, + "source_sha256": { + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py": "ba64de6db3f76031ba9c35627a0a3790244c816937a97a3b65c98b9c699bb33d", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962" + } + }, + "command": [ + "benchmarks/bfv_parameter_audit.py", + "estimate", + "--manifest", + "/tmp/xtrace-bfv-review-manifest.json", + "--estimator", + "/tmp/xtrace-bfv-lattice-estimator", + "--json-out", + "benchmarks/results/native_bfv_review_parameter_estimates.json" + ], + "interpretation": "Heuristic generic LWE attack estimates with Gaussian-shape approximation for centered binomial errors. Reports m=N and unbounded independent samples; RLWE structure, evaluation-key assumptions, decryption failures, implementation leakage and protocol attacks are not certified. ADPS16-quantum is a quantum sieving cost sensitivity model, not a complete quantum security analysis. Unlisted attacks are not evaluated.", + "runs": [ + { + "samples": "N", + "model": "MATZOV-classical", + "attack": "primal_usvp", + "seconds": 0.05196200706996024, + "log2_rop": 341.1723757172288, + "cost": { + "rop": "5.04799345125361e102", + "red": "5.04799345125361e102", + "delta": "1.00191078002672", + "beta": "1092", + "d": "32083", + "tag": "usvp", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)" + } + }, + { + "samples": "N", + "model": "MATZOV-classical", + "attack": "primal_bdd", + "seconds": 5.101183895953, + "log2_rop": 340.7298617250473, + "cost": { + "rop": "3.71457214481230e102", + "red": "3.45038992834586e102", + "svp": "2.64182216466433e101", + "beta": "1090", + "eta": "1129", + "d": "32344", + "tag": "bdd", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)" + } + }, + { + "samples": "N", + "model": "MATZOV-classical", + "attack": "dual", + "seconds": 0.04392242501489818, + "log2_rop": 343.3327532463879, + "cost": { + "rop": "2.25661370326375e103", + "mem": "27004289658070035888654106211172915795011311755896265023025172176204854", + "m": "16204", + "beta": "1096", + "d": "32588", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "N", + "model": "MATZOV-classical", + "attack": "dual_hybrid", + "seconds": 14.51800377201289, + "log2_rop": 339.58676911414483, + "cost": { + "rop": "1.68191341284720e102", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)", + "red": "1.68190843684125e102", + "guess": "4.97600594903040e96", + "beta": "1083", + "p": "3", + "zeta": "0", + "t": "190", + "beta_": "1066", + "N": "7.82287607371639e65", + "m": "16384" + } + }, + { + "samples": "N", + "model": "ADPS16-classical", + "attack": "primal_usvp", + "seconds": 0.04052111296914518, + "log2_rop": 318.864, + "cost": { + "rop": "9.71915565210739e95", + "red": "9.71915565210739e95", + "delta": "1.00191078002672", + "beta": "1092", + "d": "32083", + "tag": "usvp", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)" + } + }, + { + "samples": "N", + "model": "ADPS16-classical", + "attack": "primal_bdd", + "seconds": 5.235912185860798, + "log2_rop": 319.160068553334, + "cost": { + "rop": "1.19331212308321e96", + "red": "1.18995159682775e96", + "svp": "3.36052625546777e93", + "beta": "1093", + "eta": "1063", + "d": "32383", + "tag": "bdd", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)" + } + }, + { + "samples": "N", + "model": "ADPS16-classical", + "attack": "dual", + "seconds": 0.03102129092440009, + "log2_rop": 319.73999999999995, + "cost": { + "rop": "1.78373696975310e96", + "mem": "273648654219124481565040050707290299685135857117579618854340086718143754", + "m": "16193", + "beta": "1095", + "d": "32577", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "N", + "model": "ADPS16-classical", + "attack": "dual_hybrid", + "seconds": 8.881262043956667, + "log2_rop": 316.82, + "cost": { + "rop": "2.35680269580672e95", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)", + "red": "2.35680269580672e95", + "guess": "3.23123409447432e77", + "beta": "1085", + "p": "3", + "zeta": "0", + "t": "150", + "beta_": "1085", + "N": "3.83009980795253e67", + "m": "16384" + } + }, + { + "samples": "N", + "model": "ADPS16-quantum", + "attack": "primal_usvp", + "seconds": 0.03971595619805157, + "log2_rop": 289.38, + "cost": { + "rop": "1.29437508644779e87", + "red": "1.29437508644779e87", + "delta": "1.00191078002672", + "beta": "1092", + "d": "32083", + "tag": "usvp", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)" + } + }, + { + "samples": "N", + "model": "ADPS16-quantum", + "attack": "primal_bdd", + "seconds": 5.272714625112712, + "log2_rop": 289.65199368299255, + "cost": { + "rop": "1.56292595152198e87", + "red": "1.55536776548515e87", + "svp": "7.55818603682939e84", + "beta": "1093", + "eta": "1063", + "d": "32383", + "tag": "bdd", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)" + } + }, + { + "samples": "N", + "model": "ADPS16-quantum", + "attack": "dual", + "seconds": 0.02500018198043108, + "log2_rop": 290.175, + "cost": { + "rop": "2.24584092635855e87", + "mem": "273648654219124481565040050707290299685135857117579618854340086718143754", + "m": "16193", + "beta": "1095", + "d": "32577", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "N", + "model": "ADPS16-quantum", + "attack": "dual_hybrid", + "seconds": 8.84267289401032, + "log2_rop": 287.5250000013029, + "cost": { + "rop": "3.57807552817983e86", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=16384, tag=None)", + "red": "3.57807552494859e86", + "guess": "3.23123409447432e77", + "beta": "1085", + "p": "3", + "zeta": "0", + "t": "150", + "beta_": "1085", + "N": "3.83009980795253e67", + "m": "16384" + } + }, + { + "samples": "unbounded", + "model": "MATZOV-classical", + "attack": "primal_usvp", + "seconds": 0.04323683097027242, + "log2_rop": 341.1723757172288, + "cost": { + "rop": "5.04799345125361e102", + "red": "5.04799345125361e102", + "delta": "1.00191078002672", + "beta": "1092", + "d": "32083", + "tag": "usvp", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "MATZOV-classical", + "attack": "primal_bdd", + "seconds": 5.31509437202476, + "log2_rop": 340.7298617250473, + "cost": { + "rop": "3.71457214481230e102", + "red": "3.45038992834586e102", + "svp": "2.64182216466433e101", + "beta": "1090", + "eta": "1129", + "d": "32344", + "tag": "bdd", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "MATZOV-classical", + "attack": "dual", + "seconds": 0.03988387901335955, + "log2_rop": 343.3327532463879, + "cost": { + "rop": "2.25661370326375e103", + "mem": "27004289658070035888654106211172915795011311755896265023025172176204854", + "m": "16204", + "beta": "1096", + "d": "32588", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "unbounded", + "model": "MATZOV-classical", + "attack": "dual_hybrid", + "seconds": 14.420737589942291, + "log2_rop": 339.58676911414483, + "cost": { + "rop": "1.68191341284720e102", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)", + "red": "1.68190843684125e102", + "guess": "4.97600594903040e96", + "beta": "1083", + "p": "3", + "zeta": "0", + "t": "190", + "beta_": "1066", + "N": "7.82287607371639e65", + "m": "16384" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-classical", + "attack": "primal_usvp", + "seconds": 0.040466082049533725, + "log2_rop": 318.864, + "cost": { + "rop": "9.71915565210739e95", + "red": "9.71915565210739e95", + "delta": "1.00191078002672", + "beta": "1092", + "d": "32083", + "tag": "usvp", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-classical", + "attack": "primal_bdd", + "seconds": 5.354171877028421, + "log2_rop": 319.160068553334, + "cost": { + "rop": "1.19331212308321e96", + "red": "1.18995159682775e96", + "svp": "3.36052625546777e93", + "beta": "1093", + "eta": "1063", + "d": "32383", + "tag": "bdd", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-classical", + "attack": "dual", + "seconds": 0.027775981929153204, + "log2_rop": 319.73999999999995, + "cost": { + "rop": "1.78373696975310e96", + "mem": "273648654219124481565040050707290299685135857117579618854340086718143754", + "m": "16193", + "beta": "1095", + "d": "32577", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-classical", + "attack": "dual_hybrid", + "seconds": 8.707200119970366, + "log2_rop": 316.82, + "cost": { + "rop": "2.35680269580672e95", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)", + "red": "2.35680269580672e95", + "guess": "3.23123409447432e77", + "beta": "1085", + "p": "3", + "zeta": "0", + "t": "150", + "beta_": "1085", + "N": "3.83009980795253e67", + "m": "16384" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-quantum", + "attack": "primal_usvp", + "seconds": 0.040943962056189775, + "log2_rop": 289.38, + "cost": { + "rop": "1.29437508644779e87", + "red": "1.29437508644779e87", + "delta": "1.00191078002672", + "beta": "1092", + "d": "32083", + "tag": "usvp", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-quantum", + "attack": "primal_bdd", + "seconds": 5.37198440101929, + "log2_rop": 289.65199368299255, + "cost": { + "rop": "1.56292595152198e87", + "red": "1.55536776548515e87", + "svp": "7.55818603682939e84", + "beta": "1093", + "eta": "1063", + "d": "32383", + "tag": "bdd", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-quantum", + "attack": "dual", + "seconds": 0.026228047907352448, + "log2_rop": 290.175, + "cost": { + "rop": "2.24584092635855e87", + "mem": "273648654219124481565040050707290299685135857117579618854340086718143754", + "m": "16193", + "beta": "1095", + "d": "32577", + "repetitions": "1", + "tag": "dual" + } + }, + { + "samples": "unbounded", + "model": "ADPS16-quantum", + "attack": "dual_hybrid", + "seconds": 8.744628780055791, + "log2_rop": 287.5250000013029, + "cost": { + "rop": "3.57807552817983e86", + "problem": "LWEParameters(n=16384, q=1532495540865191964058577030327937992314109603578642433, Xs=D(\u03c3=0.82), Xe=D(\u03c3=3.24), m=+Infinity, tag=None)", + "red": "3.57807552494859e86", + "guess": "3.23123409447432e77", + "beta": "1085", + "p": "3", + "zeta": "0", + "t": "150", + "beta_": "1085", + "N": "3.83009980795253e67", + "m": "16384" + } + } + ], + "complete": true +} diff --git a/docs/index.rst b/docs/index.rst index 2b902aa..3694ff7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -88,6 +88,7 @@ Choose your path: research/native-bfv-fused-rns research/native-bfv-security research/native-bfv-verified-sessions + research/native-bfv-predecryption research/bfv-packed-hamming .. toctree:: diff --git a/docs/research/native-bfv-predecryption.md b/docs/research/native-bfv-predecryption.md new file mode 100644 index 0000000..0567abe --- /dev/null +++ b/docs/research/native-bfv-predecryption.md @@ -0,0 +1,466 @@ +# BFV approval before decryption, private arithmetic and parameter review + +Date: 2026-09-11. Branch: `research/bfv-packed-hamming`. + +**The new `BFVGuardedClient` blocks the demonstrated malicious-response oracle +under an explicit additional trust assumption:** an owner-controlled verifier +independently recomputes the prescribed public BFV result and approves its exact +bytes. The client checks that approval before parsing or decrypting the result. +The verifier needs public BFV keys and ciphertexts, but no BFV secret key, +plaintext vectors or private result summary. + +This is an implemented protocol baseline for review, not a claim of general +CCA2-secure BFV or production readiness. It costs another full evaluation and +depends on the verifier's integrity and signing-key custody. It is neither a +succinct cryptographic proof nor attestation of an untrusted server. A signer +controlled by the malicious evaluator provides no protection. If the owner can +simply use the trusted verifier as the evaluator, the extra untrusted evaluation +can be avoided at the application level; this implementation intentionally +measures independent checking of an existing evaluator. + +The [earlier session report](native-bfv-verified-sessions.md) remains the record +of the vulnerable post-decryption check. Its attack test is retained. The raw +BFV and old session APIs retain their behavior and require explicit migration +to acquire the new gate. Paillier, the SEAL experiment and production endpoints +are unchanged. This work is internal, AI-assisted engineering, with no +independent cryptographic review. + +## Code map and local use + +All paths below are relative to the repository root. + +| File | Purpose | +| --- | --- | +| `src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py` | Owner verifier, Ed25519 receipts, mandatory pre-decryption checks and bound private exports | +| `src/xtrace_sdk/x_vec/crypto/bfv_assurance.py` | Larger-ring review profile and exact conservative circuit bounds | +| `src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py` | Python adapter to the separate private native decoder | +| `src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h` | Fixed-width private NTT, CRT, rounding, locked buffers and wiping | +| `src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp` | Bounded CPython interface, capsule ownership and GIL handling | +| `src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py` | Small change: optional native private decoding; original default remains Python | +| `tests/x_vec/test_bfv_guarded_client.py` | Forgery/oracle, pin, replay, concurrency and saved-state regressions | +| `tests/x_vec/test_bfv_private.py` | Independent arithmetic, boundary, ownership and memory-lock tests | +| `tests/x_vec/test_bfv_assurance.py` | Bound checks and observed-phase comparisons | +| `tests/x_vec/native/test_bfv_private.cpp` | Standalone sanitizer, wipe and compiled secret-taint tests | +| `benchmarks/bfv_guarded.py` | Same-ciphertext performance and all communication legs | +| `benchmarks/bfv_parameter_audit.py` | Actual parameter manifest and pinned external lattice estimates | + +Build both optional CPU extensions on Linux with GCC or Clang: + +```bash +make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" +``` + +The public `_bfv_rns` ABI remains 4; the new, separate `_bfv_private` ABI is 1. +The guarded client defaults to native private decoding and the review profile. +Missing binaries, incompatible ABIs, unsupported parameters and memory-lock +failure raise errors; there is no automatic private-backend fallback. +`private_backend="python"` is an explicit research option with the same receipt +gate and the old variable-time private implementation. + +```python +import secrets + +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_guarded_client import ( + BFVGuardedClient, BFVPublicVerifier, bfv_verifier_public_key, +) +from xtrace_sdk.x_vec.crypto.bfv_verified_client import BFVVerifiedServer + +policy = bfv_review_policy() # N=16384; fresh keys and index required. +authentication_key = secrets.token_bytes(32) # Shared with evaluator. +signing_seed = secrets.token_bytes(32) # Owner-controlled verifier ONLY. +wrapping_key = secrets.token_bytes(32) # Private client storage ONLY. +pin = bfv_verifier_public_key(signing_seed) # Provision through a trusted path. +client = BFVGuardedClient( + BFVClient(**policy.config()), authentication_key, pin, policy=policy +) +setup = client.prepare_index( + [[0] * 512, [1] * 512], vector_ids=["document-a", "document-b"] +) + +# Untrusted evaluator; only public BFV material is imported. +server = BFVVerifiedServer(setup, authentication_key, policy=policy) + +# Run locally on the owner, or in a SEPARATELY TRUSTED owner-controlled service. +# Obtain the expected digest directly from the owner, not from the evaluator. +verifier = BFVPublicVerifier( + setup, authentication_key, signing_seed, + expected_setup_digest=client.setup_digest, policy=policy, +) +request = client.begin_query([0] * 512) +response = server.search(request) +receipt = verifier.approve(request, response) # A second full public evaluation. +distances = client.finish_query(response, receipt) +assert distances == [0, 512] + +protected = client.protect_state(wrapping_key) +restored = BFVGuardedClient.restore( + setup, protected, authentication_key, wrapping_key, pin, policy=policy +) +``` + +This example is local orchestration, not a network service or secure process +isolation. In deployment the signing seed must never reach the untrusted +evaluator. Pin both the verifier identity and owner setup through trusted +configuration, authenticate owner authorization at the verifier, and review +the actual components running inside that trusted service. + +## Why approval changes the oracle boundary + +The previous result fingerprint checked plaintext *after* decryption. A +malicious server could submit carefully chosen ciphertexts and learn a secret +coefficient from the client's acceptance. Transport authentication could not +exclude that server because it knew the shared authentication key. + +The new sequence is: + +1. The owner creates an immutable authenticated index setup and a fresh query + ticket, retaining the SHA-256 digest of its exact encrypted request. +2. The evaluator produces its response. The owner-controlled verifier uses an + independently imported copy of the owner-pinned setup to recompute the + complete canonical response, including terminal modulus switching. +3. The verifier compares every response byte, then signs only that transcript. + Its API cannot sign arbitrary caller-provided hashes. These decisions use + public ciphertexts only; even malicious queries cannot invoke BFV decryption + at the verifier. +4. The client checks fixed receipt length, version, circuit, setup, pending + owner-request binding, response digest and Ed25519 signature **before BFV + response parsing or any private decoding**. Rejection cannot expose the + BFV decryption predicate used in the historical attack. +5. The inner session still consumes the request once, enforces response shape + and modulus, and checks the private Hamming fingerprint before returning + the complete distance list. Concurrent replays cannot decrypt twice. + +The owner-request check is essential. The evaluator knows the transport MAC +key and can construct its own authenticated requests. A valid receipt for one +of those requests must not authorize decryption on the owner. Tests cover this +case as well as the earlier rounding-oracle ciphertexts and check that neither +the GMP nor native decoder is reached. + +Each receipt is **198 bytes**: a 6-byte version prefix, four 32-byte fields +(circuit identifier, setup digest, request digest, response digest), and a +64-byte Ed25519 signature. Signing uses a versioned Ed25519 context. The pin +comes only from local configuration; small-order public keys, including the +encoded identity accepted by the underlying import API, are explicitly +rejected. Cryptographic signing uses the existing dependency, not a new +signature implementation. +[PyCryptodome EdDSA documentation](https://www.pycryptodome.org/src/signature/eddsa). + +The circuit identifier describes the implemented deterministic computation; +it is not a proof that arbitrary software follows it. Semantic or canonical +wire changes require a new circuit version, compatible trusted code and fresh +deployment review. A bad proposed result consumes the verifier's query ticket, +so a retry needs a new owner request. Missing, forged or unrelated receipts do +not consume the client's legitimate pending ticket. The outer AES-GCM saved +state additionally binds this gate and verifier pin, preventing accidental +restoration through the old unguarded session API. It adds 34 bytes over the +inner protected export and does not supply durable anti-rollback protection. + +This implements an explicit trusted-computation boundary, **not** the +SNARK-based transformation analyzed in the verifiable-FHE literature. +That literature motivates binding permitted outputs to original inputs before +opening a decryption interface; it does not prove this SDK protocol secure. +[Manulis and Nguyen, EUROCRYPT 2024](https://eprint.iacr.org/2024/202). +SEAL also warns that decryptor outputs/noise information require private +handling; importing a ciphertext does not establish its authenticity. +[SEAL security guidance](https://github.com/microsoft/SEAL/blob/main/SECURITY.md). + +## Private arithmetic: scope and evidence + +The separate C++ decoder imports a ternary key once and keeps two secret NTT +spectra. For each two-component terminal ciphertext it computes exact +negacyclic `c1*s`, adds `c0`, rounds the phase modulo the plaintext modulus, +and batch-decodes all N slots. The input/output lengths and loop bounds depend +on public parameters, never a secret coefficient or decoded distance. + +Two descending 60-bit NTT primes allow exact signed CRT reconstruction: +`|c1*s| < N*q < 2^65`, while half their product exceeds `2^117`. The private +path supports power-of-two N from 8 through 32768, `t < 2^30` with batching +roots, and terminal `t < q < 2^50`. Fixed 67- and 82-iteration binary dividers +replace division on secret phase values. Root setup and Shoup precomputation +can divide public values; private multiplication uses fixed-width products +and masked correction. The public server's faster variable-time NTT is not +reused for secret transforms. + +Testing exposed Clang 18 turning ordinary masked corrections into branches. +A GCC/Clang register value barrier now prevents that optimization in the +tested builds. This is a known compiler-control technique, also documented in +[BoringSSL's constant-time helpers](https://boringssl.googlesource.com/boringssl/+/refs/heads/main/crypto/internal.h). +The resulting GCC 13.3 and Clang 18 `-O3` kernels pass dynamic secret-taint +checks for N=16,128,8192 and positive, negative and mixed ternary keys. The test +marks persistent secret spectra and transform inputs undefined in Valgrind, +then declassifies plaintext only after decoding returns. A deliberate secret +branch is a negative control and must fail. CI runs both compilers. + +These tests detect executed secret-dependent branches and addresses. They do +not prove constant latency for every instruction, compiler, processor or +input, or cover speculative execution, faults, power analysis, hostile kernels +or physical access. Source review, the independent arithmetic oracles and +sanitizers supplement the taint tests; none is a side-channel certification. + +Persistent native secret buffers and temporary private work use separate +`mmap` allocations, mandatory `mlock`, Linux `MADV_DONTDUMP` and volatile +full-region wiping. Lock/exclusion failure stops the operation. Decode and +close share a mutex; closing waits without holding Python's GIL and wipes the +native key before rejecting further decode calls. Capsule destruction also +wipes. At N=16384, a decoder locks 262,144 persistent bytes plus 393,216 +temporary bytes during a decode; page rounding may matter for small rings. +Provision sufficient per-process lock limits. An isolated OS-limit test checks +that failure never silently falls back to unlocked storage. + +**The whole private client is still not constant time or fully erasable.** +Python/GMP key generation, key import, encryption, private summary checking, +Python key objects, temporary imported key bytes and returned plaintext remain +outside the native kernel's guarantees. Existing keys are retained for future +encryption and protected exports. Wiping native buffers does not erase those +copies, registers, allocator remnants or the caller's results. A production +review must address those paths and process/key custody as well. + +## Concrete parameter estimates + +The runner records the actual product Q, not the larger multiplication +workspace basis. Both profiles use t=65537, uniform ternary secret/ephemeral +coefficients, centered-binomial error eta=21 (variance 10.5), six 30-bit gadget +digits and a 50-bit terminal response prime. + +| Profile | N | Q primes | Public-key JSON cap | +| --- | ---: | --- | ---: | +| Original arithmetic default | 8192 | 1152921504606830593, 1152921504606748673, 1152921504606683137 | 128 MiB | +| Guarded review default | 16384 | 1152921504606748673, 1152921504606683137, 1152921504606584833 | 256 MiB | + +Sage 10.9 ran lattice-estimator commit +`53da5982597709ba0fdf94ea37a84d822310fd84`, with `ND.Uniform(-1,1)` and +`ND.CenteredBinomial(21)`. Each profile has 24 estimates: primal uSVP, primal +BDD, dual and dual-hybrid, each with m=N and unbounded independent samples, +under three reduction-cost models. Primal runs use the geometric-series shape +model. The following are minima of reported `log2(rop)` across the selected +attacks and sample counts, **not assigned security levels**. + +| Cost model | Original N=8192 | Review N=16384 | +| --- | ---: | ---: | +| MATZOV classical | 154.40 | 339.59 | +| ADPS16 classical sensitivity | 124.69 | 316.82 | +| ADPS16 quantum-sieving sensitivity | 113.42 | 287.53 | + +The disagreement matters: a historical modulus table or one favorable model +cannot justify calling the original profile 128-bit secure. The larger ring +provides more estimated margin, at a measured cost. It requires fresh keys +and index creation; existing raw defaults remain unchanged for reproducibility. + +Important limits of these estimates: generic LWE models do not certify RLWE +structural security or the circular/key-dependent-message assumptions of the +published s² and automorphism evaluation keys. m=infinity is a sensitivity +case, not a proof that related ring samples are independent. Unlisted attacks +and cost/shape models are not evaluated. The MATZOV source describes a fit +through block size 1024; some larger-ring runs exceed it and extrapolate. +ADPS16 quantum sieving is not a complete quantum attack analysis. Moreover, +Ed25519 receipts are not post-quantum signatures, so the whole guarded protocol +must not be described as post-quantum secure based on an LWE number. +[Lattice-estimator source and interpretation](https://github.com/malb/lattice-estimator), +[current HE parameter guidance](https://homomorphicencryption.org/security-guidelines/). + +Results, all attack outputs, versions and manifest/source hashes are in: + +- `benchmarks/results/native_bfv_parameter_estimates.json` +- `benchmarks/results/native_bfv_review_parameter_estimates.json` + +Reproduce with an independently installed Sage Python and the pinned estimator: + +```bash +git clone https://github.com/malb/lattice-estimator /tmp/xtrace-lattice-estimator +git -C /tmp/xtrace-lattice-estimator checkout 53da5982597709ba0fdf94ea37a84d822310fd84 +.venv/bin/python benchmarks/bfv_parameter_audit.py manifest --profile review \ + --json-out /tmp/bfv-review-manifest.json +DOT_SAGE=/tmp/xtrace-sage /opt/sage/bin/python benchmarks/bfv_parameter_audit.py estimate \ + --manifest /tmp/bfv-review-manifest.json --estimator /tmp/xtrace-lattice-estimator \ + --json-out /tmp/bfv-review-estimates.json +``` + +The Sage executable path is installation-specific. `--profile original` +exports the other profile. No BFV keys are generated or stored by this audit. + +## Conservative correctness bound for this circuit + +`hamming_noise_bound` provides a source-level derivation for independent +review. It uses exact integers and rational numbers with outward rounding, +assuming honestly generated keys, bounded CBD errors, binary owner inputs and +the implemented square-difference/rotate/mask/merge circuit. It is not a +ciphertext validity test, formal proof, or assurance for malicious setup or +arbitrary FHE circuits. The guarded client requires this sufficient bound at +index creation and restoration, separately from receipt validation. + +Let `Q = delta*t + r`, canonical plaintext polynomial coefficients lie in +`[0,t-1]`, and the phase invariant be +`Phi(c) = delta*m + v (mod Q)`, with `||v||_infinity <= V`. +For negacyclic polynomials, use `||a*b|| <= N*||a||*||b||`. + +Fresh encryption has `Venc=(2N+1)*eta`; subtracting two ciphertexts gives +`Vdiff=2*Venc+r`. A gadget switch contributes at most +`S=N*digits*(2^gadget_bits-1)*eta`. For a canonical two-component difference, +write its **integer** phase as `delta*m+v+Q*k`, with + +```text +L = ceil(((N+1)*(Q-1) + delta*(t-1) + Vdiff) / Q) +||k|| <= L +P = N*(t-1)^2 +C = ceil((P+t-1)/t) # Plaintext-product carry bound. +``` + +BFV tensor multiplication computes integer component products before rounding +each by t/Q. Expanding the two integer phases introduces lift terms +`t*(v*k' + v'*k) - r*(m*k' + m'*k)` modulo Q. Omitting them would incorrectly +treat scale-and-round as an operation on phase representatives alone. With +both input bounds equal to Vdiff, a sufficient bound after relinearization is + +```text +Vsquare = ceil( + r*C + delta*r*P/Q + + 2*N*(t-1)*Vdiff + t*N*Vdiff^2/Q + + 2*t*N*Vdiff*L + 2*r*N*(t-1)*L + + (1+N+N^2)/2 +) + S +``` + +The last fraction bounds the three component rounding errors after evaluation +under `1,s,s²`. Let W be the next power of two at least the vector dimension. +Apply `V <- 2V+S+2r` for each of the log2(W) dimension-reduction rotations and +adds. Multiplication by the plaintext selection mask gives +`Vtile=N*(t-1)*V+r*C`. At most K=W tiles enter one response ciphertext, with +K-1 further rotate/add nodes, so +`Vmerged=K*Vtile+(K-1)*(S+2r)`; partial response groups use their actual K. + +Compacting to q' gives +`Vterminal=ceil(q'*Vmerged/Q + (t-1) + (N+1)/2)`. +The extra terms cover the delta-scale mismatch and two component rounding +errors. Exact decryption of every coefficient follows from the sufficient +strict inequalities + +```text +2*(t*Vmerged + (Q mod t)*(t-1)) < Q +2*(t*Vterminal + (q' mod t)*(t-1)) < q' +``` + +For D=512 at the policy's numerical maximum of 65,536 rows, these bounds pass +for both 180-bit-Q profiles. `Vterminal` is 69,633 at N=8192 and 73,729 at +N=16384. This covers more rows than the default index byte cap permits; the +independent resource caps still apply and do not promise every maximum can +fit together. A 120-bit-Q probe passed small random correctness tests but +failed this sufficient full-workload bound, so it was not adopted as the +guarded default. A failing bound means assurance is insufficient; it does +not by itself demonstrate a decryption failure. + +## Performance and validation + +On a Ryzen 7 5800X, Python 3.12.3 and GMP 6.3.0, the final-source benchmark +uses 1,024 vectors of 512 bits, one cold trial and three warm trials. The +following times are warm medians. Setup/import and query encryption are +separate from server evaluation. Within each profile the evaluated ciphertexts +are identical; the two profiles have fresh keys and different ciphertexts. +No network timing or concurrent overlap is claimed. + +| Measurement | Original N=8192 | Review N=16384 | +| --- | ---: | ---: | +| Evaluator search, authentication and serialization | 2.35395 s | 2.43866 s | +| Additional owner-verifier recomputation and receipt | 2.35376 s | 2.43529 s | +| Paired evaluator + verifier time | 4.70963 s | 4.87396 s | +| Python reference distance decoding and parsing | 39.42 ms | 80.29 ms | +| Native distance decoding and parsing | 12.46 ms | 24.65 ms | +| Complete guarded client finish, including receipt and private check | 16.13 ms | 28.43 ms | +| Query encryption and binding | 100.59 ms | 204.76 ms | + +At the larger ring, native decoding is **3.26 times faster** than the Python +reference on the same responses. The evaluator itself is **3.6% slower** than +the original-ring run: doubling N halves the number of packed index tiles in +this workload, offsetting much of the extra arithmetic per tile. These are +local measurements, not a general scaling guarantee. The required second +evaluation dominates the added protocol cost; overall delegated compute is +about doubled, despite preserving the existing fast evaluator. + +One-time key generation takes 4.32/8.87 seconds and index encryption, private +summary construction, export and authentication take 8.08/9.42 seconds for the +original/review profiles. Each evaluator imports setup in roughly 1.73/3.37 +seconds; the verifier has its own comparable import cost. Cold searches take +2.67/3.07 seconds as lazy public key/mask caches are prepared. The benchmark +process, holding both evaluators and the private reference, peaks at +1,094,104/1,985,372 KiB RSS. This is whole-process memory, not a per-server or +locked-secret-buffer measurement. + +| Serialized item, decimal bytes | Original N=8192 | Review N=16384 | +| --- | ---: | ---: | +| BFV public/evaluation keys, per evaluator | 85,602,438 | 171,205,156 | +| Encrypted-index array, per evaluator | 23,598,467 | 23,595,715 | +| Complete authenticated setup, per evaluator | 109,205,199 | 194,805,166 | +| Client query | 368,840 | 737,480 | +| Evaluator response | 102,614 | 205,018 | +| Approval receipt | 198 | 198 | +| Client response + receipt download | 102,812 | 205,216 | +| Client query + response + receipt | 471,652 | 942,696 | +| Additional query + response input to a remote owner verifier | 471,454 | 942,498 | + +The review profile approximately doubles the client exchange and public key +size. Packed index size stays almost constant because fewer larger tiles hold +the same rows; small framing/JSON differences depend on the particular random +keys and ciphertexts. The private summary stays at 16,448 bytes for D=512 and +never goes to either evaluator. Both public evaluators need their own setup. + +Results live in `benchmarks/results/native_bfv_guarded_original_1024.json` and +`benchmarks/results/native_bfv_guarded_review_1024.json`. JSON contains synthetic +workload dimensions, timings, sizes and source/binary hashes, with no keys, +ciphertexts or private summaries. + +```bash +.venv/bin/python benchmarks/bfv_guarded.py --profile original \ + --json-out /tmp/bfv-guarded-original.json +.venv/bin/python benchmarks/bfv_guarded.py --profile review \ + --json-out /tmp/bfv-guarded-review.json +.venv/bin/python -m pytest tests/x_vec/test_bfv_guarded_client.py \ + tests/x_vec/test_bfv_private.py tests/x_vec/test_bfv_assurance.py -q +``` + +For a remote owner verifier, each search additionally sends it the query and +proposed response, on top of provisioning a second copy of public keys/index. +The client receives the 198-byte receipt. Counting only those 198 bytes as the +cost of the entire distributed protocol would omit that additional traffic +and full recomputation. If verification runs locally, the owner instead needs +the public setup and compute resources locally. The larger ring separately +changes query, response and key sizes. + +Validation of the final implementation: + +- **351 offline tests pass:** 323 SDK tests, including 269 BFV cases, plus + 28 retained SEAL experiment tests. Production HTTP integrations and GPU + hardware tests are excluded. All 49 new security/arithmetic/bound cases pass. +- **183 binding/security tests pass under AddressSanitizer and + UndefinedBehaviorSanitizer.** One OS memory-lock test is deliberately skipped + because ASan intercepts `mlock`; it passes with the release binary and a + zero-lock-limit child process. Both sanitized extension paths are explicitly + loaded, preventing an accidental release-binary fallback. LeakSanitizer is + disabled in this environment because execution uses ptrace. +- The standalone private arithmetic/wipe test passes ASan/UBSan. The GCC 13.3 + and Clang 18 optimized secret-taint runs each report zero errors and no heap + leaks; both deliberately branching negative controls fail as required. +- SDK and BFV benchmark lint/type checks pass. The platform wheel contains + both native modules and the new Python APIs. Documentation builds offline + with the pre-existing SEAL research-link warning. The documented N=16384 + example, including guarded state restoration, passes from the extracted wheel. + +The benchmark JSON records base revision `0cd7b41` with a dirty working tree; +the recorded source/binary hashes identify the measured implementation and +match the final files. Estimator outputs likewise match their pinned manifests +and runner hash. These checks are reproducible engineering evidence, not an +independent security audit. + +## Remaining production gates + +| Area | Evidence added here | Remaining requirement | +| --- | --- | --- | +| Malicious-result oracle | Exact public recomputation receipt checked before private work; old attack blocked in guarded regression | Independent protocol review and actual owner-verifier isolation, pinning and key custody; no general CCA2 or succinct proof claim | +| Private implementation | Separate fixed-work decoder, locked/wiped native buffers, GCC/Clang taint tests, arithmetic oracles and sanitizers | Full private key generation/encryption/import/summary path, compiler/CPU review, Python copies and deployment side channels | +| Parameters and failures | Pinned actual-distribution estimates and conservative circuit-specific bounds; larger review profile | Independent analysis of models, omitted attacks, ring/evaluation-key assumptions and bound derivation | +| Service integrity/availability | Existing per-object tickets, caps and admission; guarded exports bind verifier pin | Durable anti-rollback/replay and quotas, owner authorization at verifier, tenant/process isolation and transport limits | +| Application leakage | Explicitly unchanged all-distance interface | Review chosen-query capability, result-dependent fetches, IDs/sizes/timing; private top-k/PIR and circuit privacy remain unimplemented | + +The implemented trust boundary materially changes which ciphertexts may reach +decryption, while these deployment and cryptographic assurance tasks still +prevent an honest production-security sign-off. diff --git a/docs/research/native-bfv-security.md b/docs/research/native-bfv-security.md index fda596c..3a1f700 100644 --- a/docs/research/native-bfv-security.md +++ b/docs/research/native-bfv-security.md @@ -16,6 +16,12 @@ with those checks. The findings below describe the original raw BFV APIs; the follow-up records the new mitigations, their performance and the remaining production blockers. +**Later 2026-09-11 follow-up:** [approval before decryption](native-bfv-predecryption.md) +adds a guarded API with owner-controlled public recomputation, a separate +private C++ decoder, pinned lattice estimates and conservative circuit bounds. +It conditionally blocks the demonstrated oracle at an additional trust and +compute cost. The historical findings below remain applicable to the raw APIs. + The optimization baseline reviewed is commit `06d8d17`, following `e5c0751`. This report and the accompanying hardening are on `research/bfv-packed-hamming`. No production endpoint or `main` change is part diff --git a/docs/research/native-bfv-verified-sessions.md b/docs/research/native-bfv-verified-sessions.md index c0c722e..9ec6100 100644 --- a/docs/research/native-bfv-verified-sessions.md +++ b/docs/research/native-bfv-verified-sessions.md @@ -9,6 +9,12 @@ recovery from observable accept/reject feedback even through the new result checker. This follow-up is internal, AI-assisted engineering and testing, not an independent cryptographic audit or production certification. +**Subsequent work:** [the guarded pre-decryption API](native-bfv-predecryption.md) +now blocks this demonstrated oracle using exact public recomputation by a +separately trusted owner verifier. That report covers its trust/compute cost, +native private arithmetic and parameter review. The results and open-status +table below describe the original result-checked API, which remains available. + This extends the [original audit](native-bfv-security.md). The raw BFV, Paillier and SEAL paths remain available. Existing `BFVClient` calls do not automatically acquire these protections; applications must explicitly use the diff --git a/docs/research/native-bfv.md b/docs/research/native-bfv.md index c7fa042..bd22479 100644 --- a/docs/research/native-bfv.md +++ b/docs/research/native-bfv.md @@ -16,6 +16,12 @@ adds request/index binding, private Hamming result checks, encrypted private exports and admission limits. It preserves the server arithmetic and raw APIs; the follow-up report measures its overhead and records the remaining blockers. +The newer [pre-decryption approval layer](native-bfv-predecryption.md) requires +an owner-controlled verifier to recompute and approve exact response bytes. +It also introduces a separate native private decoder and a larger parameter +review profile. Its report explains the additional trust/compute cost, attack +regressions and remaining production review requirements. + ## Code map All paths are relative to the repository root. diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_assurance.py b/src/xtrace_sdk/x_vec/crypto/bfv_assurance.py new file mode 100644 index 0000000..142b170 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_assurance.py @@ -0,0 +1,129 @@ +"""Reviewable public parameter policy and conservative Hamming noise bounds. + +These are engineering evidence, not independent security certification. Bounds +assume honestly generated ternary secrets/ephemerals, bounded CBD errors, and +the exact existing square-difference, rotation, mask and merge circuit. They +do not validate an attacker-created ciphertext or replace the receipt gate. +""" + +from __future__ import annotations + +from dataclasses import asdict, dataclass +from fractions import Fraction + +from xtrace_sdk.x_vec.crypto.bfv_security import BFVExecutionPolicy +from xtrace_sdk.x_vec.crypto.encryption.bfv import _coefficient_modulus, _parameter_modulus +from xtrace_sdk.x_vec.utils.xtrace_types import BFVParameters + + +def bfv_review_policy() -> BFVExecutionPolicy: + """Larger ring for parameter review, preserving Q, t and the existing circuit. + + The raw BFV and earlier session defaults remain unchanged. This profile + needs fresh keys/index and a larger public-key import allowance. Its name + deliberately does not assert a production or cryptographic security level. + """ + return BFVExecutionPolicy( + params=BFVParameters(poly_modulus_degree=16384, rns_modulus=True), + max_public_key_bytes=256 * 1024 * 1024, + ) + + +def _ceil(value: Fraction) -> int: + return -(-value.numerator // value.denominator) + + +@dataclass(frozen=True) +class BFVHammingNoiseBound: + """Bounds for v in phase(c) = floor(Q/t)*m + v (mod Q), coefficientwise.""" + + encrypted: int + difference: int + squared: int + key_switch: int + tile: int + merged: int + compacted: int + original_rounding_error_numerator: int + terminal_rounding_error_numerator: int + original_modulus: int + terminal_modulus: int + + @property + def sufficient(self) -> bool: + # Strict inequalities cover both rounding boundaries. The numerator + # already includes plaintext representative error from Q mod t. + return ( + 2 * self.original_rounding_error_numerator < self.original_modulus + and 2 * self.terminal_rounding_error_numerator < self.terminal_modulus + ) + + def as_dict(self) -> dict[str, int | bool]: + return {**asdict(self), "sufficient": self.sufficient} + + +def hamming_noise_bound(policy: BFVExecutionPolicy, vector_count: int) -> BFVHammingNoiseBound: + """Bound every coefficient for the full supported binary Hamming workload. + + Uses ||a*b||_infinity <= N ||a||_infinity ||b||_infinity in Z[X]/(X^N+1). + All arithmetic is exact integer/rational arithmetic, with outward rounding. + See docs/research/native-bfv-predecryption.md for the multiplication lift + terms and the limits of this source-level derivation. + """ + if type(vector_count) is not int or not 1 <= vector_count <= policy.max_vectors: + raise ValueError("Vector count exceeds the local policy") + p = policy.params + n, t, eta = p.poly_modulus_degree, p.plain_modulus, p.error_eta + q = int(_parameter_modulus(p)) + target = min(int(_coefficient_modulus(policy.response_modulus_bits)), q) + delta, remainder = divmod(q, t) + width = 1 << (policy.embed_len - 1).bit_length() + capacity = n // width + tiles = min(width, (vector_count + capacity - 1) // capacity) + digits = (p.coeff_modulus_bits + p.decomposition_bits - 1) // p.decomposition_bits + switch = n * digits * ((1 << p.decomposition_bits) - 1) * eta + + # e_pk*u + e0 + e1*s, followed by the canonical plaintext subtraction. + encrypted = (2 * n + 1) * eta + difference = 2 * encrypted + remainder + # Canonical components bound the INTEGER phase lift k in + # phase = delta*m + v + Q*k, even when squaring the same ciphertext. + lift = _ceil(Fraction((n + 1) * (q - 1) + delta * (t - 1) + difference, q)) + message_product = n * (t - 1) ** 2 + message_carry = (message_product + t - 1 + t - 1) // t + tensor_rounding = Fraction(1 + n + n * n, 2) + squared = ( + _ceil( + remainder * message_carry + + Fraction(delta * remainder * message_product, q) + + 2 * n * (t - 1) * difference + + Fraction(t * n * difference**2, q) + + 2 * t * n * difference * lift + + 2 * remainder * n * (t - 1) * lift + + tensor_rounding + ) + + switch + ) + + # Each automorphism changes coefficient signs and performs one key switch; + # the following add introduces at most one further plaintext carry. + tile = squared + for _ in range(width.bit_length() - 1): + tile = 2 * tile + switch + 2 * remainder + tile = n * (t - 1) * tile + remainder * message_carry + # A merge of K tiles has K-1 rotate/add nodes, regardless of partial-tree shape. + merged = tiles * tile + (tiles - 1) * (switch + 2 * remainder) + compacted = _ceil(Fraction(target * merged, q) + (t - 1) + Fraction(n + 1, 2)) + return BFVHammingNoiseBound( + encrypted, + difference, + squared, + switch, + tile, + merged, + compacted, + t * merged + remainder * (t - 1), + t * compacted + (target % t) * (t - 1), + q, + target, + ) diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile index 555413e..be65f1a 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile @@ -7,17 +7,23 @@ CXX ?= c++ PY_INC := $(shell $(PYTHON) -c 'import sysconfig; print(sysconfig.get_paths()["include"])') EXT := $(shell $(PYTHON) -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))') TARGET := _bfv_rns$(EXT) +PRIVATE_TARGET := _bfv_private$(EXT) CXXFLAGS ?= -O3 CPPFLAGS ?= LDFLAGS ?= LDLIBS ?= -lgmpxx -lgmp .PHONY: all clean -all: $(TARGET) +all: $(TARGET) $(PRIVATE_TARGET) $(TARGET): bindings.cpp rns_ntt.h profile.h bfv_server.h bfv_residue.h rns_scale.h packed_wire.h Makefile $(CXX) $(CPPFLAGS) $(CXXFLAGS) -std=c++17 -Wall -Wextra -fPIC -shared \ -I$(PY_INC) bindings.cpp $(LDFLAGS) $(LDLIBS) -o $@ +$(PRIVATE_TARGET): private_bindings.cpp private_decoder.h rns_ntt.h profile.h Makefile + $(CXX) $(CPPFLAGS) $(CXXFLAGS) -std=c++17 -Wall -Wextra -fPIC -shared \ + -fstack-protector-strong -D_FORTIFY_SOURCE=3 -I$(PY_INC) private_bindings.cpp \ + $(LDFLAGS) -Wl,-z,relro,-z,now,-z,noexecstack $(LDLIBS) -o $@ + clean: - rm -f $(TARGET) + rm -f $(TARGET) $(PRIVATE_TARGET) diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.pyi b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.pyi new file mode 100644 index 0000000..602d4fe --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.pyi @@ -0,0 +1,7 @@ +"""Opaque native private decoder; this API must stay on the trusted client.""" + +ABI_VERSION: int + +def create_decoder(n: int, q: int, t: int, p0: int, p1: int, secret: bytes) -> object: ... +def decode_slots(context: object, c0: bytes, c1: bytes) -> bytes: ... +def close_decoder(context: object) -> None: ... diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp new file mode 100644 index 0000000..40311e1 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp @@ -0,0 +1,104 @@ +// Optional, separate private decoder extension. The public server ABI is unchanged. +#define PY_SSIZE_T_CLEAN +#include +#include "private_decoder.h" + +namespace { +using Decoder = xtrace_bfv_private::PrivateDecoder; +using Word = xtrace_bfv_private::Word; +using Handle = std::shared_ptr; +constexpr const char* capsule_name = "xtrace.bfv.private.decoder.v1"; +struct PythonError {}; +struct WithoutGIL { + PyThreadState* state = PyEval_SaveThread(); + ~WithoutGIL() { PyEval_RestoreThread(state); } +}; +template PyObject* checked(F&& f) { + try { return f(); } + catch (const PythonError&) { return nullptr; } + catch (const std::bad_alloc&) { return PyErr_NoMemory(); } + catch (const std::invalid_argument& e) { PyErr_SetString(PyExc_ValueError, e.what()); } + catch (const std::exception& e) { PyErr_SetString(PyExc_RuntimeError, e.what()); } + return nullptr; +} +Word integer(PyObject* object) { + if (!PyLong_CheckExact(object)) throw std::invalid_argument("Private BFV parameters must be integers"); + Word value = PyLong_AsUnsignedLongLong(object); + if (PyErr_Occurred()) throw PythonError{}; + return value; +} +Handle handle(PyObject* object) { + auto result = static_cast(PyCapsule_GetPointer(object, capsule_name)); + if (!result) throw PythonError{}; + return *result; +} +void destroy(PyObject* capsule) { + delete static_cast(PyCapsule_GetPointer(capsule, capsule_name)); +} +PyObject* create(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *n_arg, *q_arg, *t_arg, *p0_arg, *p1_arg, *secret; + if (!PyArg_ParseTuple(args, "OOOOOO", &n_arg, &q_arg, &t_arg, &p0_arg, &p1_arg, &secret)) return nullptr; + Word n = integer(n_arg), q = integer(q_arg), t = integer(t_arg), p0 = integer(p0_arg), p1 = integer(p1_arg); + if (n > 32768 || !PyBytes_CheckExact(secret) || PyBytes_GET_SIZE(secret) != static_cast(n)) + throw std::invalid_argument("Incorrect fixed-width private key length"); + Handle decoder; + { WithoutGIL release; + decoder = std::make_shared(n, q, t, p0, p1, + reinterpret_cast(PyBytes_AS_STRING(secret))); } + auto owner = std::make_unique(std::move(decoder)); + PyObject* result = PyCapsule_New(owner.get(), capsule_name, destroy); + if (result) owner.release(); + return result; + }); +} +std::vector read_component(PyObject* object, const Decoder& decoder) { + if (!PyBytes_CheckExact(object) || PyBytes_GET_SIZE(object) != static_cast(8 * decoder.degree())) + throw std::invalid_argument("Incorrect fixed-width private ciphertext length"); + const auto* bytes = reinterpret_cast(PyBytes_AS_STRING(object)); + std::vector result(decoder.degree()); + for (std::size_t i = 0; i < result.size(); ++i) { + Word value = 0; + for (unsigned j = 0; j < 8; ++j) value |= Word(bytes[8 * i + j]) << (8 * j); + if (value >= decoder.modulus()) throw std::invalid_argument("Noncanonical private ciphertext coefficient"); + result[i] = value; + } + return result; +} +PyObject* decode(PyObject*, PyObject* args) { + return checked([&]() -> PyObject* { + PyObject *context, *left, *right; + if (!PyArg_ParseTuple(args, "OOO", &context, &left, &right)) return nullptr; + auto decoder = handle(context); + auto c0 = read_component(left, *decoder), c1 = read_component(right, *decoder); + PyObject* output = PyBytes_FromStringAndSize(nullptr, 4 * decoder->degree()); + if (!output) return nullptr; + try { + { WithoutGIL release; + decoder->decode(c0.data(), c1.data(), reinterpret_cast(PyBytes_AS_STRING(output))); } + } catch (...) { Py_DECREF(output); throw; } + return output; + }); +} +PyObject* close(PyObject*, PyObject* context) { + return checked([&]() -> PyObject* { + auto decoder = handle(context); + // Waiting for an in-flight decoder must not hold the GIL it needs to return. + { WithoutGIL release; decoder->close(); } + Py_RETURN_NONE; + }); +} +PyMethodDef methods[] = { + {"create_decoder", create, METH_VARARGS, "Import a bounded ternary key into locked native buffers."}, + {"decode_slots", decode, METH_VARARGS, "Decode two canonical terminal components into fixed-width slots."}, + {"close_decoder", close, METH_O, "Wipe a decoder and reject subsequent private operations."}, + {nullptr, nullptr, 0, nullptr} +}; +PyModuleDef module = {PyModuleDef_HEAD_INIT, "_bfv_private", "Fixed-work BFV terminal decryption.", -1, methods, nullptr, nullptr, nullptr, nullptr}; +} +PyMODINIT_FUNC PyInit__bfv_private() { + PyObject* result = PyModule_Create(&module); + if (!result) return nullptr; + if (PyModule_AddIntConstant(result, "ABI_VERSION", 1) < 0) { Py_DECREF(result); return nullptr; } + return result; +} diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h new file mode 100644 index 0000000..de23676 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h @@ -0,0 +1,245 @@ +// Private BFV terminal decryption. No secret-dependent division or table index. +// This is a narrow, independently testable kernel, not an end-to-end CT claim. +#pragma once +#include "rns_ntt.h" // Public parameter setup only; private transforms below are separate. +#include +#include +#include + +namespace xtrace_bfv_private { +using Word = std::uint64_t; +using Wide = unsigned __int128; +using SignedWide = __int128; + +inline Word select_mask(Word condition) { + Word mask = Word(0) - condition; + // GCC/Clang register value barrier: prevent optimizing a masked operation + // back into a secret-dependent branch. The Clang -O3 taint regression + // detects those branches if this barrier is removed. Recheck each toolchain. +#if defined(__GNUC__) || defined(__clang__) + __asm__ volatile("" : "+r"(mask)); +#else +#error "Private BFV requires a reviewed GCC/Clang value barrier" +#endif + return mask; +} +inline Word normalize(Word value, Word p) { return value - (p & select_mask(value >= p)); } + +// Public multipliers (NTT roots or the PUBLIC c1 spectrum). Only multiply() +// receives secret data. Its inputs are > 64; + return normalize(input * value - estimate * p, p); + } +}; + +// Binary long division has PUBLIC loop count, no hardware divide, and remainder +// fixed_divide(Wide input, Word p, unsigned bits) { + Word remainder = 0, quotient = 0; + for (unsigned i = bits; i-- > 0;) { + remainder = (remainder << 1) | Word((input >> i) & 1); + Word subtract = remainder >= p; + remainder -= p & select_mask(subtract); + quotient = (quotient << 1) | subtract; + } + return {quotient, remainder}; +} + +class SecureWords { + Word* data_ = nullptr; + std::size_t bytes_; +public: + explicit SecureWords(std::size_t count) { + if (!count || count > 3 * 32768) throw std::invalid_argument("Invalid private buffer size"); + const auto page_value = sysconf(_SC_PAGESIZE); + if (page_value < 1) throw std::runtime_error("Cannot obtain private buffer page size"); + const auto page = static_cast(page_value); + bytes_ = ((count * sizeof(Word) + page - 1) / page) * page; + auto memory = mmap(nullptr, bytes_, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (memory == MAP_FAILED) throw std::bad_alloc(); + data_ = static_cast(memory); + if (mlock(data_, bytes_) != 0) { + munmap(data_, bytes_); data_ = nullptr; + throw std::runtime_error("Private BFV buffer locking failed; no unlocked fallback"); + } +#ifdef MADV_DONTDUMP + if (madvise(data_, bytes_, MADV_DONTDUMP) != 0) { + munlock(data_, bytes_); munmap(data_, bytes_); data_ = nullptr; + throw std::runtime_error("Private BFV dump exclusion failed"); + } +#endif + } + SecureWords(const SecureWords&) = delete; + SecureWords& operator=(const SecureWords&) = delete; + Word* data() { return data_; } + const Word* data() const { return data_; } + void clear() noexcept { + // Volatile writes prevent dead-store removal; wipe the full mapped region. + auto output = reinterpret_cast(data_); + if (output) for (std::size_t i = 0; i < bytes_; ++i) output[i] = 0; + } + ~SecureWords() { + if (data_) { clear(); munlock(data_, bytes_); munmap(data_, bytes_); } + } +}; + +class FixedNTT { + std::size_t n_; + std::vector reversal_; + std::vector twist_, untwist_, forward_, inverse_; + + void transform(Word* data, const std::vector& roots) const { + for (std::size_t i = 0; i < n_; ++i) + if (i < reversal_[i]) std::swap(data[i], data[reversal_[i]]); // Public permutation. + std::size_t base = 0; + for (std::size_t length = 2; length <= n_; length *= 2) { + for (std::size_t start = 0; start < n_; start += length) { + for (std::size_t j = 0; j < length / 2; ++j) { + Word u = data[start + j]; + Word v = roots[base + j].multiply(data[start + j + length / 2], p); + data[start + j] = normalize(u + v, p); + data[start + j + length / 2] = normalize(u + p - v, p); + } + } + base += length / 2; + } + } +public: + const Word p; + FixedNTT(std::size_t n, Word modulus) : n_(n), p(modulus) { + if (n < 8 || n > 32768 || (n & (n - 1)) || p < 3 || p >= (Word(1) << 60) || p % (2 * n) != 1) + throw std::invalid_argument("Invalid private transform parameters"); + mpz_class prime(p); + if (!mpz_probab_prime_p(prime.get_mpz_t(), 32)) throw std::invalid_argument("Nonprime private transform modulus"); + Word psi = 0; + for (Word candidate = 2; ; ++candidate) { + psi = xtrace_bfv::power_mod(candidate, (p - 1) / (2 * n), p); + if (xtrace_bfv::power_mod(psi, n, p) == p - 1) break; + } + const Word inv_psi = xtrace_bfv::power_mod(psi, p - 2, p); + const Word inv_n = xtrace_bfv::power_mod(n, p - 2, p); + Word power = 1, inverse_power = 1; + for (std::size_t i = 0; i < n; ++i) { + twist_.emplace_back(power, p); + untwist_.emplace_back(Word(Wide(inverse_power) * inv_n % p), p); + power = Wide(power) * psi % p; + inverse_power = Wide(inverse_power) * inv_psi % p; + std::size_t reversed = 0; + for (std::size_t at = i, bits = n; bits > 1; bits >>= 1, at >>= 1) reversed = (reversed << 1) | (at & 1); + reversal_.push_back(reversed); + } + const Word root = Wide(psi) * psi % p, inverse_root = Wide(inv_psi) * inv_psi % p; + for (std::size_t length = 2; length <= n; length *= 2) { + Word step = xtrace_bfv::power_mod(root, n / length, p); + Word inverse_step = xtrace_bfv::power_mod(inverse_root, n / length, p); + Word w = 1, iw = 1; + for (std::size_t j = 0; j < length / 2; ++j) { + forward_.emplace_back(w, p); inverse_.emplace_back(iw, p); + w = Wide(w) * step % p; iw = Wide(iw) * inverse_step % p; + } + } + } + void forward(Word* data) const { + for (std::size_t i = 0; i < n_; ++i) data[i] = twist_[i].multiply(data[i], p); + transform(data, forward_); + } + void inverse(Word* data) const { + transform(data, inverse_); + for (std::size_t i = 0; i < n_; ++i) data[i] = untwist_[i].multiply(data[i], p); + } +}; + +class PrivateDecoder { + const std::size_t n_; + const Word q_, t_; + const FixedNTT first_, second_, plain_; + const PublicMultiplier inverse_first_; + SecureWords spectra_; + std::mutex mutex_; + bool closed_ = false; + + static std::size_t checked_n(std::size_t n, Word q, Word t, Word p0, Word p1) { + if (n < 8 || n > 32768 || (n & (n - 1)) || q <= t || q >= (Word(1) << 50) || + t < 3 || t >= (Word(1) << 30) || p0 <= p1 || p1 <= (Word(1) << 59) || p0 >= (Word(1) << 60)) + throw std::invalid_argument("Private BFV requires N<=32768, t<2^30, q<2^50 and two descending 60-bit primes"); + return n; + } +public: + PrivateDecoder(std::size_t n, Word q, Word t, Word p0, Word p1, const unsigned char* secret) + : n_(checked_n(n, q, t, p0, p1)), q_(q), t_(t), first_(n, p0), second_(n, p1), plain_(n, t), + inverse_first_(xtrace_bfv::power_mod(p0 % p1, p1 - 2, p1), p1), spectra_(2 * n) { + unsigned invalid = 0; + for (std::size_t i = 0; i < n_; ++i) { + unsigned value = secret[i]; + invalid |= 1U ^ unsigned((value == 0) | (value == 1) | (value == 255)); + Word plus = value == 1, minus = value == 255; + spectra_.data()[i] = plus + (p0 - 1) * minus; + spectra_.data()[n_ + i] = plus + (p1 - 1) * minus; + } + if (invalid) throw std::invalid_argument("Private BFV key must be ternary"); + first_.forward(spectra_.data()); second_.forward(spectra_.data() + n_); + } + std::size_t degree() const { return n_; } + Word modulus() const { return q_; } + void close() { + std::lock_guard guard(mutex_); + spectra_.clear(); closed_ = true; + } + void decode(const Word* c0, const Word* c1, unsigned char* output) { + std::lock_guard guard(mutex_); + if (closed_) throw std::runtime_error("Private BFV decoder is closed"); + std::vector public_spectrum(c1, c1 + n_); + SecureWords work(3 * n_); + for (unsigned prime = 0; prime < 2; ++prime) { + const FixedNTT& transform = prime ? second_ : first_; + if (prime) std::copy(c1, c1 + n_, public_spectrum.begin()); + transform.forward(public_spectrum.data()); + Word* product = work.data() + prime * n_; + const Word* secret = spectra_.data() + prime * n_; + for (std::size_t i = 0; i < n_; ++i) + product[i] = PublicMultiplier(public_spectrum[i], transform.p).multiply(secret[i], transform.p); + transform.inverse(product); + } + const Word p0 = first_.p, p1 = second_.p; + const Wide product = Wide(p0) * p1, half = product / 2; + Word* plaintext = work.data() + 2 * n_; + for (std::size_t i = 0; i < n_; ++i) { + Word a = work.data()[i], b = work.data()[n_ + i]; + Word difference = normalize(b + p1 - normalize(a, p1), p1); + Word k = inverse_first_.multiply(difference, p1); + Wide value = a + Wide(p0) * k; + // Both operands are <2^120. The high bit of unsigned half-value + // distinguishes the negative CRT representative without a 128-bit + // relational operator that could compile to a secret branch. + Wide negative_mask = Wide(0) - ((half - value) >> 127); + SignedWide signed_value = SignedWide(value) - SignedWide(product & negative_mask); + // |c1*s| < N*q < 2^65, so adding N*q yields a nonnegative value + // below 2^67, congruent to c0+c1*s modulo q. + Wide shifted = Wide(signed_value + SignedWide(Wide(n_) * q_) + c0[i]); + Word phase = fixed_divide(shifted, q_, 67).second; + Word message = fixed_divide(2 * Wide(t_) * phase + q_, 2 * q_, 82).first; + plaintext[i] = normalize(message, t_); + } + plain_.forward(plaintext); + Word exponent = 1; + for (std::size_t i = 0; i < n_ / 2; ++i) { + const std::size_t positions[2] = {(exponent - 1) / 2, (2 * n_ - exponent - 1) / 2}; + for (unsigned row = 0; row < 2; ++row) { + Word value = plaintext[positions[row]]; + for (unsigned byte = 0; byte < 4; ++byte) + output[4 * (row * n_ / 2 + i) + byte] = static_cast(value >> (8 * byte)); + } + exponent = exponent * 3 % (2 * n_); // Public slot order. + } + } +#ifdef XTRACE_BFV_CT_TEST + // Test-only access for dynamic secret-taint checks; absent from the extension. + Word* test_secret_spectra() { return spectra_.data(); } +#endif +}; +} // namespace xtrace_bfv_private diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py new file mode 100644 index 0000000..ecf0fc5 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py @@ -0,0 +1,259 @@ +"""Approve the exact public BFV computation before allowing private decryption. + +The verifier is controlled by the data owner, separately from the untrusted +evaluator. Its Ed25519 key is NOT the evaluator's transport authentication key. +Recomputation costs another full evaluation; a receipt is not a SNARK or an +attestation of an untrusted machine. No network service is supplied here. +""" + +from __future__ import annotations + +import hashlib +import hmac +from collections.abc import Sequence + +from Crypto.Signature import eddsa + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy, hamming_noise_bound +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + BFVProtocolError, + _bytes, + _key, + protect_bfv_state, + unprotect_bfv_state, +) +from xtrace_sdk.x_vec.crypto.bfv_verified_client import BFVVerifiedClient, BFVVerifiedServer +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVServerBackend +from xtrace_sdk.x_vec.crypto.encryption.bfv_private import BFVPrivateBackend + + +_CONTEXT = b"XTRACE-BFV-PUBLIC-RECOMPUTATION-v1" +_MAGIC = b"XBFVR1" +# A versioned circuit identifier, independent of the chosen arithmetic backend. +# Bump this for any semantic or canonical-wire change, including compaction. +_CIRCUIT = hashlib.sha256( + b"XTrace BFV packed binary Hamming v1; two-row layout; exact tensor rounding; " + b"binary gadget keys; canonical v1 ciphertexts; terminal prime compaction" +).digest() +_MESSAGE_BYTES = len(_MAGIC) + 4 * 32 +_RECEIPT_BYTES = _MESSAGE_BYTES + 64 +_STATE_CONTEXT = b"XTRACE-BFV-GUARDED-PRIVATE-STATE-v1" + + +def bfv_verifier_public_key(signing_seed: bytes) -> bytes: + """Derive the Ed25519 pin on the owner; provision it through a trusted path.""" + return eddsa.import_private_key(_key(signing_seed)).public_key().export_key(format="raw") + + +def _receipt_verifier(pin: bytes) -> eddsa.EdDSASigScheme: + key = eddsa.import_public_key(pin) + # PyCryptodome accepts the encoded identity point. It is not a usable + # signing identity: cofactored Ed25519 verification would accept trivial + # signatures under a small-order key. Reject this configuration explicitly. + if (key.pointQ * 8).is_point_at_infinity(): + raise ValueError("BFV verifier pin must not be a small-order point") + return eddsa.new(key, "rfc8032", context=_CONTEXT) + + +class BFVPublicVerifier: + """Owner-controlled public-only recomputation and narrowly scoped signing. + + Supply the expected setup digest directly from the owner. Never provision + this signer to the untrusted evaluator. The only signing operation first + recomputes the prescribed result; it never signs an arbitrary caller hash. + All acceptance decisions here depend on PUBLIC ciphertexts and setup only. + """ + + def __init__( + self, + setup: bytes, + authentication_key: bytes, + signing_seed: bytes, + *, + expected_setup_digest: bytes, + policy: BFVExecutionPolicy | None = None, + backend: BFVServerBackend = "residue", + ) -> None: + self.policy = policy or bfv_review_policy() + if type(setup) is not bytes or len(setup) > self.policy.max_setup_bytes: + raise BFVProtocolError("BFV setup exceeds the local size policy") + self._setup_digest = hashlib.sha256(setup).digest() + if not hmac.compare_digest(self._setup_digest, _bytes(expected_setup_digest, 32)): + raise BFVProtocolError("BFV verifier setup is not the owner-pinned setup") + if hmac.compare_digest(_key(signing_seed), _key(authentication_key)): + raise ValueError("The verifier signing seed must differ from the evaluator key") + self._signer = eddsa.new( + eddsa.import_private_key(signing_seed), "rfc8032", context=_CONTEXT + ) + self._server = BFVVerifiedServer( + setup, authentication_key, policy=self.policy, backend=backend + ) + + def approve(self, query: bytes, response: bytes) -> bytes: + """Recompute, compare exact bytes, and sign only that approved transcript. + + A well-formed authenticated query consumes the verifier's replay ticket + even if the proposed result is wrong. Its existing server policy bounds + parsing, concurrent work and total requests. No BFV secret is available. + """ + if type(response) is not bytes or len(response) > self.policy.max_response_bytes: + raise BFVProtocolError("BFV proposed response exceeds its size limit") + expected = self._server.search(query) + if not hmac.compare_digest(response, expected): + raise BFVProtocolError("BFV proposed result differs from public recomputation") + message = ( + _MAGIC + + _CIRCUIT + + self._setup_digest + + hashlib.sha256(query).digest() + + hashlib.sha256(response).digest() + ) + return message + self._signer.sign(message) + + +class BFVGuardedClient: + """Require a pinned verifier's computation receipt BEFORE private decryption. + + This composes the prior result-checked session rather than inheriting its + unguarded finish method. There is no missing-receipt fallback. Only the + owner configures the verification key; received packets cannot select it. + Python object internals are not a boundary against malicious local code. + """ + + def __init__( + self, + client: BFVClient, + authentication_key: bytes, + verifier_public_key: bytes, + *, + policy: BFVExecutionPolicy | None = None, + private_backend: BFVPrivateBackend = "native", + ) -> None: + self._pin = _bytes(verifier_public_key, 32) + self._verifier = _receipt_verifier(self._pin) + self._session = BFVVerifiedClient( + client, + authentication_key, + policy=policy or bfv_review_policy(), + private_backend=private_backend, + ) + + @property + def policy(self) -> BFVExecutionPolicy: + return self._session.policy + + @property + def setup_digest(self) -> bytes: + if self._session._setup_digest is None: + raise BFVProtocolError("Prepare a BFV index first") + return self._session._setup_digest + + @property + def vector_ids(self) -> tuple[str, ...]: + return self._session.vector_ids + + @property + def verification_state_bytes(self) -> int: + return self._session.verification_state_bytes + + def prepare_index( + self, vectors: list[list[int]], *, vector_ids: Sequence[str] | None = None + ) -> bytes: + if not hamming_noise_bound(self.policy, len(vectors)).sufficient: + raise BFVProtocolError("BFV policy fails the conservative Hamming correctness bound") + return self._session.prepare_index(vectors, vector_ids=vector_ids) + + def begin_query(self, query: list[int]) -> bytes: + return self._session.begin_query(query) + + def finish_query(self, response: bytes, receipt: bytes) -> list[int]: + """Check bounded receipt/signature/bindings before parsing BFV ciphertexts. + + A bad receipt never reaches private decryption or plaintext checking. + Replays of a valid receipt still face the inner one-use query ticket. + Successful decryption, honest-circuit failures and local side channels + still require correctness/implementation analysis; this is not a claim + of general CCA2 security for a malleable homomorphic encryption scheme. + """ + try: + if type(response) is not bytes or len(response) > self.policy.max_response_bytes: + raise BFVProtocolError("BFV response rejected") + _bytes(receipt, _RECEIPT_BYTES) + offset = len(_MAGIC) + if ( + receipt[:offset] != _MAGIC + or receipt[offset : offset + 32] != _CIRCUIT + or receipt[offset + 32 : offset + 64] != self.setup_digest + or not hmac.compare_digest( + receipt[offset + 96 : _MESSAGE_BYTES], hashlib.sha256(response).digest() + ) + ): + raise BFVProtocolError("BFV response rejected") + # Bind the receipt to a pending, OWNER-generated request. The peer + # knows the transport MAC key and could otherwise request signatures + # for its own ciphertexts from the verifier. + request_digest = receipt[offset + 64 : offset + 96] + with self._session._lock: + if not any( + p.request_digest == request_digest for p in self._session._pending.values() + ): + raise BFVProtocolError("BFV response rejected") + self._verifier.verify(receipt[:_MESSAGE_BYTES], receipt[_MESSAGE_BYTES:]) + except (ValueError, TypeError, OverflowError): + raise BFVProtocolError("BFV response rejected before decryption") from None + return self._session.finish_query(response) + + def cancel_pending_queries(self) -> None: + self._session.cancel_pending_queries() + + def _storage_binding(self) -> bytes: + return hashlib.sha256(_STATE_CONTEXT + self.setup_digest + self._pin).digest() + + def protect_state(self, wrapping_key: bytes) -> bytes: + """Bind the saved session to this receipt gate and pinned verification key. + + The outer AEAD prevents restoring this export through the old unguarded + session API or changing the verifier pin without authenticating storage. + It does not provide durable anti-rollback protection. + """ + return protect_bfv_state( + self._session.protect_state(wrapping_key), wrapping_key, self._storage_binding() + ) + + @classmethod + def restore( + cls, + setup: bytes, + protected: bytes, + authentication_key: bytes, + wrapping_key: bytes, + verifier_public_key: bytes, + *, + policy: BFVExecutionPolicy | None = None, + private_backend: BFVPrivateBackend = "native", + ) -> BFVGuardedClient: + policy = policy or bfv_review_policy() + if type(setup) is not bytes or len(setup) > policy.max_setup_bytes: + raise BFVProtocolError("BFV setup exceeds the local size policy") + pin = _bytes(verifier_public_key, 32) + verifier = _receipt_verifier(pin) + binding = hashlib.sha256(_STATE_CONTEXT + hashlib.sha256(setup).digest() + pin).digest() + inner = unprotect_bfv_state(protected, wrapping_key, binding) + session = BFVVerifiedClient.restore( + setup, + inner, + authentication_key, + wrapping_key, + policy=policy, + private_backend=private_backend, + ) + if not hamming_noise_bound(policy, len(session.vector_ids)).sufficient: + raise BFVProtocolError("BFV policy fails the conservative Hamming correctness bound") + # Avoid importing a second native private context just to restore metadata. + result = cls.__new__(cls) + result._pin = pin + result._verifier = verifier + result._session = session + return result diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py index eee5f00..e70a3b7 100644 --- a/src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py +++ b/src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py @@ -33,6 +33,7 @@ ) from xtrace_sdk.x_vec.crypto.encryption.bfv import _coefficient_modulus, _load_key_json from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVServerBackend +from xtrace_sdk.x_vec.crypto.encryption.bfv_private import BFVPrivateBackend, BFVPrivateDecoder _FIELD = (1 << 255) - 19 @@ -168,7 +169,10 @@ def __init__( authentication_key: bytes, *, policy: BFVExecutionPolicy | None = None, + private_backend: BFVPrivateBackend = "python", ) -> None: + if private_backend not in ("python", "native"): + raise ValueError("Invalid BFV private backend") self.policy = policy or BFVExecutionPolicy() self.policy.check_client(client) keys = client._keys() @@ -180,6 +184,9 @@ def __init__( "pk": self._client.public_key, "sk": {"s": keys["sk"]["s"], "key_id": keys["sk"]["key_id"]}, } + self._private_decoder = ( + BFVPrivateDecoder(self._client) if private_backend == "native" else None + ) self._authentication_key = _key(authentication_key) self._session = secrets.token_bytes(16) self._setup_digest: bytes | None = None @@ -312,7 +319,11 @@ def finish_query(self, response: bytes) -> list[int]: ) if any(ct[3] != target for ct in ciphertexts): raise BFVProtocolError("BFV response rejected") - distances = self._client.decode_hamming_client_packed(ciphertexts, len(self._ids)) + distances = ( + self._client.decode_hamming_client_packed(ciphertexts, len(self._ids)) + if self._private_decoder is None + else self._private_decoder.decode_packed(ciphertexts, len(self._ids)) + ) if not self._summary.matches(distances, pending.expected): raise BFVProtocolError("BFV response rejected") return distances @@ -351,6 +362,7 @@ def restore( wrapping_key: bytes, *, policy: BFVExecutionPolicy | None = None, + private_backend: BFVPrivateBackend = "python", ) -> BFVVerifiedClient: """Restore only with the exact authenticated setup and a private wrapping key.""" policy = policy or BFVExecutionPolicy() @@ -375,7 +387,7 @@ def restore( client.load_stringified_keys( public, secret.decode(), max_public_key_chars=policy.max_public_key_bytes ) - result = cls(client, authentication_key, policy=policy) + result = cls(client, authentication_key, policy=policy, private_backend=private_backend) result._session, result._setup_digest, result._ids, result._started = ( session, digest, diff --git a/src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py new file mode 100644 index 0000000..8acf797 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py @@ -0,0 +1,98 @@ +"""Fixed-width native terminal decryption behind the existing BFV wire format. + +Only the native decryption/slot transform has the fixed-work design. Key +generation/import, encryption, Python plaintext handling and verification are +outside that claim. Imported native key/scratch buffers are locked and wiped; +existing Python key objects and returned plaintexts are not erased by this API. +Never expose this raw private primitive to an untrusted caller. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from typing import TYPE_CHECKING, Literal + +import gmpy2 + +from xtrace_sdk.x_vec.crypto.encryption.bfv import ( + BFV, + _coefficient_modulus, + _rns_coefficient_primes, +) + +if TYPE_CHECKING: + from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + +BFVPrivateBackend = Literal["python", "native"] + + +class BFVPrivateDecoder: + """A private native context; unavailable/unsupported configurations fail closed.""" + + def __init__(self, client: BFVClient) -> None: + try: + from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_private + except ImportError as exc: + raise RuntimeError( + "Build the BFV private CPU extension before selecting native private decoding" + ) from exc + if _bfv_private.ABI_VERSION != 1: + raise RuntimeError("Rebuild the BFV private CPU extension (ABI 1 required)") + self._native = _bfv_private + self._pk = client._pk() + self._n = client.params.poly_modulus_degree + self._q = min(int(_coefficient_modulus(client.response_modulus_bits)), int(self._pk["q"])) + self._t = client.params.plain_modulus + self._capacity, self._lanes = client.vectors_per_ciphertext, client.lanes_per_row + self._embed_len = client.embed_len + keys = client._keys() + secret = keys["sk"]["s"] + if ( + keys["sk"]["key_id"] != self._pk["key_id"] + or len(secret) != self._n + or any(c not in (-1, 0, 1) for c in secret) + ): + raise ValueError("Invalid native private key") + p0, p1 = _rns_coefficient_primes(self._n, 120) + self._context = self._native.create_decoder( + self._n, self._q, self._t, p0, p1, bytes(int(c) & 255 for c in secret) + ) + + def decode_slots(self, wire: Sequence[int | bytes]) -> list[int]: + # Framing, canonical coefficients and packing conversions are PUBLIC. + # This public parser is called only after the receipt in guarded sessions. + ct = BFV.ciphertext_from_ints(wire, self._pk) + if len(ct.components) != 2 or ct.modulus != self._q: + raise ValueError( + "Native private decoding requires the exact two-component terminal response" + ) + components = [ + int(gmpy2.pack(list(poly), 64)).to_bytes(8 * self._n, "little") + for poly in ct.components + ] + encoded = self._native.decode_slots(self._context, *components) + return [int.from_bytes(encoded[4 * i : 4 * i + 4], "little") for i in range(self._n)] + + def decode_packed( + self, ciphertexts: Sequence[Sequence[int | bytes]], vector_count: int + ) -> list[int]: + if ( + type(vector_count) is not int + or vector_count < 1 + or len(ciphertexts) != (vector_count + self._n - 1) // self._n + ): + raise ValueError("Invalid private packed response count") + result = [] + for group, wire in enumerate(ciphertexts): + slots = self.decode_slots(wire) + for position in range(min(self._n, vector_count - group * self._n)): + tile, lane = divmod(position, self._capacity) + row, column = divmod(lane, self._lanes) + result.append(slots[row * (self._n // 2) + tile * self._lanes + column]) + if any(not 0 <= d <= self._embed_len for d in result): + raise ValueError("BFV distance out of range") + return result + + def close(self) -> None: + """Wipe native key buffers; wait for an active decode and reject future ones.""" + self._native.close_decoder(self._context) diff --git a/tests/x_vec/native/run_binding_sanitizers.py b/tests/x_vec/native/run_binding_sanitizers.py index fea6be8..9a30bd4 100644 --- a/tests/x_vec/native/run_binding_sanitizers.py +++ b/tests/x_vec/native/run_binding_sanitizers.py @@ -1,6 +1,6 @@ """Load an ASan/UBSan extension in place of the regular binary, then run its tests. -Usage: python tests/x_vec/native/run_binding_sanitizers.py /tmp/bfv_sanitized.so +Usage: python tests/x_vec/native/run_binding_sanitizers.py PUBLIC_SO PRIVATE_SO Start Python with the sanitizer and C++ runtimes in LD_PRELOAD (see CI). """ @@ -10,15 +10,17 @@ import xtrace_sdk.x_vec.crypto.bfv_cpu_ext as package -name = "xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_rns" -library = Path(sys.argv[1]).resolve() -spec = importlib.util.spec_from_file_location(name, library) -assert spec is not None and spec.loader is not None -module = importlib.util.module_from_spec(spec) -sys.modules[name] = module -spec.loader.exec_module(module) -package._bfv_rns = module -assert Path(module.__file__).resolve() == library +assert len(sys.argv) == 3, "Supply both sanitized extension paths; no release fallback" +for short_name, path in zip(("_bfv_rns", "_bfv_private"), sys.argv[1:], strict=True): + name = f"xtrace_sdk.x_vec.crypto.bfv_cpu_ext.{short_name}" + library = Path(path).resolve() + spec = importlib.util.spec_from_file_location(name, library) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + setattr(package, short_name, module) + assert Path(module.__file__).resolve() == library import pytest # noqa: E402 @@ -29,6 +31,9 @@ "tests/x_vec/test_bfv_residue.py", "tests/x_vec/test_bfv_security.py", "tests/x_vec/test_bfv_verified_client.py", + "tests/x_vec/test_bfv_private.py", + "tests/x_vec/test_bfv_guarded_client.py", + "tests/x_vec/test_bfv_assurance.py", "-q", ] ) diff --git a/tests/x_vec/native/test_bfv_private.cpp b/tests/x_vec/native/test_bfv_private.cpp new file mode 100644 index 0000000..f0b7ae6 --- /dev/null +++ b/tests/x_vec/native/test_bfv_private.cpp @@ -0,0 +1,114 @@ +// Standalone private-kernel arithmetic, wiping and optional secret-taint checks. +// Compile with -DXTRACE_BFV_CTGRIND and run under Valgrind Memcheck to flag +// secret-dependent branches/addresses in the compiled decoder and NTTs. +#define XTRACE_BFV_CT_TEST +#include "private_decoder.h" +#include +#include +#ifdef XTRACE_BFV_CTGRIND +#include +#endif + +using namespace xtrace_bfv_private; + +void secret(void* data, std::size_t bytes) { +#ifdef XTRACE_BFV_CTGRIND + VALGRIND_MAKE_MEM_UNDEFINED(data, bytes); +#else + (void)data; (void)bytes; +#endif +} +void declassify(void* data, std::size_t bytes) { +#ifdef XTRACE_BFV_CTGRIND + VALGRIND_MAKE_MEM_DEFINED(data, bytes); +#else + (void)data; (void)bytes; +#endif +} + +int main(int argc, char**) { +#ifdef XTRACE_BFV_CTGRIND + assert(RUNNING_ON_VALGRIND); + if (argc > 1) { + // The CI negative control must fail under Memcheck. It demonstrates + // that the runner actually detects a branch on a marked secret. + Word value = 1; + secret(&value, sizeof(value)); + if (value) std::cout << "Deliberate secret branch in negative control\n"; + return 0; + } +#else + (void)argc; +#endif + constexpr Word q = 1125899906842597ULL, t = 65537; + constexpr Word p0 = 1152921504606830593ULL, p1 = 1152921504606748673ULL; + // Independent division/modulo oracles are PUBLIC test calculations. + for (Word modulus : {Word(17), q, 2 * q}) { + for (unsigned bits : {67U, 82U}) { + const Wide largest = (Wide(1) << bits) - 1; + for (Wide value : {Wide(0), Wide(modulus - 1), Wide(modulus), Wide(modulus + 1), largest}) { + Wide expected = value / modulus; + auto result = fixed_divide(value, modulus, bits); + assert(result.second == value % modulus); + if (expected <= UINT64_MAX) assert(result.first == expected); + } + } + } + for (Word a : {Word(0), Word(1), p0 / 2, p0 - 1}) { + for (Word b : {Word(0), Word(1), p0 / 2, p0 - 1}) { + PublicMultiplier multiplier(a, p0); + Word input = b; + secret(&input, sizeof(input)); + Word actual = multiplier.multiply(input, p0); + declassify(&actual, sizeof(actual)); + assert(actual == Wide(a) * b % p0); + } + } + // Exercise the fixed divider with tainted inputs at both actual widths. + for (unsigned bits : {67U, 82U}) { + Wide value = (Wide(1) << (bits - 1)) + 123; + Wide expected = value / q; + secret(&value, sizeof(value)); + auto result = fixed_divide(value, q, bits); + declassify(&result, sizeof(result)); + assert(result.first == expected); + } + for (std::size_t n : {std::size_t(16), std::size_t(128), std::size_t(8192)}) { + FixedNTT transform(n, p0); + std::vector input(n), expected(n); + for (std::size_t i = 0; i < n; ++i) input[i] = expected[i] = i % 3 ? p0 - 1 : 1; + secret(input.data(), n * sizeof(Word)); + transform.forward(input.data()); + transform.inverse(input.data()); + declassify(input.data(), n * sizeof(Word)); + assert(input == expected); + + std::vector key(n), output(4 * n); + std::vector c0(n), c1(n); + for (unsigned pattern = 0; pattern < 3; ++pattern) { + for (std::size_t i = 0; i < n; ++i) { + key[i] = pattern == 0 ? 1 : pattern == 1 ? 255 : (i % 3 == 0 ? 0 : i % 3 == 1 ? 1 : 255); + c0[i] = (i & 1) ? q - 1 : q / 2; + c1[i] = (i & 1) ? 0 : q - 1; + } + PrivateDecoder decoder(n, q, t, p0, p1, key.data()); + // Taint the persistent secret AFTER public parameter/import validation. + // No private output is declassified until the entire kernel returns. + secret(decoder.test_secret_spectra(), 2 * n * sizeof(Word)); + decoder.decode(c0.data(), c1.data(), output.data()); + declassify(output.data(), output.size()); + for (std::size_t i = 0; i < n; ++i) { + Word slot = 0; + for (unsigned byte = 0; byte < 4; ++byte) slot |= Word(output[4 * i + byte]) << (8 * byte); + assert(slot < t); + } + decoder.close(); + for (std::size_t i = 0; i < 2 * n; ++i) assert(decoder.test_secret_spectra()[i] == 0); + bool rejected = false; + try { decoder.decode(c0.data(), c1.data(), output.data()); } + catch (const std::runtime_error&) { rejected = true; } + assert(rejected); + } + } + std::cout << "Private BFV arithmetic, transform and wiping checks passed\n"; +} diff --git a/tests/x_vec/test_bfv_assurance.py b/tests/x_vec/test_bfv_assurance.py new file mode 100644 index 0000000..f930032 --- /dev/null +++ b/tests/x_vec/test_bfv_assurance.py @@ -0,0 +1,63 @@ +"""Checks on the conservative, public circuit bound, not a security certificate.""" + +from dataclasses import replace + +import pytest + +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy, hamming_noise_bound +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_security import BFVExecutionPolicy +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV + + +@pytest.mark.parametrize("make_policy", [BFVExecutionPolicy, bfv_review_policy]) +def test_full_supported_circuit_has_positive_worst_case_margin(make_policy): + p = make_policy() + smaller = hamming_noise_bound(p, 1) + largest = hamming_noise_bound(p, p.max_vectors) + assert smaller.sufficient and largest.sufficient + assert smaller.merged <= largest.merged + assert 2 * largest.terminal_rounding_error_numerator < largest.terminal_modulus + + +def test_reduced_q_is_not_approved_just_because_random_tests_work(): + original = BFVExecutionPolicy() + reduced = replace(original, params=replace(original.params, coeff_modulus_bits=120)) + assert not hamming_noise_bound(reduced, reduced.max_vectors).sufficient + + +def test_bound_matches_observed_phases_through_packed_merge(): + client = BFVClient(3, 32, 193, 120, 15, response_modulus_bits=40, rns_modulus=True) + policy = BFVExecutionPolicy(params=client.params, embed_len=3, response_modulus_bits=40) + query = [0, 1, 0] + vectors = ([[0, 0, 0], [0, 1, 0], [1, 0, 1], [1, 1, 1]] * 19)[:67] + index = client.encrypt_vec_packed(vectors) + responses = client.encode_hamming_server_packed( + client.encrypt_vec_one(query), index, len(vectors), compact=False + ) + bound = hamming_noise_bound(policy, len(vectors)) + assert bound.sufficient + expected = [sum(a != b for a, b in zip(query, v, strict=True)) for v in vectors] + n, t, q = client.params.poly_modulus_degree, client.params.plain_modulus, client._pk()["q"] + for group, wire in enumerate(responses): + slots = [0] * n + for i, value in enumerate(expected[group * n : (group + 1) * n]): + tile, lane = divmod(i, client.vectors_per_ciphertext) + row, col = divmod(lane, client.lanes_per_row) + slots[row * (n // 2) + tile * client.lanes_per_row + col] = value + message = BFV.batch_encode(slots, client.params) + ct = BFV.ciphertext_from_ints(wire, client._pk()) + phase = BFV._phase(ct, client._keys()) + actual = max( + abs((value - (q // t) * m + q // 2) % q - q // 2) + for value, m in zip(phase, message, strict=True) + ) + assert actual <= bound.merged + small = BFV.modulus_switch(ct, 40, client._pk()) + assert BFV.batch_decode(BFV.decrypt(small, client._keys()), client.params) == slots + + +@pytest.mark.parametrize("count", [0, -1, True, 65537]) +def test_bound_rejects_invalid_counts(count): + with pytest.raises(ValueError): + hamming_noise_bound(BFVExecutionPolicy(), count) diff --git a/tests/x_vec/test_bfv_guarded_client.py b/tests/x_vec/test_bfv_guarded_client.py new file mode 100644 index 0000000..7c2bb70 --- /dev/null +++ b/tests/x_vec/test_bfv_guarded_client.py @@ -0,0 +1,264 @@ +"""Pre-decryption provenance checks, including the previously successful oracle. + +Unlike the historical demonstration, attack cases here must NEVER call either +private decoder. The verifier is owner-controlled and receives public BFV data. +""" + +import hashlib +import secrets +from concurrent.futures import ThreadPoolExecutor + +import pytest +from Crypto.Signature import eddsa +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + BFVProtocolError, + _authenticate, + _authenticated_body, + _decode_ciphertexts, + _encode_ciphertexts, + _pack, + _unpack, +) +from xtrace_sdk.x_vec.crypto.bfv_guarded_client import ( + BFVGuardedClient, + BFVPublicVerifier, + bfv_verifier_public_key, + _CONTEXT, + _MESSAGE_BYTES, +) +from xtrace_sdk.x_vec.crypto.bfv_verified_client import BFVVerifiedClient, BFVVerifiedServer +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _coefficient_modulus +from xtrace_sdk.x_vec.crypto.encryption.bfv_private import BFVPrivateDecoder +from xtrace_sdk.x_vec.utils.xtrace_types import BFVCiphertext + + +@pytest.fixture(scope="module") +def private(): + return BFVClient(3, 16, 97, 120, 15, response_modulus_bits=40, rns_modulus=True) + + +def make_session(private, count=3, backend="native"): + if backend == "native": + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_private") + auth, signing = secrets.token_bytes(32), secrets.token_bytes(32) + p = BFVExecutionPolicy(params=private.params, embed_len=3, response_modulus_bits=40) + client = BFVGuardedClient( + private, auth, bfv_verifier_public_key(signing), policy=p, private_backend=backend + ) + vectors = ([[0, 1, 0], [1, 0, 1], [0, 0, 0]] * ((count + 2) // 3))[:count] + setup = client.prepare_index(vectors) + server = BFVVerifiedServer(setup, auth, policy=p, backend="optimized") + verifier = BFVPublicVerifier( + setup, + auth, + signing, + expected_setup_digest=client.setup_digest, + policy=p, + backend="optimized", + ) + return client, server, verifier, setup, auth, signing, vectors + + +def deny_private(monkeypatch): + monkeypatch.setattr( + BFV, "decrypt", lambda *a: pytest.fail("Forged response reached GMP decryption") + ) + monkeypatch.setattr( + BFVPrivateDecoder, + "decode_packed", + lambda *a: pytest.fail("Forged response reached native private decoding"), + ) + + +def rebind_response(query, ciphers, private, auth, policy): + setup, ticket, _ = _unpack( + _authenticated_body(query, auth, 2, policy.max_query_bytes), limit=policy.max_query_bytes + ) + body = _pack( + [setup, ticket, hashlib.sha256(query).digest(), _encode_ciphertexts(ciphers, private._pk())] + ) + return _authenticate(body, auth, 3, policy.max_response_bytes) + + +@pytest.mark.parametrize("backend", ["python", "native"]) +@pytest.mark.parametrize("count", [1, 5, 39]) +def test_correct_receipt_and_public_only_verifier(private, backend, count, monkeypatch): + client, server, verifier, _, _, _, vectors = make_session(private, count, backend) + request = client.begin_query([0, 1, 0]) + with monkeypatch.context() as scope: + deny_private(scope) + response = server.search(request) + receipt = verifier.approve(request, response) + assert verifier._server._client.keys is None + assert len(receipt) == 198 + assert client.finish_query(response, receipt) == [ + sum(a != b for a, b in zip(v, [0, 1, 0], strict=True)) for v in vectors + ] + with pytest.raises(BFVProtocolError): + client.finish_query(response, receipt) + + +def test_previous_rounding_oracle_blocked_before_any_decryption(private, monkeypatch): + client, server, verifier, _, auth, _, _ = make_session(private) + n, t = private.params.poly_modulus_degree, private.params.plain_modulus + q = _coefficient_modulus(private.response_modulus_bits) + deny_private(monkeypatch) + for index in range(n): + for offset in (0, 1): + request = client.begin_query([0, 1, 0]) + honest = server.search(request) + receipt = verifier.approve(request, honest) + c0 = (q // (2 * t) + offset,) + (mpz(0),) * (n - 1) + c1 = [mpz(0)] * n + c1[0 if index == 0 else n - index] = mpz(1) + ct = BFVCiphertext((c0, tuple(c1)), q, private._pk()["key_id"]) + forged = rebind_response( + request, [BFV.ciphertext_to_ints(ct, private._pk())], private, auth, client.policy + ) + # Even a genuine receipt for this query cannot approve other bytes. + with pytest.raises(BFVProtocolError, match="before decryption"): + client.finish_query(forged, receipt) + client.cancel_pending_queries() + + +def test_verifier_will_not_sign_a_plausible_false_result(private, monkeypatch): + client, _, verifier, _, auth, _, _ = make_session(private) + request = client.begin_query([0, 1, 0]) + q = _coefficient_modulus(private.response_modulus_bits) + zero = (mpz(0),) * private.params.poly_modulus_degree + ct = BFVCiphertext((zero, zero), q, private._pk()["key_id"]) + forged = rebind_response( + request, [BFV.ciphertext_to_ints(ct, private._pk())], private, auth, client.policy + ) + deny_private(monkeypatch) + with pytest.raises(BFVProtocolError, match="recomputation"): + verifier.approve(request, forged) + + +@pytest.mark.parametrize( + "mutation", ["missing", "truncated", "extra", "wrong_signer", "circuit", "query"] +) +def test_bad_receipts_fail_before_parsing_or_private_work(private, mutation, monkeypatch): + client, server, verifier, _, auth, _, _ = make_session(private) + query = client.begin_query([0, 1, 0]) + response = server.search(query) + receipt = verifier.approve(query, response) + if mutation == "missing": + bad = b"" + elif mutation == "truncated": + bad = receipt[:-1] + elif mutation == "extra": + bad = receipt + b"x" + elif mutation == "wrong_signer": + signer = eddsa.new(eddsa.import_private_key(auth), "rfc8032", context=_CONTEXT) + bad = receipt[:_MESSAGE_BYTES] + signer.sign(receipt[:_MESSAGE_BYTES]) + else: + position = 6 if mutation == "circuit" else 6 + 64 + bad = receipt[:position] + bytes([receipt[position] ^ 1]) + receipt[position + 1 :] + with monkeypatch.context() as scope: + deny_private(scope) + import xtrace_sdk.x_vec.crypto.bfv_security as framing + + scope.setattr( + framing.msgpack, "unpackb", lambda *a, **kw: pytest.fail("Unapproved BFV body parsed") + ) + with pytest.raises(BFVProtocolError, match="before decryption"): + client.finish_query(response, bad) + # An outsider cannot burn the legitimate request by sending bad signatures. + assert client.finish_query(response, receipt) == [0, 3, 1] + + +def test_valid_receipt_for_server_created_query_is_not_owner_authorization(private, monkeypatch): + client, server, verifier, _, auth, _, _ = make_session(private) + owned = client.begin_query([0, 1, 0]) + fields = _unpack( + _authenticated_body(owned, auth, 2, client.policy.max_query_bytes), + limit=client.policy.max_query_bytes, + ) + fields[2] = _encode_ciphertexts([private.encrypt_vec_one([1, 0, 1])], private._pk()) + attacker_query = _authenticate(_pack(fields), auth, 2, client.policy.max_query_bytes) + result = server.search(attacker_query) + receipt = verifier.approve(attacker_query, result) + deny_private(monkeypatch) + with pytest.raises(BFVProtocolError, match="before decryption"): + client.finish_query(result, receipt) + + +def test_foreign_setup_rejected_before_public_key_import(private, monkeypatch): + client, _, _, setup, auth, signing, _ = make_session(private) + monkeypatch.setattr( + BFV, "deserialize_public_key", lambda *a, **kw: pytest.fail("Unpinned setup parsed") + ) + with pytest.raises(BFVProtocolError, match="owner-pinned"): + BFVPublicVerifier( + setup + b"x", + auth, + signing, + expected_setup_digest=client.setup_digest, + policy=client.policy, + ) + with pytest.raises(ValueError, match="differ"): + BFVPublicVerifier( + setup, auth, auth, expected_setup_digest=client.setup_digest, policy=client.policy + ) + + +def test_verifier_replay_budget_and_concurrent_client_consumption(private): + client, server, verifier, _, _, _, _ = make_session(private) + request = client.begin_query([0, 1, 0]) + response = server.search(request) + receipt = verifier.approve(request, response) + with pytest.raises(BFVProtocolError, match="replayed"): + verifier.approve(request, response) + + def finish(): + try: + return client.finish_query(response, receipt) + except BFVProtocolError: + return None + + with ThreadPoolExecutor(max_workers=4) as pool: + results = list(pool.map(lambda _: finish(), range(4))) + assert results.count([0, 3, 1]) == 1 + + +def test_private_export_cannot_downgrade_gate_or_change_pin(private): + client, server, verifier, setup, auth, signing, _ = make_session(private) + wrap = secrets.token_bytes(32) + saved = client.protect_state(wrap) + with pytest.raises(BFVProtocolError): + BFVVerifiedClient.restore(setup, saved, auth, wrap, policy=client.policy) + with pytest.raises(BFVProtocolError): + BFVGuardedClient.restore( + setup, + saved, + auth, + wrap, + bfv_verifier_public_key(secrets.token_bytes(32)), + policy=client.policy, + ) + restored = BFVGuardedClient.restore( + setup, saved, auth, wrap, bfv_verifier_public_key(signing), policy=client.policy + ) + request = restored.begin_query([1, 0, 1]) + response = server.search(request) + assert restored.finish_query(response, verifier.approve(request, response)) == [3, 0, 2] + + +def test_guarded_default_requires_review_profile(private): + with pytest.raises(BFVProtocolError, match="pinned"): + BFVGuardedClient( + private, secrets.token_bytes(32), bfv_verifier_public_key(secrets.token_bytes(32)) + ) + + +def test_small_order_verifier_pin_rejected_before_private_import(private, monkeypatch): + monkeypatch.setattr( + BFVVerifiedClient, "__init__", lambda *a, **kw: pytest.fail("Invalid pin imported a key") + ) + with pytest.raises(ValueError, match="small-order"): + BFVGuardedClient(private, secrets.token_bytes(32), b"\x01" + bytes(31)) diff --git a/tests/x_vec/test_bfv_private.py b/tests/x_vec/test_bfv_private.py new file mode 100644 index 0000000..316d3a6 --- /dev/null +++ b/tests/x_vec/test_bfv_private.py @@ -0,0 +1,188 @@ +"""Independent arithmetic and ownership tests for the private terminal decoder.""" + +import random +import os +import subprocess +import sys +import textwrap +from concurrent.futures import ThreadPoolExecutor +from types import SimpleNamespace + +import pytest +from gmpy2 import mpz + +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.encryption.bfv import ( + BFV, + _coefficient_modulus, + _rns_coefficient_primes, +) +from xtrace_sdk.x_vec.crypto.encryption.bfv_private import BFVPrivateDecoder +from xtrace_sdk.x_vec.utils.xtrace_types import BFVCiphertext + + +@pytest.fixture(scope="module") +def native(): + return pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_private") + + +def pack(values): + return b"".join(int(v).to_bytes(8, "little") for v in values) + + +def slots(encoded): + return [int.from_bytes(encoded[i : i + 4], "little") for i in range(0, len(encoded), 4)] + + +def independent_slots(c0, c1, secret, q, t): + n = len(secret) + phase = [ + (c0[i] + sum(c1[j] * (secret[i - j] if i >= j else -secret[n + i - j]) for j in range(n))) + % q + for i in range(n) + ] + plaintext = [((2 * t * value + q) // (2 * q)) % t for value in phase] + candidate = 2 + while True: + psi = pow(candidate, (t - 1) // (2 * n), t) + if pow(psi, n, t) == t - 1: + break + candidate += 1 + exponents = [pow(3, i, 2 * n) for i in range(n // 2)] + exponents += [(-e) % (2 * n) for e in exponents] + return [sum(m * pow(psi, e * j, t) for j, m in enumerate(plaintext)) % t for e in exponents] + + +@pytest.mark.parametrize("n,t", [(8, 17), (16, 97), (32, 193)]) +def test_dense_chosen_components_match_schoolbook_and_direct_evaluation(native, n, t): + rng = random.Random(931 + n) + q = int(_coefficient_modulus(40)) + for _ in range(15): + secret = [rng.randrange(-1, 2) for _ in range(n)] + context = native.create_decoder( + n, q, t, *_rns_coefficient_primes(n, 120), bytes(s & 255 for s in secret) + ) + c0, c1 = ([rng.randrange(q) for _ in range(n)] for _ in range(2)) + assert slots(native.decode_slots(context, pack(c0), pack(c1))) == independent_slots( + c0, c1, secret, q, t + ) + native.close_decoder(context) + + +def test_every_plaintext_rounding_boundary(native): + n, t, q = 8, 17, int(_coefficient_modulus(40)) + secret = [1, -1, 0, 1, 0, -1, 1, 0] + context = native.create_decoder( + n, q, t, *_rns_coefficient_primes(n, 120), bytes(s & 255 for s in secret) + ) + for message in range(t): + for offset in (-2, -1, 0, 1, 2): + c0 = [(q * (2 * message + 1) // (2 * t) + offset) % q] * n + c1 = [0] * n + assert slots(native.decode_slots(context, pack(c0), pack(c1))) == independent_slots( + c0, c1, secret, q, t + ) + + +@pytest.mark.parametrize("n", [8192, 16384, 32768]) +def test_worst_supported_signed_convolution_matches_gmp(native, n): + # Known synthetic secret and extremal PUBLIC components, not generated production keys. + raw = BFVClient(3, n, 65537, 180, 30, rns_modulus=True) + q = int(_coefficient_modulus(50)) + secret = (mpz(1),) * n + raw.keys["sk"]["s"] = secret + decoder = BFVPrivateDecoder(raw) + for c0_value, c1_value in ((q - 1, q - 1), (0, q - 1), (q - 1, 0)): + ct = BFVCiphertext(((mpz(c0_value),) * n, (mpz(c1_value),) * n), q, raw._pk()["key_id"]) + expected = BFV.batch_decode(BFV.decrypt(ct, raw._keys()), raw.params) + assert decoder.decode_slots(BFV.ciphertext_to_ints(ct, raw._pk())) == expected + decoder.close() + + +@pytest.mark.parametrize("bad", [b"", b"\x02" * 16, b"\x80" * 16, b"\x00" * 17]) +def test_invalid_secret_encoding_rejected(native, bad): + with pytest.raises((ValueError, TypeError)): + native.create_decoder( + 16, int(_coefficient_modulus(40)), 97, *_rns_coefficient_primes(16, 120), bad + ) + + +@pytest.mark.parametrize( + "field,value", [(0, True), (0, 65536), (1, 1 << 50), (2, 1 << 30), (3, 1), (4, 1)] +) +def test_private_parameters_rejected_before_transform(native, field, value): + args = [16, int(_coefficient_modulus(40)), 97, *_rns_coefficient_primes(16, 120), b"\x00" * 16] + args[field] = value + with pytest.raises((ValueError, TypeError, OverflowError)): + native.create_decoder(*args) + + +def test_noncanonical_ciphertexts_and_closed_context(native): + q = int(_coefficient_modulus(40)) + context = native.create_decoder(16, q, 97, *_rns_coefficient_primes(16, 120), b"\x01" * 16) + for value in (b"", b"\x00" * 127, pack([q] + [0] * 15), b"\xff" * 128): + with pytest.raises(ValueError): + native.decode_slots(context, value, b"\x00" * 128) + native.close_decoder(context) + native.close_decoder(context) + with pytest.raises(RuntimeError, match="closed"): + native.decode_slots(context, b"\x00" * 128, b"\x00" * 128) + + +def test_close_and_decode_do_not_deadlock_or_use_wiped_key(native): + q = int(_coefficient_modulus(40)) + context = native.create_decoder(128, q, 257, *_rns_coefficient_primes(128, 120), b"\x01" * 128) + + def run(): + try: + return native.decode_slots(context, b"\x00" * 1024, b"\x00" * 1024) + except RuntimeError: + return None + + with ThreadPoolExecutor(max_workers=4) as pool: + futures = [pool.submit(run) for _ in range(10)] + pool.submit(native.close_decoder, context).result(timeout=10) + for future in futures: + assert future.result(timeout=10) in (None, b"\x00" * 512) + + +def test_missing_or_wrong_private_extension_never_falls_back(monkeypatch): + import xtrace_sdk.x_vec.crypto.bfv_cpu_ext as package + + raw = BFVClient(3, 16, 97, 120, 15, response_modulus_bits=40, rns_modulus=True) + monkeypatch.setattr(package, "_bfv_private", SimpleNamespace(ABI_VERSION=0), raising=False) + with pytest.raises(RuntimeError, match="ABI 1"): + BFVPrivateDecoder(raw) + + +def test_memory_lock_failure_has_no_unlocked_fallback(native): + if "asan" in os.environ.get("LD_PRELOAD", ""): + pytest.skip("ASan intercepts mlock; exercise the real OS limit in the release suite") + # Change the limit only in an isolated child, loading the exact tested binary. + result = subprocess.run( + [ + sys.executable, + "-c", + textwrap.dedent(""" + import importlib.util + import resource + import sys + spec = importlib.util.spec_from_file_location("_bfv_private", sys.argv[1]) + native = importlib.util.module_from_spec(spec) + spec.loader.exec_module(native) + resource.setrlimit(resource.RLIMIT_MEMLOCK, (0, 0)) + try: + native.create_decoder(16, 1125899906842597, 97, + 1152921504606830593, 1152921504606748673, bytes(16)) + except RuntimeError as error: + assert "locking failed" in str(error) + else: + raise AssertionError("Private key accepted without memory locking") + """), + native.__file__, + ], + capture_output=True, + text=True, + timeout=10, + ) + assert result.returncode == 0, result.stderr From be71b0dd4fa3693f6a6628e61f31e4b54ece9760 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Tue, 15 Sep 2026 14:07:05 -0400 Subject: [PATCH 11/12] Add AWS Nitro BFV verification and staging audit coverage --- .github/workflows/ci.yml | 12 + CHANGELOG.md | 10 +- benchmarks/bfv_nitro.py | 129 ++ benchmarks/results/.gitignore | 3 + .../results/native_bfv_nitro_audit_1024.json | 164 +++ .../results/native_bfv_nitro_local_1024.json | 164 +++ docs/index.rst | 3 + docs/research/bfv-packed-hamming.md | 2 +- docs/research/native-bfv-code-audit.md | 170 +++ docs/research/native-bfv-nitro.md | 297 ++++ docs/research/native-bfv-verification-plan.md | 363 +++++ experiments/bfv_nitro/.gitignore | 5 + experiments/bfv_nitro/Builder.Dockerfile | 6 + .../bfv_nitro/Builder.Dockerfile.dockerignore | 2 + experiments/bfv_nitro/Dockerfile | 30 + experiments/bfv_nitro/Dockerfile.dockerignore | 20 + experiments/bfv_nitro/README.md | 265 ++++ experiments/bfv_nitro/__init__.py | 1 + .../bfv_nitro/build-record-before-audit.json | 109 ++ experiments/bfv_nitro/build-record.json | 123 ++ experiments/bfv_nitro/client.py | 85 ++ experiments/bfv_nitro/nsmlib.Cargo.lock | 419 ++++++ experiments/bfv_nitro/relay.py | 66 + experiments/bfv_nitro/requirements.lock | 1203 +++++++++++++++++ experiments/bfv_nitro/service.py | 94 ++ experiments/bfv_nitro/transport.py | 127 ++ pyproject.toml | 5 + .../x_vec/crypto/bfv_attested_client.py | 518 +++++++ src/xtrace_sdk/x_vec/crypto/bfv_nitro.py | 382 ++++++ tests/x_vec/fixtures/nitro/LICENSE | 21 + tests/x_vec/fixtures/nitro/README.md | 12 + .../x_vec/fixtures/nitro/aws-2025-01-06.cose | Bin 0 -> 4781 bytes tests/x_vec/native/nitro_nsm_stub.c | 38 + tests/x_vec/native/run_binding_sanitizers.py | 8 + tests/x_vec/nitro_fixtures.py | 179 +++ tests/x_vec/test_bfv_attested_client.py | 312 +++++ tests/x_vec/test_bfv_nitro.py | 275 ++++ tests/x_vec/test_bfv_nitro_aws.py | 41 + tests/x_vec/test_bfv_nitro_transport.py | 130 ++ uv.lock | 302 ++++- 40 files changed, 6058 insertions(+), 37 deletions(-) create mode 100644 benchmarks/bfv_nitro.py create mode 100644 benchmarks/results/.gitignore create mode 100644 benchmarks/results/native_bfv_nitro_audit_1024.json create mode 100644 benchmarks/results/native_bfv_nitro_local_1024.json create mode 100644 docs/research/native-bfv-code-audit.md create mode 100644 docs/research/native-bfv-nitro.md create mode 100644 docs/research/native-bfv-verification-plan.md create mode 100644 experiments/bfv_nitro/.gitignore create mode 100644 experiments/bfv_nitro/Builder.Dockerfile create mode 100644 experiments/bfv_nitro/Builder.Dockerfile.dockerignore create mode 100644 experiments/bfv_nitro/Dockerfile create mode 100644 experiments/bfv_nitro/Dockerfile.dockerignore create mode 100644 experiments/bfv_nitro/README.md create mode 100644 experiments/bfv_nitro/__init__.py create mode 100644 experiments/bfv_nitro/build-record-before-audit.json create mode 100644 experiments/bfv_nitro/build-record.json create mode 100644 experiments/bfv_nitro/client.py create mode 100644 experiments/bfv_nitro/nsmlib.Cargo.lock create mode 100644 experiments/bfv_nitro/relay.py create mode 100644 experiments/bfv_nitro/requirements.lock create mode 100644 experiments/bfv_nitro/service.py create mode 100644 experiments/bfv_nitro/transport.py create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py create mode 100644 src/xtrace_sdk/x_vec/crypto/bfv_nitro.py create mode 100644 tests/x_vec/fixtures/nitro/LICENSE create mode 100644 tests/x_vec/fixtures/nitro/README.md create mode 100644 tests/x_vec/fixtures/nitro/aws-2025-01-06.cose create mode 100644 tests/x_vec/native/nitro_nsm_stub.c create mode 100644 tests/x_vec/nitro_fixtures.py create mode 100644 tests/x_vec/test_bfv_attested_client.py create mode 100644 tests/x_vec/test_bfv_nitro.py create mode 100644 tests/x_vec/test_bfv_nitro_aws.py create mode 100644 tests/x_vec/test_bfv_nitro_transport.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2946df3..a68c4f9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,6 +4,7 @@ on: pull_request: branches: - main + - staging # `labeled` lets reviewers opt a PR into integration / GPU coverage by # adding `run-integration` or `run-gpu` to the PR. Per-job `if:` checks # the label set, so unrelated jobs still skip. @@ -70,6 +71,17 @@ jobs: uv run python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns, _bfv_private; assert _bfv_rns.ABI_VERSION == 4; assert _bfv_private.ABI_VERSION == 1' uv run pytest tests/x_vec/test_bfv_encryption.py tests/x_vec/test_bfv_evaluator.py tests/x_vec/test_bfv_rns.py tests/x_vec/test_bfv_native.py tests/x_vec/test_bfv_residue.py tests/x_vec/test_bfv_client.py tests/x_vec/test_bfv_security.py tests/x_vec/test_bfv_verified_client.py tests/x_vec/test_bfv_private.py tests/x_vec/test_bfv_guarded_client.py tests/x_vec/test_bfv_assurance.py + - name: BFV Nitro attestation and protocol regressions (offline) + run: uv run --extra bfv-nitro pytest tests/x_vec/test_bfv_nitro.py tests/x_vec/test_bfv_attested_client.py tests/x_vec/test_bfv_nitro_transport.py tests/x_vec/test_bfv_nitro_aws.py + + - name: SEAL comparison (synthetic inputs only) + run: | + uv pip install --python .venv/bin/python -r experiments/bfv/requirements.txt + .venv/bin/python -m pytest experiments/bfv/test_packed_hamming.py tests/x_vec/test_bfv_client.py::test_optional_seal_reference + + - name: Build documentation + run: uv run --extra bfv-nitro sphinx-build -W -b html docs /tmp/xtrace-docs + - name: BFV native arithmetic sanitizers run: | g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext tests/x_vec/native/test_bfv_rns.cpp -lgmpxx -lgmp -o /tmp/test_bfv_rns diff --git a/CHANGELOG.md b/CHANGELOG.md index 29c0184..bf1e4c1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,10 @@ All notable changes to this project will be documented in this file. ### Added -- Complete C++ BFV server evaluation through `server_backend="native"`: direct packed ciphertext buffers, prepared masks, native result merging and compaction, lazy transforms, Barrett reduction and certified CRT conversion modulo q with exact fallback. Includes opt-in native phase profiling; existing BFV keys, wire formats and backends remain compatible. Rebuild the optional extension for ABI version 2. +- Optional AWS Nitro `BFVAttestedClient`/`BFVAttestedServer` sessions: signed owner requests, exact image measurement pins, fresh attestation, and enclave response receipts checked before private decryption. The enclave evaluates once; the BFV secret stays on the owner. Includes the `bfv-nitro` extra, bounded relay/service, local image build records and an opt-in real AWS test. This remains experimental and requires hardware validation and independent review before production use. +- Persistent BFV `server_backend="residue"` arithmetic with a product of 60-bit primes, fused NTTs and exact RNS tensor scaling. Existing GMP, reference and earlier native backends remain available for differential tests; selecting the product modulus requires fresh keys and a new index. +- BFV execution policies, bounded authenticated sessions, protected private exports, and an owner-controlled recomputing verifier. A separate native private decoder provides fixed-work terminal arithmetic with locked/wiped buffers; this is not an end-to-end constant-time guarantee. +- Complete C++ BFV server evaluation through `server_backend="native"`: direct packed ciphertext buffers, prepared masks, native result merging and compaction, lazy transforms, Barrett reduction and certified CRT conversion modulo q with exact fallback. Includes opt-in native phase profiling; existing BFV keys, wire formats and backends remain compatible. The current optional extensions require public ABI version 4 and private ABI version 1. - Optional native BFV `server_backend="rns"`, with independently implemented C++ RNS/NTT polynomial arithmetic, cached transformed evaluation keys, exact GMP reconstruction, and a native Hamming tile circuit. Preserves native BFV keys, ciphertext outputs and wire format. Includes native-boundary tests, sanitizer oracles, optional SEAL benchmarks and correctly tagged binary wheels. The previous GMP and reference backends remain available. - Native GMP BFV primitives in `crypto/encryption/bfv.py` and a `BFVClient` in `crypto/bfv_client.py`, with CRT batching, ciphertext multiplication/relinearization, public-key Hamming evaluation, packed distance responses, and terminal modulus compaction. Includes CPU benchmarks and tests against plaintext algebra and an optional SEAL oracle. This is an experimental local implementation; the existing Paillier clients and SEAL research example are preserved. @@ -18,6 +21,11 @@ All notable changes to this project will be documented in this file. - `xtrace_sdk.x_vec.crypto.device.resolve_device` — shared device-resolution helper used by both Paillier clients and intended for future homomorphic clients with optional GPU backends. - `test_cross_device_interop` — parametrised test covering CPU↔GPU portability for both `PaillierClient` and `PaillierLookupClient`. Verifies that contexts saved on one device load correctly on the other (hash equality), and that ciphertexts produced on either backend round-trip correctly through the other for both encryption and server-side homomorphic add. +### Security + +- BFV regression tests demonstrate secret-key recovery from malicious ciphertexts and observable post-decryption accept/reject behavior, even with private result checks. The guarded and Nitro clients require approval before decryption. Attestation authenticates approved execution under the TEE's assumptions; raw BFV remains malleable and is not given a general chosen-ciphertext security claim. +- Nitro rejects malformed certificate versions and duplicate extensions through the bounded protocol error path. Staging PRs now run quality checks, SEAL reference tests and strict documentation builds; the native sanitizer harness also supports the CFFI dependency installed by the Nitro extra. + ## [0.2.0] - 2026-04-18 ### Added diff --git a/benchmarks/bfv_nitro.py b/benchmarks/bfv_nitro.py new file mode 100644 index 0000000..765d5ff --- /dev/null +++ b/benchmarks/bfv_nitro.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Compare one Nitro-protocol evaluation with the same-ciphertext BFV baseline. + +--mode local uses a SYNTHETIC TEST CA and measures protocol overhead only. It +cannot measure Nitro isolation, NSM or EC2 overhead. --mode aws requires an +empty real enclave plus owner-trusted EIF pins; its server timer includes RPC. +Only generated synthetic plaintext is seeded; no keys or ciphertexts are logged. +""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.metadata +import json +import platform +import resource +import secrets +import statistics +import subprocess +import sys +from contextlib import ExitStack +from datetime import UTC, datetime +from pathlib import Path +from unittest.mock import patch + +from bfv_client_matrix import REPO_ROOT, make_data, time_call +sys.path.insert(0, str(REPO_ROOT)) +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy +from xtrace_sdk.x_vec.crypto.bfv_attested_client import BFVAttestedClient, BFVAttestedServer +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_verified_client import BFVVerifiedServer +from xtrace_sdk.x_vec.crypto.bfv_nitro import NitroAttestationPolicy + + +def main(): + """Record explicit evidence provenance, raw samples, sizes and exact equality.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--mode", choices=("local", "aws"), required=True) + parser.add_argument("--pins", type=Path) + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=9000) + parser.add_argument("--num-vectors", type=int, default=1024) + parser.add_argument("--repeats", type=int, default=4) + parser.add_argument("--json-out", type=Path) + args = parser.parse_args() + policy = bfv_review_policy() + if not 1 <= args.num_vectors <= policy.max_vectors or not 2 <= args.repeats <= 128: + parser.error("Use a bounded vector count and 2..128 repeats") + if args.mode == "aws" and args.pins is None: + parser.error("AWS mode requires owner-trusted --pins") + with ExitStack() as stack: + if args.mode == "local": + # This benchmark-only trust substitution is intentionally absent + # from the SDK's public API and the measured service image. + from tests.x_vec.nitro_fixtures import PCRS, SyntheticNitro + from xtrace_sdk.x_vec.crypto import bfv_nitro + issuer = SyntheticNitro() + stack.enter_context(patch.object(bfv_nitro, "_AWS_ROOT_SHA256", issuer.root_digest)) + pins = NitroAttestationPolicy(PCRS) + print("LOCAL SYNTHETIC EVIDENCE: protocol measurements, no TEE hardware", file=sys.stderr, flush=True) + else: + from experiments.bfv_nitro.client import load_measurements + pins = load_measurements(args.pins) + vectors, query, expected = make_data(args.num_vectors, 512, 1337) + setup_times = {} + setup_times["key_generation_s"], raw = time_call(lambda: BFVClient(**policy.config())) + auth = secrets.token_bytes(32) + client = BFVAttestedClient(raw, auth, pins, index_epoch=1, policy=policy) + setup_times["index_encrypt_export_s"], setup = time_call(lambda: client.prepare_index(vectors)) + setup_times["registration_encode_s"], registration = time_call(lambda: client.registration_packet(setup)) + setup_times["baseline_import_s"], baseline = time_call(lambda: BFVVerifiedServer(setup, auth, policy=policy)) + if args.mode == "local": + setup_times["protocol_import_s"], endpoint = time_call(lambda: BFVAttestedServer.from_registration(registration, issuer, policy=policy)) + else: + from experiments.bfv_nitro.transport import NitroRemote + endpoint = NitroRemote(args.host, args.port, policy=policy) + setup_times["remote_registration_s"], _ = time_call(lambda: endpoint.register(registration)) + enrollment = [] + for _ in range(args.repeats): + hello = client.begin_attestation() + issue_s, (document, proof) = time_call(lambda: endpoint.attest(hello)) + verify_s, _ = time_call(lambda: client.accept_attestation(document, proof)) + enrollment.append({"issue_or_rpc_s": issue_s, "client_verify_s": verify_s, "document_bytes": len(document), "possession_bytes": len(proof), "challenge_bytes": len(hello)}) + runs = [] + for repeat in range(args.repeats): + encrypt_s, request = time_call(lambda: client.begin_query(query)) + baseline_call = lambda: baseline.search(request[38:-64]) + protocol_call = lambda: endpoint.search(request) + # Alternate order to reduce a systematic warm-cache advantage. + if repeat % 2: + protocol_s, (response, receipt) = time_call(protocol_call) + baseline_s, reference = time_call(baseline_call) + else: + baseline_s, reference = time_call(baseline_call) + protocol_s, (response, receipt) = time_call(protocol_call) + if response != reference: + raise RuntimeError("Nitro-protocol output differs from exact BFV baseline") + finish_s, distances = time_call(lambda: client.finish_query(response, receipt)) + if distances != expected: + raise RuntimeError("Incorrect Hamming distances") + runs.append({"repeat": repeat, "query_encrypt_sign_s": encrypt_s, "baseline_local_search_s": baseline_s, "protocol_search_or_rpc_s": protocol_s, "client_finish_s": finish_s}) + print(f"trial {repeat}: baseline {baseline_s:.6f}s, protocol/RPC {protocol_s:.6f}s, client {finish_s:.6f}s", file=sys.stderr, flush=True) + summaries = { + key: {"first_s": runs[0][key], "warm_median_s": statistics.median(r[key] for r in runs[1:]), "warm_samples_s": [r[key] for r in runs[1:]]} + for key in runs[0] if key != "repeat" + } + paths = [Path(__file__), *REPO_ROOT.glob("src/xtrace_sdk/x_vec/crypto/bfv*.py"), *REPO_ROOT.glob("src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/*.so"), *REPO_ROOT.glob("experiments/bfv_nitro/*.py")] + output = { + "mode": args.mode, + "hardware_attestation": args.mode == "aws", + "measurement_notes": "All variants evaluate identical ciphertexts; exactly one evaluation produces a protocol response. Local mode substitutes a test CA and includes real certificate, owner-signature and receipt checks, but no NSM, vsock, hardware isolation or EC2. AWS protocol timers include network/relay; the reference runs locally and ratios are meaningful only on matched hardware. One cold and remaining warm trials; no tail-latency claim. No production-security certification.", + "environment": {"utc": datetime.now(UTC).isoformat(), "python": platform.python_version(), "platform": platform.platform(), "cpu": next((line.split(':', 1)[1].strip() for line in Path('/proc/cpuinfo').read_text().splitlines() if line.startswith('model name')), 'unknown'), "versions": {name: importlib.metadata.version(name) for name in ("gmpy2", "pycryptodome", "cbor2", "pyOpenSSL", "cryptography")}, "revision": subprocess.run(["git", "rev-parse", "HEAD"], cwd=REPO_ROOT, check=True, text=True, capture_output=True).stdout.strip()}, + "config": policy.config(), "vectors": len(vectors), "setup_timings": setup_times, + "enrollment": enrollment, "runs": runs, "summary": summaries, + "sizes": {"setup_bytes": len(setup), "registration_bytes": len(registration), "query_bytes": len(request), "response_bytes": len(response), "receipt_bytes": len(receipt), "query_response_receipt_bytes": len(request) + len(response) + len(receipt)}, + "whole_process_peak_rss_kib_linux": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, + "identical_ciphertexts": True, "all_distances_correct": True, + "source_sha256": {str(path.resolve().relative_to(REPO_ROOT)): hashlib.sha256(path.read_bytes()).hexdigest() for path in sorted(set(paths))}, + } + text = json.dumps(output, indent=2) + "\n" + if args.json_out: + args.json_out.parent.mkdir(parents=True, exist_ok=True) + args.json_out.write_text(text) + print(text, end="") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/results/.gitignore b/benchmarks/results/.gitignore new file mode 100644 index 0000000..ecf5b70 --- /dev/null +++ b/benchmarks/results/.gitignore @@ -0,0 +1,3 @@ +# The Nitro protocol evidence is intended to accompany its implementation. +!native_bfv_nitro_local_1024.json +!native_bfv_nitro_audit_1024.json diff --git a/benchmarks/results/native_bfv_nitro_audit_1024.json b/benchmarks/results/native_bfv_nitro_audit_1024.json new file mode 100644 index 0000000..fe4a74d --- /dev/null +++ b/benchmarks/results/native_bfv_nitro_audit_1024.json @@ -0,0 +1,164 @@ +{ + "mode": "local", + "hardware_attestation": false, + "measurement_notes": "All variants evaluate identical ciphertexts; exactly one evaluation produces a protocol response. Local mode substitutes a test CA and includes real certificate, owner-signature and receipt checks, but no NSM, vsock, hardware isolation or EC2. AWS protocol timers include network/relay; the reference runs locally and ratios are meaningful only on matched hardware. One cold and remaining warm trials; no tail-latency claim. No production-security certification.", + "environment": { + "utc": "2026-09-15T17:55:52.950792+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "cpu": "AMD Ryzen 7 5800X 8-Core Processor", + "versions": { + "gmpy2": "2.3.0", + "pycryptodome": "3.23.0", + "cbor2": "6.1.4", + "pyOpenSSL": "26.4.0", + "cryptography": "50.0.1" + }, + "revision": "46b637b604e1c2835653140610b7b2396f10b284" + }, + "config": { + "embed_len": 512, + "poly_modulus_degree": 16384, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 8.675407716073096, + "index_encrypt_export_s": 9.33344871387817, + "registration_encode_s": 0.3417240921407938, + "baseline_import_s": 3.3312883710023016, + "protocol_import_s": 3.5784192520659417 + }, + "enrollment": [ + { + "issue_or_rpc_s": 0.0022853699047118425, + "client_verify_s": 0.0075106131844222546, + "document_bytes": 1936, + "possession_bytes": 64, + "challenge_bytes": 134 + }, + { + "issue_or_rpc_s": 0.0021545880008488894, + "client_verify_s": 0.005094030173495412, + "document_bytes": 1936, + "possession_bytes": 64, + "challenge_bytes": 134 + }, + { + "issue_or_rpc_s": 0.002078167162835598, + "client_verify_s": 0.00487562594935298, + "document_bytes": 1936, + "possession_bytes": 64, + "challenge_bytes": 134 + }, + { + "issue_or_rpc_s": 0.002088826848194003, + "client_verify_s": 0.00495729804970324, + "document_bytes": 1936, + "possession_bytes": 64, + "challenge_bytes": 134 + } + ], + "runs": [ + { + "repeat": 0, + "query_encrypt_sign_s": 0.20626455708406866, + "baseline_local_search_s": 3.060150825884193, + "protocol_search_or_rpc_s": 3.06122068897821, + "client_finish_s": 0.028632056899368763 + }, + { + "repeat": 1, + "query_encrypt_sign_s": 0.20665679895319045, + "baseline_local_search_s": 2.4312804299406707, + "protocol_search_or_rpc_s": 2.436255524866283, + "client_finish_s": 0.02823221101425588 + }, + { + "repeat": 2, + "query_encrypt_sign_s": 0.20398432505317032, + "baseline_local_search_s": 2.432181579992175, + "protocol_search_or_rpc_s": 2.435416348045692, + "client_finish_s": 0.028609218075871468 + }, + { + "repeat": 3, + "query_encrypt_sign_s": 0.2063854408916086, + "baseline_local_search_s": 2.4394479650072753, + "protocol_search_or_rpc_s": 2.4388709040358663, + "client_finish_s": 0.02783559402450919 + } + ], + "summary": { + "query_encrypt_sign_s": { + "first_s": 0.20626455708406866, + "warm_median_s": 0.2063854408916086, + "warm_samples_s": [ + 0.20665679895319045, + 0.20398432505317032, + 0.2063854408916086 + ] + }, + "baseline_local_search_s": { + "first_s": 3.060150825884193, + "warm_median_s": 2.432181579992175, + "warm_samples_s": [ + 2.4312804299406707, + 2.432181579992175, + 2.4394479650072753 + ] + }, + "protocol_search_or_rpc_s": { + "first_s": 3.06122068897821, + "warm_median_s": 2.436255524866283, + "warm_samples_s": [ + 2.436255524866283, + 2.435416348045692, + 2.4388709040358663 + ] + }, + "client_finish_s": { + "first_s": 0.028632056899368763, + "warm_median_s": 0.02823221101425588, + "warm_samples_s": [ + 0.02823221101425588, + 0.028609218075871468, + 0.02783559402450919 + ] + } + }, + "sizes": { + "setup_bytes": 194804883, + "registration_bytes": 194805024, + "query_bytes": 737582, + "response_bytes": 205018, + "receipt_bytes": 198, + "query_response_receipt_bytes": 942798 + }, + "whole_process_peak_rss_kib_linux": 2320364, + "identical_ciphertexts": true, + "all_distances_correct": true, + "source_sha256": { + "benchmarks/bfv_nitro.py": "6189c69c762e11d7b3cffcb66a695e7e7306d8350f0cb4245b2c89542611aefe", + "experiments/bfv_nitro/__init__.py": "96f217ab43bd80b51d6b928cc128a7f7d6fb9c8bce318267ac157436bb49df75", + "experiments/bfv_nitro/client.py": "5a5d68dbde3f07f053e0d67b3ab22a1da1cc493cf631b540237408b0f5f7fe6a", + "experiments/bfv_nitro/relay.py": "113c99417e3ff4b9f7769084ccf8bb19225d351c2b545786980e4a7e1939e00f", + "experiments/bfv_nitro/service.py": "a4b322e678d9f49a16d44f76f53e9ea3fe2094ed213cb6f133aef21643c9e701", + "experiments/bfv_nitro/transport.py": "1fbb2767aa59209d0bdadd17b7aaafbfeb8f11a72e64cbb5f7997368a1bb36d5", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py": "ba64de6db3f76031ba9c35627a0a3790244c816937a97a3b65c98b9c699bb33d", + "src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py": "d3bed7a596fd19f834ea308fc332f0a28109a1136c7dd63026819b4cc106c7de", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.cpython-312-x86_64-linux-gnu.so": "c55be18d74e56a1d6bb82c9fa31f55cb9ecf568aecf191a26278a660502b6c72", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py": "2e684a8a7f2364a1134aa0271ee34fc6e2562367f26ae453f61a03fa80e31db3", + "src/xtrace_sdk/x_vec/crypto/bfv_nitro.py": "791107dd7daaea99b07cb9cc70116ec71fe932c0ece608a9bd65ce8a7aad6660", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "6a384b16c347d84f90e40ccc9903b1ae916ae277e6171bd668181366d56e88a0" + } +} diff --git a/benchmarks/results/native_bfv_nitro_local_1024.json b/benchmarks/results/native_bfv_nitro_local_1024.json new file mode 100644 index 0000000..e446a18 --- /dev/null +++ b/benchmarks/results/native_bfv_nitro_local_1024.json @@ -0,0 +1,164 @@ +{ + "mode": "local", + "hardware_attestation": false, + "measurement_notes": "All variants evaluate identical ciphertexts; exactly one evaluation produces a protocol response. Local mode substitutes a test CA and includes real certificate, owner-signature and receipt checks, but no NSM, vsock, hardware isolation or EC2. AWS protocol timers include network/relay; the reference runs locally and ratios are meaningful only on matched hardware. One cold and remaining warm trials; no tail-latency claim. No production-security certification.", + "environment": { + "utc": "2026-09-13T03:34:41.769069+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "cpu": "AMD Ryzen 7 5800X 8-Core Processor", + "versions": { + "gmpy2": "2.3.0", + "pycryptodome": "3.23.0", + "cbor2": "6.1.4", + "pyOpenSSL": "26.4.0", + "cryptography": "50.0.1" + }, + "revision": "46b637b604e1c2835653140610b7b2396f10b284" + }, + "config": { + "embed_len": 512, + "poly_modulus_degree": 16384, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "vectors": 1024, + "setup_timings": { + "key_generation_s": 8.720161478966475, + "index_encrypt_export_s": 9.22493529994972, + "registration_encode_s": 0.33406802802346647, + "baseline_import_s": 3.288292540004477, + "protocol_import_s": 3.5667430909816176 + }, + "enrollment": [ + { + "issue_or_rpc_s": 0.002338145859539509, + "client_verify_s": 0.00759821105748415, + "document_bytes": 1938, + "possession_bytes": 64, + "challenge_bytes": 134 + }, + { + "issue_or_rpc_s": 0.002176213078200817, + "client_verify_s": 0.0049460879527032375, + "document_bytes": 1938, + "possession_bytes": 64, + "challenge_bytes": 134 + }, + { + "issue_or_rpc_s": 0.0021135020069777966, + "client_verify_s": 0.005014879861846566, + "document_bytes": 1938, + "possession_bytes": 64, + "challenge_bytes": 134 + }, + { + "issue_or_rpc_s": 0.002089841989800334, + "client_verify_s": 0.004991688998416066, + "document_bytes": 1938, + "possession_bytes": 64, + "challenge_bytes": 134 + } + ], + "runs": [ + { + "repeat": 0, + "query_encrypt_sign_s": 0.2101371439639479, + "baseline_local_search_s": 3.065155303804204, + "protocol_search_or_rpc_s": 3.0666417400352657, + "client_finish_s": 0.028333893977105618 + }, + { + "repeat": 1, + "query_encrypt_sign_s": 0.20469022588804364, + "baseline_local_search_s": 2.4343131280038506, + "protocol_search_or_rpc_s": 2.438590140081942, + "client_finish_s": 0.028182381065562367 + }, + { + "repeat": 2, + "query_encrypt_sign_s": 0.2030612831003964, + "baseline_local_search_s": 2.439071057131514, + "protocol_search_or_rpc_s": 2.4393830781336874, + "client_finish_s": 0.028104739962145686 + }, + { + "repeat": 3, + "query_encrypt_sign_s": 0.20315458718687296, + "baseline_local_search_s": 2.4386219407897443, + "protocol_search_or_rpc_s": 2.4390968158841133, + "client_finish_s": 0.0280205390881747 + } + ], + "summary": { + "query_encrypt_sign_s": { + "first_s": 0.2101371439639479, + "warm_median_s": 0.20315458718687296, + "warm_samples_s": [ + 0.20469022588804364, + 0.2030612831003964, + 0.20315458718687296 + ] + }, + "baseline_local_search_s": { + "first_s": 3.065155303804204, + "warm_median_s": 2.4386219407897443, + "warm_samples_s": [ + 2.4343131280038506, + 2.439071057131514, + 2.4386219407897443 + ] + }, + "protocol_search_or_rpc_s": { + "first_s": 3.0666417400352657, + "warm_median_s": 2.4390968158841133, + "warm_samples_s": [ + 2.438590140081942, + 2.4393830781336874, + 2.4390968158841133 + ] + }, + "client_finish_s": { + "first_s": 0.028333893977105618, + "warm_median_s": 0.028104739962145686, + "warm_samples_s": [ + 0.028182381065562367, + 0.028104739962145686, + 0.0280205390881747 + ] + } + }, + "sizes": { + "setup_bytes": 194805676, + "registration_bytes": 194805817, + "query_bytes": 737582, + "response_bytes": 205018, + "receipt_bytes": 198, + "query_response_receipt_bytes": 942798 + }, + "whole_process_peak_rss_kib_linux": 2319872, + "identical_ciphertexts": true, + "all_distances_correct": true, + "source_sha256": { + "benchmarks/bfv_nitro.py": "6189c69c762e11d7b3cffcb66a695e7e7306d8350f0cb4245b2c89542611aefe", + "experiments/bfv_nitro/__init__.py": "96f217ab43bd80b51d6b928cc128a7f7d6fb9c8bce318267ac157436bb49df75", + "experiments/bfv_nitro/client.py": "8dcefbf02435d1ab4ffa73d860a025b16e89cba7f52d8ea70444947b5316b48e", + "experiments/bfv_nitro/relay.py": "a773e845ef680cd3f9674728d2a781d1c8cd5d80385c942439d1f3031ab10e59", + "experiments/bfv_nitro/service.py": "1354ca96c4c9ac65753dcfaf2cf86b9993a3cb35f4f9e7c3e165be56f50e3147", + "experiments/bfv_nitro/transport.py": "ede43317af73b53fa19fb95a84b147dcd9e47e3032386e67afa5daefed0e3b2a", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py": "ba64de6db3f76031ba9c35627a0a3790244c816937a97a3b65c98b9c699bb33d", + "src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py": "cbab29f860b07d44264203c6689fece4044a08b76c7564a909eacff973993dcf", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.cpython-312-x86_64-linux-gnu.so": "c55be18d74e56a1d6bb82c9fa31f55cb9ecf568aecf191a26278a660502b6c72", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py": "2e684a8a7f2364a1134aa0271ee34fc6e2562367f26ae453f61a03fa80e31db3", + "src/xtrace_sdk/x_vec/crypto/bfv_nitro.py": "94ac7be118ae20d4e6c58cc903cf26bb98e66d3928dd9d143616c5b479b5a654", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "6a384b16c347d84f90e40ccc9903b1ae916ae277e6171bd668181366d56e88a0" + } +} diff --git a/docs/index.rst b/docs/index.rst index 3694ff7..b4eac81 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -89,6 +89,9 @@ Choose your path: research/native-bfv-security research/native-bfv-verified-sessions research/native-bfv-predecryption + research/native-bfv-verification-plan + research/native-bfv-nitro + research/native-bfv-code-audit research/bfv-packed-hamming .. toctree:: diff --git a/docs/research/bfv-packed-hamming.md b/docs/research/bfv-packed-hamming.md index eecbb6a..7b4c4bb 100644 --- a/docs/research/bfv-packed-hamming.md +++ b/docs/research/bfv-packed-hamming.md @@ -257,7 +257,7 @@ backend from these estimates. ## Boundaries and next steps The implementation and commands are in the -[experiment README](../../experiments/bfv/README.md). Production source and +[experiment README](https://github.com/XTraceAI/xtrace-sdk/blob/46b637b604e1c2835653140610b7b2396f10b284/experiments/bfv/README.md). Production source and dependencies remain unchanged. The generated assets contain measurements only; no document data, API credentials, ciphertext indexes or secret keys are committed. diff --git a/docs/research/native-bfv-code-audit.md b/docs/research/native-bfv-code-audit.md new file mode 100644 index 0000000..4f1f921 --- /dev/null +++ b/docs/research/native-bfv-code-audit.md @@ -0,0 +1,170 @@ +# BFV code audit before staging review + +Date: 2026-09-15. Branch: `research/bfv-packed-hamming`. +Comparison target: `origin/staging` at `d94ddf0`. +The previous committed BFV baseline is `46b637b`; the Nitro implementation and +the fixes described here are included in the staging pull request. + +This internal audit reviewed readability, organization, correctness and test +coverage across the BFV work. It found and fixed integration and error-handling +defects. It is a bounded source review with local validation, not an independent +cryptographic assessment or approval to deploy the new protocol in production. + +## Scope and organization + +The audit followed the packed Hamming path from key generation and index +packing through public evaluation, response verification and private decoding. +It also examined native ownership and buffer boundaries, wire/import limits, +session state, Nitro evidence validation, deployment tooling, packaging and CI. +Existing Paillier/GPU work is already in the staging target and is preserved. + +| Layer | Location under `src/xtrace_sdk/x_vec/crypto/` | Responsibility | +| --- | --- | --- | +| BFV scheme | `encryption/bfv.py` | GMP reference arithmetic, keys, encryption, batching and ciphertext operations | +| Client | `bfv_client.py` | Existing client-style API, packed index/query layout and distance decoding | +| Public evaluator adapters | `encryption/bfv_evaluator.py`, `encryption/bfv_rns.py`, `encryption/bfv_native.py` | Reference, cached GMP, RNS, complete native and persistent-residue backends | +| Public native kernels | `bfv_cpu_ext/` | RNS/NTT, exact scaling, complete search and packed wire handling | +| Private arithmetic | `encryption/bfv_private.py`, `bfv_cpu_ext/private_decoder.h` | Optional fixed-work terminal decoder, separate from public evaluation | +| Protocol baselines | `bfv_security.py`, `bfv_verified_client.py`, `bfv_guarded_client.py` | Execution policy, bounded sessions, private result checks and independent recomputation receipts | +| Attested protocol | `bfv_attested_client.py`, `bfv_nitro.py` | Owner authorization, AWS evidence, enrollment and verification before private work | +| Parameter review | `bfv_assurance.py` | Fixed review profile and conservative circuit correctness bounds | + +The original Microsoft SEAL/TenSEAL example remains in `experiments/bfv/`. +Native BFV does not call SEAL. The evaluator backends remain independently +selectable so that the simpler implementations serve as differential oracles. +The Nitro service and relay stay in `experiments/bfv_nitro/`; production HTTP +endpoints and the existing execution context do not select BFV automatically. + +## Findings and fixes + +| Finding | Effect | Resolution and evidence | +| --- | --- | --- | +| CI's pull-request filter omitted `staging` | This PR would not run the quality gates intended for review | Added `staging` to the target filter; added offline Nitro tests, the preserved SEAL comparison and a strict documentation build | +| Malformed X.509 versions and duplicate extensions escaped the protocol error boundary | Two certificate parser exceptions bypassed the caller's expected rejection path; no decryption-gate bypass was observed | Catch `InvalidVersion` and `DuplicateExtension` explicitly; two tests mutate certificate DER and now assert `BFVProtocolError` | +| The Nitro dependency set changed PyCryptodome's loader during ASan tests | CFFI selected `RTLD_DEEPBIND`, causing ASan to abort before the binding suite could run | Set PyCryptodome's supported `PYCRYPTODOME_DISABLE_DEEPBIND=1` override before SDK import in the sanitizer harness only; rerun passed | +| Protocol message lengths were repeated in service and transport code | Wire changes could leave bounds inconsistent | Share named hello/query/receipt constants; format the new Nitro Python code consistently | +| A research link failed strict Sphinx builds, and the changelog lagged the implementation | Review documentation and ABI instructions were inaccurate | Fix the link, document public ABI 4/private ABI 1, and describe residue arithmetic, protocol baselines and Nitro status | +| The initial build record omitted native header hashes | The provenance record did not identify all arithmetic build inputs | Rebuild the audited image/EIF and record all included SDK source files, headers, build recipes and dependency locks; preserve the earlier record separately | + +The certificate regressions failed before the catch was corrected. The loader +problem was reproduced with the Nitro extra installed; the fix leaves ASan +interposition enabled and changes only the test process. No baseline BFV +arithmetic or Paillier implementation was changed by these audit fixes. + +## Correctness and attack coverage + +The tests compare decoded distances with plaintext Hamming distance and compare +the native backends with the reference arithmetic. They cover packed boundaries, +partial tiles, modulus compaction, malformed native buffers, ownership and +lifetime handling, import budgets and fixed execution policies. The SEAL +reference is exercised with synthetic inputs; it is not a security-equivalence +claim for the homemade parameters. + +The historical regression shows why result checks alone are insufficient: +a malicious evaluator can choose ciphertexts whose secret-dependent rounding +changes the owner's observable acceptance behavior. The attacker is allowed to +know the transport MAC key. Checks performed after private decoding therefore +do not close the oracle. This is the repository's protocol regression, not a +claim of a newly discovered flaw in Microsoft SEAL. SEAL also documents the +need to protect secret-dependent outputs at the application boundary in its +[security guidance](https://github.com/microsoft/SEAL/blob/main/SECURITY.md). + +The retained guarded baseline requires an independent owner-controlled verifier +to recompute the search and issue a receipt before decryption. The Nitro path +moves that trust to an approved enclave: it authorizes the owner's query, +performs one search and signs the bytes it produced. The owner verifies a fresh +attested receipt identity and then checks each receipt's session, pending query +and response binding before parsing or privately decoding the result. Tests +instrument both private backends to ensure forged receipts and the original +rounding probes never reach private work. + +That protection assumes the owner approves the correct measured program and +the enclave protects its signing key. Attestation authenticates execution under +the hardware/provider assumptions; it is not a mathematical execution proof or +a general chosen-ciphertext security transformation for BFV. See the +[Nitro protocol report](native-bfv-nitro.md) for the exact trust boundary, +owner signatures, replay handling and lifecycle limits. + +## Validation completed + +| Check | Result | +| --- | --- | +| Offline repository tests plus SEAL example | **433 passed, 1 skipped**, 102.91 s; only the real AWS enclave test skipped | +| ASan/UBSan Python binding suite | **211 passed, 1 skipped**, 45.50 s; ASan interposes `mlock`, so its failure-injection test runs only in the ordinary suite | +| Standalone native public/private arithmetic under ASan/UBSan | Passed | +| GCC and Clang `-O3` private-kernel secret-taint tests | Both clean; both deliberately leaking negative controls detected | +| Ruff and mypy | Passed; mypy checked all 50 SDK source files | +| Strict Sphinx HTML build | Passed with warnings treated as errors; external intersphinx fetching disabled for the local offline check | +| Wheel and source distribution | Built successfully; wheel includes BFV/Nitro code and native sources with the expected platform tag and optional extra | +| Audited evaluator container and amd64 EIF | Built; independent `describe-eif` matched measurements and passed CRC validation | + +The broad offline test command was: + +```bash +.venv/bin/python -m pytest tests experiments/bfv/test_packed_hamming.py \ + --ignore=tests/x_vec/test_meta_search.py \ + --ignore=tests/x_vec/test_hamming_search.py \ + --ignore=tests/x_vec/test_embedding.py -q +``` + +The ignored modules need live XTrace services or an external embedding model. +The environment had native extensions and the Nitro/SEAL optional dependencies +installed. Native sanitizer and compiler-taint commands are retained in +`.github/workflows/ci.yml`. Passing the narrow taint harness does not establish +that the entire Python client is constant time. + +## Audit benchmark and build provenance + +`benchmarks/results/native_bfv_nitro_audit_1024.json` records a new local run +after the SDK fixes. It uses the same ciphertexts in both paths, 1,024 vectors +of 512 bits, N=16,384 and the unchanged review profile. All distances were +correct and the ciphertext responses matched. The run alternates execution +order and separates one cold trial from three warm trials. + +| Operation | Warm median | +| --- | ---: | +| Existing public search | 2.432182 s | +| Search with owner authorization and response receipt | 2.436256 s | +| Owner query encryption and signing | 206.385 ms | +| Owner receipt verification, decryption and distance checks | 28.232 ms | + +This uses a test CA with actual certificate/signature verification. It measures +no NSM, vsock, enclave isolation or EC2 latency. The roughly 4 ms difference is +too small to establish a precise overhead from three warm samples. The useful +structural change remains eliminating the independent verifier's second full +search; the earlier guarded review-profile run took approximately 4.874 s +across server evaluation and recomputation. + +The response is 205,018 bytes. Query, response and receipt total 942,798 bytes, +excluding 26 bytes of RPC framing and lower transport layers. The 198-byte +receipt adds no BFV ciphertext expansion. Public setup is 194,804,883 bytes; +registration adds 141 bytes. Combined peak process RSS is 2,320,364 KiB and +includes the owner and two evaluator instances, so it is not enclave memory. +The initial local benchmark is preserved in `native_bfv_nitro_local_1024.json`. + +The audited `experiments/bfv_nitro/build-record.json` describes a 300,336,566-byte +version-4 amd64 EIF built with Nitro CLI 1.4.5, with SHA-256 +`25ddd36f47d70bb9a4ee1786fd868dfa5f8c52b8bef1981c5715bb172141c278`. +The earlier build record is preserved as `build-record-before-audit.json` and +has different measurements. Owners must approve the measurements of their +actual build; historical pins do not authorize future rebuilds. Generated EIFs +remain outside git. No real Nitro boot or fresh NSM evidence was obtained. + +## Remaining production work + +Before production use, run the opt-in test against a fresh real enclave and +exercise restart, renewal, failed enrollment, resource limits and concurrency +on the intended EC2 configuration. Measure cloud latency and enclave memory +separately from the local benchmark. Obtain independent protocol/native-parser +review and BFV parameter/private-side-channel assurance. The current correctness +bound does not certify a security level. + +The service supports one owner and immutable index per instance. Deployment +needs an admission policy, a durable owner epoch to resist configuration +rollback, and a reviewed process for withdrawing measurements and responding to +platform advisories. Certificate validity is enforced; automated revocation +retrieval is not implemented. The trusted image includes CPython and SDK +dependencies, and the host can still deny service or observe traffic metadata. +Verification failures must not trigger a fallback to unauthenticated private +decryption. These are explicit staging review items, not claims resolved by +the local code audit. diff --git a/docs/research/native-bfv-nitro.md b/docs/research/native-bfv-nitro.md new file mode 100644 index 0000000..9834dcb --- /dev/null +++ b/docs/research/native-bfv-nitro.md @@ -0,0 +1,297 @@ +# BFV evaluation with AWS Nitro attestation + +Date: 2026-09-12 (local), with build and benchmark artifacts recorded in UTC. +Branch: `research/bfv-packed-hamming`. Baseline: `46b637b`. +Follow-up: [2026-09-15 code audit](native-bfv-code-audit.md), including updated +measurements, an audited image rebuild and additional regression coverage. + +This implements the AWS path from the +[verification plan](native-bfv-verification-plan.md). The public BFV evaluator +runs once inside an enclave and signs its own response. The owner verifies +attestation at enrollment and a receipt before each private decryption. There +is no additional owner-side BFV evaluation. The arithmetic, packing, private +decoder and post-decryption checks reuse the existing code. + +The implementation is optional and experimental. It preserves Paillier, the +SEAL example, raw BFV and the owner-controlled recomputing verifier. Production +HTTP integration is unchanged. Local validation and an EIF build are evidence +about the implementation, not a successful hardware deployment or independent +security review. + +## Code map + +| Location | New behavior | +| --- | --- | +| `src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py` | `BFVAttestedClient`, `BFVAttestedServer`, signed owner requests, enrollment, response gate and protected state | +| `src/xtrace_sdk/x_vec/crypto/bfv_nitro.py` | `NitroAttestationPolicy`, AWS evidence verification, official NSM C adapter | +| `experiments/bfv_nitro/` | Measured service, untrusted relay, framed transport, owner demo, Docker/EIF build recipes and deployment README | +| `tests/x_vec/test_bfv_nitro.py` | Certificate, COSE, PCR, freshness and NSM ABI regressions | +| `tests/x_vec/test_bfv_attested_client.py` | Owner/session bindings, receipt gate, oracle regression and lifecycle tests | +| `tests/x_vec/test_bfv_nitro_transport.py` | Real framed-handler round trips, malicious framing and CLI measurement compatibility | +| `tests/x_vec/test_bfv_nitro_aws.py` | Explicitly enabled test against a fresh real enclave | +| `benchmarks/bfv_nitro.py` | Identical-input comparison, local protocol mode and AWS RPC mode | + +Install with `uv sync --extra bfv-nitro`; ordinary BFV/Paillier installations do +not acquire the optional attestation dependencies. The runbook is +`experiments/bfv_nitro/README.md`. Large EIF artifacts stay in its ignored +`artifacts/` directory. + +## Threat model and the decryption gate + +The attacker controls the parent server, relay and storage. It sees public BFV +keys, encrypted index/query/response bytes and the old transport MAC key. It can +alter, omit, reorder and replay messages, and observe behavior after the client +receives a result. The owner controls key generation, plaintext queries, the +expected encrypted setup, application code, approved measurements and the local +clock. The BFV secret and an independent owner request-signing secret stay on +the owner. + +The historical attack used crafted BFV ciphertexts whose secret-dependent +rounding could be detected through accept/reject behavior. Checks performed +after decryption did not stop that leakage. This protocol therefore requires +evidence of approved evaluation **before** ciphertext parsing or private +decoding. It authenticates the full packed-distance computation, not merely a +server-supplied output hash. + +Under the additional assumption that approved enclave code executes correctly +and its signing secret remains protected, a malicious parent cannot sign a +replacement ciphertext. Nor can it obtain a receipt for a chosen malicious +query: the enclave requires a separate owner signature, and the client accepts +only receipts matching its own pending queries. The existing transport MAC +does not supply either authority. + +AWS Nitro supplies isolation and attestation. The application must still +validate evidence and approve the measured program. +[AWS Nitro environment](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html). +If that program or the TEE is compromised, receipt-key misuse can reopen the +decryption oracle. This is not a chosen-ciphertext security transformation or a +mathematical proof of correctness for raw BFV. + +```text +Owner Untrusted EC2 parent Nitro enclave +BFV secret, owner signing key ciphertexts and relay public BFV setup +approved PCRs and expected epoch NSM receipt key + | signed public registration -----------------------------> | + | signed nonce + context ---------------------------------> | + | <----- AWS document binds PCRs, nonce, context, receipt key | + | <------------------------------ proof of key possession -- | + | verify evidence and store a short-lived receipt identity | + | signed encrypted query ---------------------------------> | + | evaluate ONCE | + | <----------------------- encrypted distances + signature -- | + | verify session, pending query, response hash and signature | + | decrypt, check distances, select top three locally | +``` + +## Attestation verification + +`bfv_nitro.py` uses a deliberately narrow Nitro document profile. OpenSSL checks +the certificate path; `cryptography` checks the P-384/ES384 COSE signature; +`cbor2` provides bounded decoding. The new code implements the format limits +and application policy. This integration has not received independent review. + +- The only production trust anchor is the SHA-256 fingerprint of the AWS root + DER certificate: `641a0321a3e244efe456463195d606317ed7cdcc3c1756e09893f3c68f79bb5b`. + A root sent by the peer is untrusted until that fingerprint matches. +- Validate the complete path, validity periods, critical constraints, CA + authorization and key usage. Accept no unused or duplicate certificates. + Real Nitro leaf certificates omit AKI/SKI, so OpenSSL's strict-profile flag + would incorrectly reject them; normal path validation plus explicit + application usage checks handles the actual AWS profile. +- Require protected algorithm ES384, a 96-byte signature, SHA384 PCRs and + bounded claims. Reject duplicate CBOR keys, indefinite items, semantic tags, + excessive nesting and trailing data. The optional outer COSE tag 18 is + accepted. Documents are limited to 16 KiB. +- Require exact nonzero owner-provisioned PCR0, PCR1 and PCR2. Optional other + PCRs are additional requirements. Debug measurements and peer-provided + replacement pins are never acceptable. +- Require the current owner's 32-byte nonce and context, plus a canonical + Ed25519 SubjectPublicKeyInfo. Reject small-order identities. Verify a + separate proof of possession of that receipt key. +- Check local evidence age, future clock tolerance and handshake timeout. + Receipt acceptance expires after at most five minutes and also respects + certificate expiry. Monotonic time limits the lease. + +AWS specifies the evidence format, root and PCR meanings in its +[validation guide](https://docs.aws.amazon.com/enclaves/latest/user/verify-root.html) +and [measurement guide](https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html). +Exact image pins remain required if an EIF is signed: PCR8 alone could authorize +other programs signed by the same image-signing key. + +The server calls AWS's official NSM library for entropy and evidence. The image +builds version 0.5.2 from commit +`1993eeb0620d35f5cefc50b17638b432325328f9`, with a checked archive hash and a +local Cargo lock. The SDK adapter calls its documented C ABI via `ctypes`. +[Official NSM source](https://github.com/aws/aws-nitro-enclaves-nsm-api). +There is no random-seed fallback if the NSM device fails, and no mock provider +in the deployed service entrypoint. + +## Versioned bindings and wire format + +All hashes below are SHA-256. Concatenated fields have fixed widths where +specified. Existing inner BFV records and the registration use bounded +MessagePack; epochs are unsigned 64-bit integers. Ed25519 uses separate RFC8032 +contexts for owner authorization, enrollment possession and response receipts. + +The application context commits to the shared circuit identifier, sorted JSON +execution-policy digest, **actual serialized setup** digest, 32-byte owner +public key and big-endian epoch. The setup includes the ordered IDs, count, +encrypted index, public/evaluation keys and parameters. The policy commitment +includes resource limits as well as arithmetic. Context and session hashes +have separate versioned domain prefixes in the code. + +| Message | Binding and format | +| --- | --- | +| Registration | MessagePack `[setup, transport_mac_key, owner_public_key, epoch, owner_signature]`; signature covers `XBNI01`, context and transport-key hash | +| Enrollment challenge | `XBNH01 || context[32] || nonce[32] || owner_signature[64]` (134 bytes) | +| Evidence | AWS COSE document with `nonce`, `user_data=context`, and the enclave's DER receipt public key | +| Possession proof | Ed25519 enrollment signature over the unsigned challenge and document hash (64 bytes) | +| Session ID | Domain-separated hash of context, enrollment nonce and raw receipt public key | +| Query | `XBNQ01 || session[32] || inner_query || owner_signature[64]`; signature also binds context and inner-query hash | +| Receipt | `XBNR01 || context[32] || session[32] || query_hash[32] || response_hash[32] || signature[64]` (198 bytes) | + +The server verifies the owner's registration signature before BFV key JSON +import and hashes the bytes it actually imports. Once registered, the public +setup cannot be replaced through the service. Each search verifies owner +authorization before inner parsing, performs exactly one complete evaluation, +then signs the produced response. It cannot sign an externally supplied +response, cache entry or digest. + +The client checks bounds, receipt bindings, its locally pending request digest +and the enclave signature before calling the existing inner client's +`finish_query`. The outer lock spans this gate and private work so renewal +cannot race with decryption. The inner ticket is consumed once. A valid receipt +for an unrelated or already completed request is insufficient. + +The transport adds a nine-byte header, `XBN1 || kind:u8 || length:u32be`, to each +RPC leg. It validates operation and length before reading the body, imposes an +absolute read deadline and handles one RPC per connection. A search reply is a +MessagePack `[response, receipt]`. At the measured 1,024-vector size, framing +adds 26 bytes beyond the query, response and receipt payload totals; TCP, vsock +and tunnel overhead are separate. The relay has four bounded workers. The +enclave services one request at a time and exposes no diagnostic payload logs. + +## Lifecycle and deployment limits + +This first service is single-owner and single-index. Initial registration is +self-authorized; whoever reaches an empty instance first can claim it. The +legitimate client will reject evidence for a different owner/setup. Account +admission and host availability remain deployment concerns. + +Renewing attestation replaces the active session and discards pending queries +without importing the index again. Nonces are single-use and retained in a +bounded set of 1,024 entries. Restarting the enclave discards its receipt key, +replay state and imported index. A restart requires re-upload and fresh +attestation; an index update requires a new service instance and owner epoch. + +Protected client exports include the owner signing secret and existing BFV +private state under a separate AES-GCM wrapping key. The expected setup, epoch +and both policies authenticate the envelope. They exclude enrollment and +pending queries; restore always requires fresh attestation. An owner must +retain its expected epoch in trusted durable state: restoring an entire old +owner configuration is outside what an encrypted backup can detect. + +The entrypoint fixes the larger-ring review profile and public residue backend. +No host configuration selects weaker parameters, different libraries or a GPU +outside the enclave. BFV decryption remains on the owner, and selection of the +closest results remains local. This change does not add homomorphic top-k. + +## Initial local measurements and image build + +`benchmarks/results/native_bfv_nitro_local_1024.json` records the initial local +run on a Ryzen 7 5800X with 1,024 vectors of 512 bits, N=16,384 and the unchanged +review profile. Both paths evaluated the **same ciphertexts** and returned +identical response bytes and correct plaintext distances. Measurements use one +cold trial and three warm trials, with the execution order alternated. + +| Measurement | Warm median | Interpretation | +| --- | ---: | --- | +| Existing public evaluation | 2.438622 s | One baseline search | +| New evaluation plus owner/receipt protocol | 2.439097 s | One search with signatures and session checks; synthetic attestor, no hardware | +| Owner query encryption and signing | 203.155 ms | Full query creation, not signature time alone | +| Owner receipt verification and finish | 28.105 ms | Includes parsing, private decoding and distance checks | +| Synthetic evidence verification | 4.992 ms | Test CA certificate path, COSE and possession checks; not AWS NSM latency | + +The observed search difference is about 0.5 ms, below what these few samples +can resolve reliably. There is no meaningful speedup or precise overhead claim. +The structural improvement is avoiding the second complete search: the +historical guarded baseline took approximately 4.874 s for server evaluation +plus owner recomputation. Actual enclave overhead and cloud tail latency remain +unmeasured. + +| Payload | Bytes for this run | Change from the same inner BFV payload | +| --- | ---: | --- | +| Public setup | 194,805,676 | Unchanged BFV setup format | +| Signed registration | 194,805,817 | Adds 141 bytes once | +| Query | 737,582 | Adds 102 bytes per request | +| Response | 205,018 | Identical ciphertext bytes | +| Receipt | 198 | Same length as the existing guarded receipt; distinct protocol | +| Query + response + receipt | 942,798 | Excludes 26 bytes of RPC framing and lower transport layers | + +The setup includes public/evaluation keys and encrypted index, not private +BFV state. Its JSON integer lengths vary slightly with fresh random keys. The +synthetic document was 1,938 bytes plus a 64-byte possession proof; that is not +the real AWS document size. Process peak RSS was 2,319,872 KiB and includes the +owner, two evaluators and serialization buffers. It is not enclave-only memory. + +The initial evaluator container and an **amd64 EIF** were built locally using the +official Nitro CLI 1.4.5. An independent `describe-eif` invocation reproduced +PCR0/1/2 and passed the EIF CRC check. That EIF was 300,334,519 bytes, version 4, +and has SHA-256 +`a6b354a039abdda7399768a6d9b9138f729128d47ad60830dc33b2c074bcd70c`. +It is unsigned as an image artifact; the client pins its exact measurements +and still requires a valid **AWS-signed attestation document** at runtime. +Image signing and runtime attestation are different operations. + +`experiments/bfv_nitro/build-record-before-audit.json` preserves that initial +build's PCRs and metadata. The current `build-record.json` and ignored +`artifacts/` output describe the audited rebuild documented in the +[code audit](native-bfv-code-audit.md). Its source hashes include the native +headers, and its measurements differ from the initial build. A trusted local +build record is not proof that the program has booted in an enclave. No EC2 +resources were created and no fresh NSM evidence was obtained here. + +## Validation and remaining release work + +Initial local checks: **346 BFV tests passed, one real-AWS test skipped** in +50.30 seconds. The focused Nitro set contains 77 passing tests. Ruff passed on +the SDK and all new Python files, and mypy passed on all 50 SDK source files. +The lockfile check passed offline. Sphinx built successfully with the existing +unrelated missing-document warning in `bfv-packed-hamming.md:259`. The container +and EIF build checks described above also passed. No baseline arithmetic source +or the `main` branch was modified. + +The subsequent [code audit](native-bfv-code-audit.md) fixed the documentation +warning and certificate error handling, added staging PR coverage, and passed +433 offline repository/reference tests plus 211 sanitizer-backed binding tests. +That report records the precise scope and skips for each run. + +The offline suite covers exact output equality with the existing evaluator and +an assertion that only one evaluation occurs. Both private decoding backends +are instrumented during forged-receipt and historical rounding-oracle probes +to ensure that rejection happens before private work. Additional tests cover +wrong roots and certificate constraints, wrong/zero PCRs, stale evidence, +nonce/context/key substitution, failed possession, forged owner requests, +unrelated valid receipts, replay, renewal, expiry, restore binding and +concurrent completion. Socket tests exercise the actual framed service. + +A public historical AWS document is accepted at its original valid time and +rejected at today's clock. It is not fresh evidence for this application. A +test-only C shim checks NSM argument ordering, lengths, failure statuses and +resource cleanup; the built official library loads and fails closed when the +real NSM device is absent. Synthetic roots exist only in test/benchmark code; +the production verifier rejects them by default. + +The next release work is actual Nitro EC2 end-to-end execution and lifecycle +testing, matched cloud performance and isolated memory measurements, independent +protocol review, native/parser attack-surface review, and the existing BFV +parameter and client private-key side-channel assurance. The measured software +includes CPython, the SDK import tree and all image dependencies; shrinking that +trusted footprint is useful follow-up work. + +Certificate expiry is enforced, but this verifier does not fetch CRLs or provide +an automated platform-advisory/revocation service. Owners need a reviewed process +for withdrawing measurements and updating trusted builds. Attestation establishes +program identity under AWS's assumptions; it does not remove these operational +and implementation obligations. Host denial of service and metadata observation +remain possible. Cryptographic execution proofs remain a separate future track. diff --git a/docs/research/native-bfv-verification-plan.md b/docs/research/native-bfv-verification-plan.md new file mode 100644 index 0000000..060334a --- /dev/null +++ b/docs/research/native-bfv-verification-plan.md @@ -0,0 +1,363 @@ +# BFV server verification: TEE first, cryptographic proofs later + +Date: 2026-09-12. Branch: `research/bfv-packed-hamming`. +Status: original research and proposed design, retained as the planning baseline. +The subsequent [AWS Nitro implementation](native-bfv-nitro.md) follows the user's +AWS deployment preference. Its local tests and EIF build do not yet establish +real EC2 execution. Cryptographic proofs remain future work. + +## Recommendation and preserved baseline + +Build the next experiment around **one BFV evaluation inside an attested TEE**. +The client establishes that an approved evaluator controls a response-signing +key, then checks its signed response before private decryption. This addresses +the duplicate computation in the current design while reusing our C++ RNS +arithmetic. Hardware attestation is an accepted trust assumption for this +project; preserving native performance is the main reason to prioritize it. +The performance advantage is a design expectation, not a measured TEE result. + +Keep a second research track for cryptographic proofs, beginning with a small +experiment using an existing proof system for program execution. Consider +specialized proofs of BFV arithmetic only after measuring that experiment. +This order limits the amount of new cryptography we would have to design. + +The code baseline is commit `46b637b`, described in +[the pre-decryption approval report](native-bfv-predecryption.md). +`BFVGuardedClient`, `BFVPublicVerifier`, their tests and benchmarks remain the +reference implementation. This document records the original planning-only step. +There is no change to Paillier, the SEAL example, BFV arithmetic or production +integration, and no cloud deployment is performed. + +## What the client must establish + +The threat model includes a malicious evaluator and relay that can replace, +omit, reorder and replay messages, knows the evaluator's transport MAC key, +and can observe application behavior after a response. The owner controls +key generation, the encrypted setup, query creation and the client software. +The BFV secret key stays on the owner. + +The historical attack submitted crafted ciphertexts that passed public format +checks but exposed secret-dependent rounding. Acceptance after decryption +revealed information about the secret key. Consequently, a checksum of the +plaintext, valid serialization, or a MAC made with a key known to the evaluator +cannot supply the needed gate. +The existing regression in `tests/x_vec/test_bfv_verified_client.py` documents +this behavior; the guarded-client regression checks rejection before either +private decoder is invoked. + +The proposed acceptance rule is: + +```text +response = approved deterministic BFV evaluation(owner setup, owner query) +``` + +"Owner setup" includes the parameters, public and evaluation keys, complete +encrypted index, ordered vector IDs and count. The protocol must also bind the +index version, operation, output encoding and the client's pending request. +Correct arithmetic on a different index, a server-created query, or a subset of +the rows is insufficient. + +This verifies the computation over the encrypted data the owner committed to. +It does not establish that those original embeddings were accurate. The existing +circuit returns packed encrypted distances; the client still decrypts and ranks +them. This proposal does not add encrypted top-k selection. + +## Comparing the options + +The implementation-risk judgments below are specific to this repository and +are planning assessments, not certifications or benchmark results. + +| Approach | Work on the server side | Client work before decryption | Main trust assumption | Implementation assessment | +| --- | --- | --- | --- | --- | +| Current owner verifier | One search plus another complete search | Hashes, bindings and signature | Owner verifier and its signing key | Already implemented; useful reference | +| Attested evaluator | One search inside a TEE, plus attestation and signing | Attestation at session establishment; hashes and signature per response | Approved code, TEE isolation and attestation infrastructure | Best first experiment; preserves native arithmetic | +| Proof of program execution, such as a zkVM | Execute the evaluator in a supported guest and generate a proof | Verify proof, program identity and input/output bindings | Proof-system assumptions, sound implementation and correct guest program | Less bespoke cryptography; substantial porting and potentially expensive proving | +| Specialized arithmetic proof | Evaluate BFV and prove its exact arithmetic relation | Verify proof and bindings | Proof-system assumptions and correct BFV constraints | Potentially better proving cost; highest specialist review burden | +| Two independent matching evaluators | Two complete searches, potentially concurrent | Authenticate both results and compare exact bytes | At least one honest evaluator, with independent authenticated identities | Simpler cryptography, but retains duplication and adds a non-collusion assumption | + +Two matching replicas would need genuinely independent response identities: +the shared transport MAC in the existing session protocol is not enough. Running +both replicas under the same compromised operator does not meet that assumption. + +Encrypted canaries, sampled outputs and hidden checksums are not proposed as the +production gate. Research has broken particular lightweight schemes that embed +verification secrets inside HE ciphertexts; that does not prove every possible +lightweight verifier insecure, but it makes a new construction a research task. +Our private post-decryption summary already has a separate, demonstrated oracle +problem. [Cheon and Jang, ASIACRYPT 2025](https://arxiv.org/abs/2502.12628). + +## Proposed TEE design + +The protected evaluator replaces the full recomputing verifier's role. The host +stores ciphertexts and relays traffic; there is no second full search. + +```text +Owner client Untrusted host Attested evaluator +BFV secret key Encrypted storage Public BFV setup +Approved-code policy Network relay Ephemeral signing key + | | | + |---- fresh challenge and expected session context ---->| + |<--- attestation binding approved code and signing key --| + | verify evidence locally | | + |---- owner-authorized encrypted setup and query ------->| + | | compute once + |<-------------- encrypted response + signed receipt ----| + | verify receipt and pending request | + | decrypt, check distances and rank locally | +``` + +### Establish the evaluator's identity + +1. Package the public evaluator, native extension and dependencies in a fixed + image. Provision approved measurements and protocol versions through an + owner-trusted release/configuration path. A measurement advertised only by + the host is not an independent reference value. +2. Generate an ephemeral signing key inside the protected runtime. Attestation + must bind that public key to a fresh client challenge and an unambiguous, + versioned session context. Include the circuit, expected setup digest and + index epoch in that binding, directly or through a domain-separated digest. +3. Validate the attestation signature and trust chain, approved measurements, + debug state, applicable platform security status, freshness and exact + application-data binding. Use a pinned, reviewed dependency for the chosen + platform's evidence format. Application policy remains our responsibility. +4. Require proof of possession of the bound signing key before accepting the + session. Authorize the owner and setup through the established protocol; + a transport MAC shared with the host cannot establish owner authorization. + +Use existing cryptographic and attestation libraries. Do not turn the adjacent +toy key exchange into a new production handshake. An existing attested TLS +stack could establish a channel, but our first design should retain explicit +response receipts to fit the existing pre-decryption gate and allow the relay +to remain outside the trusted runtime. + +### Compute and authenticate each response + +The protected service imports and hashes the actual owner-authorized setup, +checks the supported circuit and admission policy, then executes the complete +public BFV search internally. It signs only the response it produced, binding +the session, circuit, setup/epoch, exact request and exact response bytes. +Expose a `search` operation, not a general-purpose `sign` or `approve_hash` API. + +The client bounds the received bytes and verifies the evidence bindings and +signature before BFV parsing or private work. It then consumes its own pending +request once and follows the current private decoding and distance checks. +Preserve the existing rule that a valid receipt for a server-created request +does not authorize decryption on the owner. + +Attestation can be amortized across a bounded session while ordinary signatures +authenticate individual responses. Require a new challenge and attestation on +session renewal, key rotation, restart or a relevant code/policy change. Start +the experiment with keys that are not persisted. Saved client state must not +silently restore an expired session or bypass attestation; owner-maintained +index epochs and replay policy need explicit persistence rules. + +The current receipt is 198 bytes. A future receipt needs its own protocol +version and may be larger to bind session and index lifecycle information. +Attestation evidence and certificate/verification collateral are additional +setup traffic; their actual sizes and cache behavior must be measured. + +### Preserve the protection boundary + +All operations influencing the signed result must be protected or soundly +verified there, including setup import, preprocessing caches, RNS arithmetic, +compaction and serialization. Hashing an unchecked host-supplied result is not +verification. Offloading to an ordinary host GPU would reopen this boundary; +start with the current CPU implementation. + +The trusted software includes more than the BFV kernel: the guest OS/runtime, +parsers, native extension, dependencies and any privileged management path can +affect the signing key. Pin executable dependencies and disable host-controlled +code loading, mutable image tags, debug access and unmeasured configuration +that can change computation. A memory-corruption bug in the protected service +could expose its signing key, so the existing parser limits and native checks +remain relevant. + +Attestation establishes identity under the platform's assumptions; review and +testing establish confidence in what that identified program does. It is not +a mathematical correctness proof. If the TEE or signing-key custody fails, +the response gate can fail and the original oracle can return, despite keeping +the BFV secret key on the client. Availability, traffic analysis, client-side +private-key handling and independent BFV parameter/protocol assurance remain +separate concerns. + +## Platform candidates and verification dependencies + +**Phala/dstack on Intel TDX** is worth evaluating because the adjacent +`TEE-Key-Exchange-Toy` repository already targets eventual Phala integration. +Its current `ToyAttestationProvider` signs local claims; it supplies no hardware +isolation or genuine TDX quote verification. Reuse its lessons and test cases, +not its trust root. + +Phala documents application verification using quote report data, measured +application configuration and replay of the RTMR event log. Our policy must +compare these against owner-approved values and immutable container digests, +not merely compare two values received from the host. +[Phala application verification](https://docs.phala.com/phala-cloud/attestation/verify-your-application). + +Evaluate `dstack-verifier` and `dcap-qvl` as dependency candidates, checking +pinned versions, review coverage and failure behavior. Quote validity alone +does not approve our application. The dstack documentation separates automatic +platform checks from additional policy, KMS and network identity checks. For +the pilot, generate the response-signing key inside the runtime rather than +deriving it from a platform KMS; review the platform services that still affect +execution. Local attestation verification avoids adding a remote verification +service as another trusted party. +[Phala platform verification](https://docs.phala.com/phala-cloud/attestation/verify-the-platform), +[dcap-qvl source](https://github.com/Phala-Network/dcap-qvl). + +**AWS Nitro Enclaves** is the other concrete candidate if deployment will use +compatible EC2 instances. It offers an isolated Linux environment with a local +socket to the parent and no direct external networking or persistent storage. +That maps to a small relay plus the existing Python/C++ public evaluator. +[AWS Nitro overview](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html). + +Nitro evidence carries signed measurements and optional nonce, public-key and +application-data fields. Validate its COSE signature and certificate chain with +the provider's trust root, then apply our session and code policy. +[AWS attestation validation](https://docs.aws.amazon.com/enclaves/latest/user/verify-root.html). +Pin approved image measurements and reject debug evidence; trusting an image +signer's certificate alone permits other images signed by that key. +[AWS measurement documentation](https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html). + +At the time of this plan, provider selection was still open. The SDK's AWS dependencies do not establish +where this evaluator will run. Choose one provider for the first real hardware +experiment based on hosting fit and an acceptable verification dependency; +avoid implementing multiple evidence formats initially. + +## Performance experiment and acceptance criteria + +Use the larger-ring review profile and identical ciphertexts for comparisons. +The committed `benchmarks/results/native_bfv_guarded_review_1024.json` records +these results for 1,024 vectors of 512 bits, N=16,384, on a Ryzen 7 5800X: + +| Existing measurement | Value | Meaning for the proposed experiment | +| --- | ---: | --- | +| Warm public search | 2.439 s | One complete evaluation to move inside the TEE | +| Additional owner-verifier approval | 2.435 s | The duplicated evaluation to remove | +| Warm sequential search plus approval | 4.874 s | Current combined compute baseline | +| Guarded client finish | 28.425 ms | Includes receipt handling, parsing, native decryption and private checks | +| Query / response | 737,480 / 205,018 bytes | BFV payload sizes to preserve for this profile | +| Existing receipt | 198 bytes | Excludes future attestation/session overhead | +| Setup per evaluator | 194,805,166 bytes | Public/evaluation keys plus encrypted index and envelope | + +These are one cold trial and three warm trials, without network traffic. They +are not cloud measurements or evidence about tail latency. The historical +whole-process peak memory includes two evaluators and the client; it cannot +be used as the isolated enclave's RAM requirement. + +Measure unprotected and protected execution on the **same cloud CPU**, with +matched compiler flags, vCPU allocation, workload, parameters and caches. +Record separately: + +- Cold startup, setup upload/import, preprocessing and isolated evaluator RSS. +- Warm search time, throughput and p50/p95 latency over enough repetitions; + include concurrency and realistic index sizes after the single-worker case. +- First-session attestation verification, collateral fetching and bytes; + subsequent sessions with cached collateral and per-response receipt cost. +- Host relay/network time, complete client finish and all communication legs. +- Index replacement, session renewal, restart and failure/retry behavior. + +A provisional target is warm TEE search latency within 20% of the matched +unprotected evaluator, with exactly one full search per query. This is an +engineering target to test, not a prediction or a user-specified service level. +Report any miss by component before changing BFV arithmetic. Do not reduce +security parameters or omit checks to improve the comparison. + +## Cryptographic proof track + +### Start with an existing system for proving program execution + +A zkVM can prove execution of a supported guest program. The client must verify +the expected program identity and bind its public outputs to the requested +computation; a cryptographically valid receipt for a different program is +irrelevant. RISC Zero's receipt model explicitly separates the public journal +from the proof and verifies against a trusted image ID. +[RISC Zero receipts](https://dev.risczero.com/api/zkvm/receipts). + +Shortlist RISC Zero and SP1, then select one based on guest compatibility, +verification dependencies, actual security assumptions and profiling. Neither +should be assumed to run the current CPython extension and host GMP build +unchanged. Inspect the exact release, audits, advisories, soundness parameters +and any proof compression or trusted-setup requirements. For example, SP1 +documents different assumptions for its core proof and its wrapping proofs; +"uses a SNARK" does not specify a uniform security level. +[SP1 security model](https://docs.succinct.xyz/docs/sp1/security/security-model). + +The proof statement must bind the circuit/program, parameters, owner setup +commitment and index epoch, exact owner request and canonical response digest. +The proven execution must check all data against that commitment and process +the complete ordered index. A Merkle root alone neither verifies evaluation +nor establishes that no rows were omitted. + +The difficult relation is not just polynomial multiplication. Our implementation +also depends on canonical integer lifts, exact tensor scaling and rounding, +binary gadget decomposition and relinearization, rotations, masks, merging, +terminal modulus switching and wire encoding. RNS congruences alone do not +prove the necessary integer ranges and rounding. Host-provided arithmetic hints +or cached values must be checked in the proven computation. + +Start with small correctness fixtures, then prove **one full N=16,384 tile** +(32 vectors at 512 dimensions), including compaction. Next cover multiple tiles +and the merge before attempting the full 1,024-vector search. Tiny polynomial +degrees can test integration but cannot establish production-profile proving +cost. Include hashing/import of the large setup in the accounting; reusable +authenticated preprocessing would be a separate design step. + +Measure proving time, peak memory, verification time, proof bytes and the cost +of initial setup separately. Compare total query-to-verified-result latency, +not only the prover's final compression step. Zero knowledge is not itself +required to hide the public BFV ciphertexts from their owner; avoid adding a +proof wrapper solely for an unrelated blockchain use case. + +### Specialize only if measurements justify it + +A BFV-specific proof could avoid much of a general VM's execution overhead. +That is a plausible optimization direction, not a demonstrated result here. +It needs a reviewed integer/RNS constraint design, especially range, carry and +rounding conditions. The ringSNARK project implements ring-oriented proof +research, but explicitly labels itself a proof of concept unsuitable for +critical or production use. Use it as a reference, not a ready production +dependency for this homemade BFV implementation. +[ringSNARK source and status](https://github.com/zkFHE/ringSNARK). + +Finally, proving this relation does not automatically give raw BFV a general +chosen-ciphertext security theorem. Manulis and Nguyen's verifiable-FHE +transformation combines ciphertext-integrity machinery with a SNARK. Its proof +does not transfer just by attaching a generic execution proof to our current +API. Review the actual owner-generated-input protocol, adaptive responses and +decryption behavior with a cryptographer before making a comparable claim. +[Manulis and Nguyen, EUROCRYPT 2024](https://eprint.iacr.org/2024/202). + +## Proposed work order and repository layout + +1. **Specify the TEE protocol.** Write the session/receipt byte formats, trusted + measurement policy, key lifecycle, owner authorization, index epochs and + failure state machine. Select one platform and verification dependency. + Review the specification before treating a prototype as a security boundary. +2. **Prototype the public evaluator on real hardware.** Keep deployment files + and the relay/service harness under `experiments/bfv_attestation/`. Reuse the + current public evaluator and compare exact output bytes with the baseline. + Local simulated evidence is for tests only and must be rejected by the real + client configuration. +3. **Add an explicit opt-in client and benchmarks.** A future + `src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py` would keep the existing + client style and mandatory gate. Put provider verification behind a narrow + optional adapter; add `tests/x_vec/test_bfv_attested_client.py` and + `benchmarks/bfv_attested.py`. Preserve the baseline APIs and dependencies. +4. **Test the boundary and obtain independent review.** Reuse the existing + oracle regression with both private decoders instrumented to fail if called. + Add rejected evidence for wrong roots, measurements, debug state, freshness, + session/key binding and security policy; mutate signed responses and setup + IDs; test server-created queries, omitted/reordered rows, replay, concurrent + responses, restarts, state rollback and downgrade attempts. Include hostile + relay integration tests and bounded parser/fuzz/sanitizer coverage inside the + protected service. No test suite substitutes for a protocol/platform review. +5. **Run the proof feasibility experiment later.** Put the guest, verifier and + fixtures in `experiments/bfv_proofs/`. Decide whether to extend the generic + proof or investigate specialized arithmetic from measured full-profile + costs. Introduce a separate proof-required client mode only after review; + proof failures must never trigger automatic acceptance through another mode. + +For both future modes, verification happens before private decryption, expected +inputs come from owner state, and the selected trust policy is explicit. These +are the common boundaries to preserve while changing how evidence is produced. diff --git a/experiments/bfv_nitro/.gitignore b/experiments/bfv_nitro/.gitignore new file mode 100644 index 0000000..b65961a --- /dev/null +++ b/experiments/bfv_nitro/.gitignore @@ -0,0 +1,5 @@ +!requirements.lock +!nsmlib.Cargo.lock +!build-record.json +!build-record-before-audit.json +artifacts/ diff --git a/experiments/bfv_nitro/Builder.Dockerfile b/experiments/bfv_nitro/Builder.Dockerfile new file mode 100644 index 0000000..bf3daee --- /dev/null +++ b/experiments/bfv_nitro/Builder.Dockerfile @@ -0,0 +1,6 @@ +# Local Linux EIF builder; no AWS credentials, NSM device or cloud access needed. +# The Docker socket is needed only to export the selected local service image. +FROM amazonlinux:2023@sha256:181f98c48832fe926f8ca3b6ffeafcce128e96e77b93d08fbe9a9bc9403ce284 +RUN dnf install -y aws-nitro-enclaves-cli-1.4.5-0.amzn2023 aws-nitro-enclaves-cli-devel-1.4.5-0.amzn2023 && dnf clean all +RUN nitro-cli --version +ENTRYPOINT ["nitro-cli"] diff --git a/experiments/bfv_nitro/Builder.Dockerfile.dockerignore b/experiments/bfv_nitro/Builder.Dockerfile.dockerignore new file mode 100644 index 0000000..37e508a --- /dev/null +++ b/experiments/bfv_nitro/Builder.Dockerfile.dockerignore @@ -0,0 +1,2 @@ +# The builder needs no repository files in its build context. +** diff --git a/experiments/bfv_nitro/Dockerfile b/experiments/bfv_nitro/Dockerfile new file mode 100644 index 0000000..626c9f4 --- /dev/null +++ b/experiments/bfv_nitro/Dockerfile @@ -0,0 +1,30 @@ +# Measured public BFV evaluator. Build from the repository root using this file. +# Immutable base images and a hash-checked AWS source archive bind build inputs; +# review/pin the resulting EIF PCRs. This is not a reproducible-build proof. +FROM rust:1.94-slim-bookworm@sha256:cf9dd0ec73e75f827fe59123fff9dc65af1a1c8363c3c31ee8d7f8ad0b6a5fb2 AS nsm +ADD --checksum=sha256:f78f9683ce02c857b944c84865ca2808f90f67a52605f120ac234095b1d03c35 https://codeload.github.com/aws/aws-nitro-enclaves-nsm-api/tar.gz/1993eeb0620d35f5cefc50b17638b432325328f9 /tmp/nsm.tar.gz +WORKDIR /build/nsm +RUN tar -xzf /tmp/nsm.tar.gz --strip-components=1 +COPY experiments/bfv_nitro/nsmlib.Cargo.lock ./Cargo.lock +RUN cargo build --release --locked -p nsm-lib + +FROM python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254 AS arithmetic +RUN apt-get update && apt-get install -y --no-install-recommends g++ make libgmp-dev && rm -rf /var/lib/apt/lists/* +COPY src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext /build/bfv_cpu_ext +RUN make -C /build/bfv_cpu_ext "_bfv_rns$(python -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))')" CXXFLAGS='-O3 -fstack-protector-strong -D_FORTIFY_SOURCE=3' LDFLAGS='-Wl,-z,relro,-z,now,-z,noexecstack' + +FROM python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254 +RUN apt-get update && apt-get install -y --no-install-recommends libgmp10 libgmpxx4ldbl && rm -rf /var/lib/apt/lists/* +WORKDIR /app +COPY experiments/bfv_nitro/requirements.lock /tmp/requirements.lock +RUN pip install --no-cache-dir --require-hashes -r /tmp/requirements.lock +COPY src/xtrace_sdk /app/src/xtrace_sdk +COPY experiments/bfv_nitro/__init__.py experiments/bfv_nitro/transport.py experiments/bfv_nitro/service.py /app/experiments/bfv_nitro/ +COPY --from=arithmetic /build/bfv_cpu_ext/_bfv_rns*.so /app/src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/ +COPY --from=nsm /build/nsm/target/release/libnsm.so /usr/local/lib/libnsm.so +COPY --from=nsm /build/nsm/LICENSE /build/nsm/NOTICE /usr/local/share/licenses/aws-nsm/ +COPY LICENSE /usr/local/share/licenses/xtrace-sdk/LICENSE +ENV PYTHONPATH=/app/src PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 +# Check imports/ABI during the build; the NSM device is available only at runtime. +RUN python -c 'from xtrace_sdk.x_vec.crypto.bfv_cpu_ext import _bfv_rns; assert _bfv_rns.ABI_VERSION == 4; from experiments.bfv_nitro.service import EnclaveApplication' +ENTRYPOINT ["python", "-m", "experiments.bfv_nitro.service"] diff --git a/experiments/bfv_nitro/Dockerfile.dockerignore b/experiments/bfv_nitro/Dockerfile.dockerignore new file mode 100644 index 0000000..0e95f7c --- /dev/null +++ b/experiments/bfv_nitro/Dockerfile.dockerignore @@ -0,0 +1,20 @@ +# Only measured public source/build inputs enter Docker's context. Never copy +# owner keys, local indices, .env files, the virtualenv, git metadata or host .so. +** +!LICENSE +!src/ +!src/xtrace_sdk/ +!src/xtrace_sdk/** +!experiments/ +!experiments/bfv_nitro/ +!experiments/bfv_nitro/Dockerfile +!experiments/bfv_nitro/Dockerfile.dockerignore +!experiments/bfv_nitro/nsmlib.Cargo.lock +!experiments/bfv_nitro/requirements.lock +!experiments/bfv_nitro/__init__.py +!experiments/bfv_nitro/transport.py +!experiments/bfv_nitro/service.py +**/__pycache__ +**/*.so +**/*.pyc +**/.env* diff --git a/experiments/bfv_nitro/README.md b/experiments/bfv_nitro/README.md new file mode 100644 index 0000000..fe19f67 --- /dev/null +++ b/experiments/bfv_nitro/README.md @@ -0,0 +1,265 @@ +# BFV on AWS Nitro Enclaves + +This optional experiment runs the existing public C++ RNS evaluator inside an +AWS Nitro enclave. The owner checks attestation and a signed response receipt +before decrypting locally. Each request performs one complete search. The +existing Paillier, BFV, SEAL example and owner-recomputing verifier remain usable +through their existing APIs. + +This is an implementation for review and hardware testing, not a production +security certification. Local tests and benchmarks do not establish that an EIF +boots correctly or how fast it runs on EC2. See +[`docs/research/native-bfv-nitro.md`](../../docs/research/native-bfv-nitro.md) +for the protocol, evidence and remaining review work. + +## Files and trust boundary + +| File | Responsibility | +| --- | --- | +| `src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py` | Owner authorization, attested client/server sessions, receipt gate and protected client state | +| `src/xtrace_sdk/x_vec/crypto/bfv_nitro.py` | AWS certificate/COSE verification, owner PCR policy and official NSM C-library adapter | +| `service.py` | Fixed-policy evaluator entrypoint inside the enclave | +| `transport.py` / `relay.py` | Bounded framing and untrusted TCP-to-vsock forwarding | +| `client.py` | Owner-side synthetic end-to-end demonstration | +| `Dockerfile` / `Builder.Dockerfile` | Public evaluator container and optional local EIF build tooling | +| `benchmarks/bfv_nitro.py` | Local protocol benchmark or real-enclave RPC benchmark | + +The owner holds the BFV secret, an independent Ed25519 request-signing secret, +and the private distance-check summary. The enclave receives only public BFV +keys, encrypted vectors, owner-signed requests and the existing transport MAC +key. That MAC key is visible to the host and is **not** an owner credential. +The enclave creates its own ephemeral receipt-signing key using NSM entropy. +It does not receive an owner decryption key or expose a general signing API. + +The measured service supports one owner, one immutable index and one active +attestation session. The host can interrupt, replay or replace traffic and can +deny service. The client accepts only evidence for its approved code, actual +setup digest, owner identity, execution policy and index epoch. + +## Build on an owner-trusted Linux machine + +Run these commands from the repository root. Install the optional client +dependencies and compile the existing native kernels: + +```bash +uv sync --extra bfv-nitro +make -C src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext PYTHON="$PWD/.venv/bin/python" +docker build -f experiments/bfv_nitro/Dockerfile -t xtrace-bfv-nitro:research . +``` + +The Dockerfile pins base-image digests, a hash-checked official AWS NSM source +archive, Cargo dependencies and Python wheel hashes. It compiles only the +**public** BFV extension into the service image. Its restricted build context +excludes local binaries, test attestors, keys, indices, `.env` and git metadata. +OS packages still come from signed distribution repositories; this recipe does +not promise bit-for-bit reproducible builds. Review each resulting EIF and its +measurements through an owner-trusted release process. + +AWS supports building EIFs on Linux outside AWS. Running one requires a +compatible EC2 parent with Nitro Enclaves enabled. +[AWS build command](https://docs.aws.amazon.com/enclaves/latest/user/cmd-nitro-build-enclave.html). + +If `nitro-cli` is installed on the trusted builder: + +```bash +mkdir -p experiments/bfv_nitro/artifacts +nitro-cli build-enclave \ + --docker-uri xtrace-bfv-nitro:research \ + --output-file experiments/bfv_nitro/artifacts/xtrace-bfv.eif \ + > experiments/bfv_nitro/artifacts/measurements.json +``` + +Alternatively, the included builder container installs the official Amazon +Linux Nitro CLI. Its only host mounts are the local Docker socket and the +output directory. Treat the builder as trusted tooling: Docker socket access +confers control over the local daemon. No AWS credentials are needed. + +```bash +docker build -f experiments/bfv_nitro/Builder.Dockerfile \ + -t xtrace-bfv-nitro-builder:research . +mkdir -p experiments/bfv_nitro/artifacts +docker run --rm --network none \ + --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \ + --mount "type=bind,src=$PWD/experiments/bfv_nitro/artifacts,dst=/out" \ + xtrace-bfv-nitro-builder:research build-enclave \ + --docker-uri xtrace-bfv-nitro:research --output-file /out/xtrace-bfv.eif \ + > experiments/bfv_nitro/artifacts/measurements.json +``` + +Keep the measurement JSON on the owner machine. The client requires nonzero +PCR0, PCR1 and PCR2 from this trusted build. Additional PCR pins are checked +conjunctively. Do not replace these values with measurements supplied solely by +the untrusted hosting endpoint. A rebuild can change the PCRs; it needs explicit +owner review and a new policy. Exact image pins are mandatory even for an EIF +signed with a release certificate. +[AWS measurement definitions](https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html). + +The audited local amd64 build is recorded in `build-record.json`; its EIF, +measurement JSON and independent `describe-eif` output are in `artifacts/`. +That record can also be passed to `--pins`, since it contains the same +`Measurements` object. It refers to that particular build, not future rebuilds. +The EIF has no image-signing certificate; runtime AWS attestation is still +mandatory. No real enclave execution is claimed by a successful local build. +`build-record-before-audit.json` preserves the earlier build for comparison; +its measurements do not match the rebuilt artifact. See the +[code audit](../../docs/research/native-bfv-code-audit.md) for validation and +updated local measurements. + +## Run the pilot on an existing Nitro-enabled EC2 parent + +Transfer the EIF to a compatible parent of the same architecture. On Amazon +Linux 2023, install `aws-nitro-enclaves-cli` and +`aws-nitro-enclaves-cli-devel`. Configure +`/etc/nitro_enclaves/allocator.yaml`, add the operator to the `ne` and `docker` +groups, and start the allocator and Docker services as described in +[AWS's installation guide](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave-cli-install.html). + +For an initial pilot, reserving 4 vCPUs and 8,192 MiB is a conservative starting +allocation to measure, not an established memory requirement or minimum. Leave +sufficient resources for the parent. The review-profile setup and preprocessing +need substantially more memory than the allocator's small default. The current +evaluator runs requests serially; this allocation does not imply four-way BFV +parallelism. + +Launch the reviewed artifact, substituting its actual path: + +```bash +nitro-cli run-enclave --enclave-cid 16 --cpu-count 4 --memory 8192 \ + --eif-path /path/to/xtrace-bfv.eif +``` + +Do not add `--debug-mode` or `--attach-console`: zero/debug measurements are +rejected by the client. Record the enclave ID for later lifecycle operations. +The enclave listens on vsock port 5000. From a checkout on the parent, run the +stdlib-only relay with Python 3.10 or newer: + +```bash +python3 -m experiments.bfv_nitro.relay --cid 16 +``` + +The relay binds `127.0.0.1:9000`. On the owner machine, establish an authenticated +tunnel using your normal SSH identity and host-key verification: + +```bash +ssh -N -L 9000:127.0.0.1:9000 ec2-user@YOUR_PARENT_HOST +``` + +Then, from the owner's SDK checkout: + +```bash +.venv/bin/python -m experiments.bfv_nitro.client \ + --pins experiments/bfv_nitro/artifacts/measurements.json --vectors 32 --queries 3 +``` + +The demonstration generates synthetic data and fresh private keys locally, +uploads the signed public setup, verifies AWS evidence, and checks every +returned distance against the known synthetic answer. It prints the closest +three locally. It sends no result-acceptance or decryption feedback to the relay. + +Start a **fresh empty enclave** for each invocation of the demonstration, +integration test or benchmark: each registers a new immutable owner/index. +Registration is self-authorized by its first owner. Hosting/account admission +controls remain a deployment responsibility; a malicious parent can claim an +empty service first and cause denial of service. It cannot make the legitimate +client approve that other owner's setup. + +## Sessions and application integration + +The deployed entrypoint fixes `bfv_review_policy()` and the `residue` backend: +N=16,384, 512-bit binary vectors, t=65,537, 180-bit coefficient modulus, existing +error distribution, and 50-bit terminal response modulus. The peer cannot +request alternate code, a weaker parameter set or a different backend. + +The owner API follows the existing session clients: + +```python +setup = client.prepare_index(vectors, vector_ids=vector_ids) +remote.register(client.registration_packet(setup)) +client.accept_attestation(*remote.attest(client.begin_attestation())) +query = client.begin_query(query_vector) +distances = client.finish_query(*remote.search(query)) +``` + +Renew attestation with the same two handshake calls before the five-minute +lease expires. Renewal retains the imported index, discards pending client +queries and replaces the active session. Complete outstanding work first. +Enrollment has a 30-second client deadline; evidence can be at most 120 seconds +old, with 5 seconds of future clock tolerance. Certificates are checked against +the owner's clock. A failed enrollment never enables decryption. + +At most 1,024 enrollment nonces are retained per enclave instance; query and +resource budgets also remain bounded by the existing execution policy. Exhausted +budgets require a planned new instance/session state, rather than silently +disabling checks. A restart loses public preprocessing and its receipt key; +re-upload the setup and attest the new key. Index replacement uses a new +instance and owner-maintained epoch. + +`protect_state()` encrypts the BFV secret, owner request key and verification +state under an independent wrapping key. It does not preserve an attested +session. `restore()` requires the expected setup, epoch and policies from owner +configuration and always requires fresh attestation. Keep the expected epoch +durably outside rollback-prone backups; encrypted state alone cannot detect +restoring an entire old owner configuration. Use a separate local workflow for +policy migration. Do not catch rejection and fall back to a raw decrypt API. + +This service is an explicit experiment. It is not yet connected to the existing +production XTrace HTTP endpoints, multi-tenant storage or deployment automation. +The underlying circuit returns packed distances; top-k selection remains local. + +## Tests and benchmarks + +Offline tests use a synthetic CA only through test-local monkeypatching. The +public client has no custom-root or fake-attestation option. Tests also include +an authentic historical AWS document and the official NSM C ABI via a test shim. + +```bash +uv run --extra bfv-nitro pytest tests/x_vec/test_bfv_nitro.py \ + tests/x_vec/test_bfv_attested_client.py tests/x_vec/test_bfv_nitro_transport.py +uv run --extra bfv-nitro python benchmarks/bfv_nitro.py \ + --mode local --num-vectors 1024 --repeats 4 \ + --json-out benchmarks/results/native_bfv_nitro_local_1024.json +``` + +The local benchmark checks identical ciphertexts with the baseline and includes +real certificate/signature verification against a **test** CA. It measures no +hardware isolation, NSM, vsock or cloud overhead. The report distinguishes the +one cold trial from the three warm trials. + +Against a fresh real enclave and a running tunnel: + +```bash +XTRACE_BFV_NITRO_PINS="$PWD/experiments/bfv_nitro/artifacts/measurements.json" \ + uv run --extra bfv-nitro pytest tests/x_vec/test_bfv_nitro_aws.py -v +``` + +The real test skips unless configured and cannot be satisfied by a synthetic +root. For a separate fresh enclave, use `benchmarks/bfv_nitro.py --mode aws +--pins PATH --num-vectors 1024 --repeats 20`. AWS-mode search timings include +relay and network latency; the comparison evaluator runs on the machine +invoking the benchmark. Comparing hardware overhead requires a matched cloud +CPU and compiler configuration. Measure enclave memory separately from the +benchmark's combined-process RSS. + +## Security review still required + +Attestation authenticates a measured program under AWS Nitro's trust model; +it does not prove that program correct. The current trusted software includes +the kernel/bootstrap, CPython, SDK imports, native extension, parsers and all +dependencies in the image. Reducing that dependency footprint is further work. +The service excludes a remote shell, configurable execution or host GPU +offload, but a vulnerability in measured code could still expose its receipt +key. That would undermine the pre-decryption gate. + +The verifier uses OpenSSL path validation, `cryptography` for ES384 verification, +bounded `cbor2` decoding and the pinned AWS root fingerprint. Application +bindings and policy are new code requiring independent review. It checks +certificate validity but does not fetch CRLs or implement an automated AWS +platform-revocation/advisory policy. Define an owner-managed patch and trusted +measurement withdrawal process before customer deployment. +[AWS evidence validation](https://docs.aws.amazon.com/enclaves/latest/user/verify-root.html). + +Real EC2 end-to-end execution, lifecycle/failure testing, measured enclave +performance, independent protocol and BFV parameter review, and the existing +client private-key side-channel assurance work remain release requirements. +Host denial of service and traffic metadata remain visible. There is no claim +of general chosen-ciphertext security for the raw BFV encryption scheme. diff --git a/experiments/bfv_nitro/__init__.py b/experiments/bfv_nitro/__init__.py new file mode 100644 index 0000000..dbe0489 --- /dev/null +++ b/experiments/bfv_nitro/__init__.py @@ -0,0 +1 @@ +"""Optional Nitro enclave service, untrusted relay and owner-side transport.""" diff --git a/experiments/bfv_nitro/build-record-before-audit.json b/experiments/bfv_nitro/build-record-before-audit.json new file mode 100644 index 0000000..0b19e1e --- /dev/null +++ b/experiments/bfv_nitro/build-record-before-audit.json @@ -0,0 +1,109 @@ +{ + "status": "Local image build and independent describe-eif check passed; not booted on Nitro hardware", + "baseline_commit": "46b637b604e1c2835653140610b7b2396f10b284", + "working_tree_changes": true, + "architecture": "amd64", + "build_time_utc": "2026-09-13T03:30:46.720693305+00:00", + "build_tool": "nitro-cli", + "build_tool_version": "1.4.5", + "kernel_version": "4.14.256", + "docker_image_id": "sha256:aafb441783a6c25056ef684e0f65430f96a598a7337d4b5f90f87ce19a7a95b5", + "eif_sha256": "a6b354a039abdda7399768a6d9b9138f729128d47ad60830dc33b2c074bcd70c", + "eif_bytes": 300334519, + "eif_version": 4, + "eif_signed": false, + "eif_crc_valid": true, + "Measurements": { + "HashAlgorithm": "Sha384 { ... }", + "PCR0": "7cff13985ab21bd26340420aeca94447353c3abb1ae98fcf46231bb011fa7ad2d209685075d71a3cfb1d842830b2a713", + "PCR1": "4b4d5b3661b3efc12920900c80e126e4ce783c522de6c02a2a5bf7af3a2b9327b86776f188e4be1c1c404a129dbda493", + "PCR2": "b375b4b2656e53032b4fa395a2b095a993972e8b9fdfb94874777f891485be71e120a3a3a227d23ee3dbcd69b23de7cc" + }, + "source_sha256": { + "src/xtrace_sdk/__init__.py": "6d4af37502706c09ea4b31926b9ef4f761ded7ee4a0bd9720de8055ad4318ea3", + "src/xtrace_sdk/cli/__init__.py": "3a3c2413d0459d993f48f930397eaf8922c7e436ff2b7d077bb8a5e481e6e9b0", + "src/xtrace_sdk/cli/commands/_utils/_admin_key.py": "c227b33ea39cf94d77446b6967c41aeeed6144d8d4a558be7942be9fb1c9d155", + "src/xtrace_sdk/cli/commands/_utils/_aio_http.py": "d8486db1486410a415129679ab4912bd960255df87170524fe39e3d833ef879f", + "src/xtrace_sdk/cli/commands/_utils/_disk_loader.py": "af94ecb29baf10fe96ef11dd7d77ee26fda33ef1e93c830411546b5b8cf2d1ab", + "src/xtrace_sdk/cli/commands/_utils/_errors.py": "f245d638e840f741e386b7088bde25b7d2754c0417feefe075d2ca1a93d9d2bb", + "src/xtrace_sdk/cli/commands/_utils/_getHash.py": "2cc4285f48d0100d43d11c1188ac9f9f9992ea80db2ffa2fcb5c3bd7a9c3babd", + "src/xtrace_sdk/cli/commands/_utils/_prompt.py": "0a6ee523acfb11b48a70ab6685422df9a3cc7d9729900219888294a3fe150879", + "src/xtrace_sdk/cli/commands/fetch.py": "7425beecc3180a72d3bed1a1887ed15e98034b7c5dc36132615b9be15956b056", + "src/xtrace_sdk/cli/commands/head.py": "e550813f462ef5773599cf0fd5580d1d13478e68b6786b05d71ea5fec0e9af08", + "src/xtrace_sdk/cli/commands/init.py": "04f356ab18513de12410088428aa85d4b2d1ced755cd7bff9f2b859c7d47fc02", + "src/xtrace_sdk/cli/commands/knowledgebase/__init__.py": "b538ac9e58fdd518c0131d9b44acf66ae0aa0190a1aa0a7f308d5aa20751daff", + "src/xtrace_sdk/cli/commands/knowledgebase/create.py": "fdf0ac9e6ff68107bf800fe6169bb2e861f86ab700ff4b02f7fb8dedf26556fc", + "src/xtrace_sdk/cli/commands/knowledgebase/delete.py": "34dd10f2d7354b5666b9171175112895ae08c4655756e4284f5b6f9bf5a72dc1", + "src/xtrace_sdk/cli/commands/knowledgebase/describe.py": "9f84a4804cafc4846bc7b8ce57ea6c04684517d3a360a13ce3d44014c59a5f69", + "src/xtrace_sdk/cli/commands/knowledgebase/list.py": "4e9d0edbc83fa870f69acf7a49bb09c94ca053d1c00704ad7e09ecc4a3381a51", + "src/xtrace_sdk/cli/commands/load.py": "f32511bdc1e5d3b9b28ba76d7edbadc03ca218fd669e468f1f803c17b24c1f8c", + "src/xtrace_sdk/cli/commands/retrieve.py": "9b5d245768fab15b217e8d9cac637c0e186888d372e6890af8265afd2ce4332f", + "src/xtrace_sdk/cli/commands/shell.py": "f1d5d8c26ce1e7c1a23f6135f2a9d0590a3ca3f671f66078365bf8343968d2e1", + "src/xtrace_sdk/cli/commands/upsert.py": "b1bada37a775e5336146edba757167ba39c0678f9de681ae35d037e9f6c1beac", + "src/xtrace_sdk/cli/commands/upsert_file.py": "90bbfb7af47384807c0a6afa07414f8d308206de79edf6b7f179e9bb0c95e7ca", + "src/xtrace_sdk/cli/commands/version.py": "412cbcb58c62a5cd20642b45f14c7d4abea5341866c543d3fda09e808be78fa3", + "src/xtrace_sdk/cli/main.py": "f6af42c342ffcd3e9d5b44959a04c1447f41b6cd3c0a3603684a9a8abdc8706d", + "src/xtrace_sdk/cli/shell.py": "efb3c27fb332741eff74644ce0b66c66fe6a459b4d4935c177222384bb0c06da", + "src/xtrace_sdk/cli/state.py": "f5d0afb300e091e443775d8a3540e3dedd479e7fb109af152ae2239319694786", + "src/xtrace_sdk/integrations/__init__.py": "75ef03361ac525c5793513a8a41792ad74a7f6fa552e3b268ca7f86f42108cd9", + "src/xtrace_sdk/integrations/xtrace.py": "20efa57daba4585b8c7efd4343c6c99e0682b67352d33293c97f14ba7feea8f8", + "src/xtrace_sdk/x_mem/__init__.py": "1eda9ebdbd9e6966eb7e98cf2da065dc69663d0f65f6ac36d291eaa9a53ae304", + "src/xtrace_sdk/x_vec/__init__.py": "1abee1815d691eaf2613565b221c3bff4f700dff6bed45ea326a462a1916aa90", + "src/xtrace_sdk/x_vec/crypto/__init__.py": "4e756987a9074daa86d19d451b12e49f9ffe77f7d2ed6d1c620aac8d74893cf5", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py": "ba64de6db3f76031ba9c35627a0a3790244c816937a97a3b65c98b9c699bb33d", + "src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py": "cbab29f860b07d44264203c6689fece4044a08b76c7564a909eacff973993dcf", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/__init__.py": "5b403430d7bea9a571afeecf51f39d99b1360f162f33f6340dfca0b257d2b320", + "src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py": "2e684a8a7f2364a1134aa0271ee34fc6e2562367f26ae453f61a03fa80e31db3", + "src/xtrace_sdk/x_vec/crypto/bfv_nitro.py": "94ac7be118ae20d4e6c58cc903cf26bb98e66d3928dd9d143616c5b479b5a654", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "6a384b16c347d84f90e40ccc9903b1ae916ae277e6171bd668181366d56e88a0", + "src/xtrace_sdk/x_vec/crypto/commitment/__init__.py": "be7146c1fab4084a721cda5ab1063eb5b44912b22845b3d90236664dc220d37b", + "src/xtrace_sdk/x_vec/crypto/commitment/merkle.py": "2871ed5ed2739fea5687d868d535808306f00a170296f0b9402f1173602ca1ad", + "src/xtrace_sdk/x_vec/crypto/device.py": "8452899bc9aeb66e6556dfd5b4c876272e77379a90b2bd3320e93238cb773b9b", + "src/xtrace_sdk/x_vec/crypto/encryption/__init__.py": "261a50ed1758cee38c880cc625187f8acb051c6d5c96ba7a71ee956bdbea2cc5", + "src/xtrace_sdk/x_vec/crypto/encryption/aes.py": "2c833351db3b249c381e315307d858791ee4e5b633d3465127cad763b4d095d5", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py": "b1291e6d017be84638b054fb84e651734a44b2d54d601eb50dae2e9d2016bdfa", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/goldwasser_micali.py": "4b32296c6210663c098718fa49bd810c427b1f8102d026d12a7d734f58e8e3d1", + "src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py": "915da9e500d0a142651243bb13de7631dc96103fea2ec9e5b9288bee498010e7", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier.py": "525a1944caef38d5d216ae84fcac982627d2a2fddea50b747e47ef05e2e91c9f", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier_lookup.py": "eed0be7e0e2c1f6e8e8c0138262ec071fef73648e334a948ef1f82df6e9b9712", + "src/xtrace_sdk/x_vec/crypto/goldwasser_micali_client.py": "0982ad7532d07c8e26a701359ce8f4f7d4c1a3beb714050e741f3b534d59935f", + "src/xtrace_sdk/x_vec/crypto/hamming_client_base.py": "039fb7eb1f15347b8e79303cd4c80eec59e8271954815b6e69268cb6378ea57a", + "src/xtrace_sdk/x_vec/crypto/key_provider.py": "d55e1ac740172ecda6f025ae7b31e3e917a02b05ffe92b91ed7da3f8e97adca2", + "src/xtrace_sdk/x_vec/crypto/paillier_client.py": "40582d7acb86ee7dd2060e2d795c76829f4879db3b1eca9518be7a9599de4788", + "src/xtrace_sdk/x_vec/crypto/paillier_gpu_ext/__init__.py": "c79c1f793ec8686baa73d3b3e45840b8a9920d1a1fa68aa5aabf0eeba27b8f98", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_client.py": "4faa0dd71d03b3e9bc9e49c4e60e8be7b173ff514b52cf1d94a3b1a2426bf699", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_gpu_ext/__init__.py": "e710a08f9a703f1cabf87021e3ceba821947d1179acb52cb9e1b49aa0b181779", + "src/xtrace_sdk/x_vec/crypto/signature/__init__.py": "ea3e1c42bf2e9d6ab548f2eda28346f898b977016c2883a6e805573bf2f55d59", + "src/xtrace_sdk/x_vec/data_loaders/__init__.py": "6144396d539308208f8467840aa6a454fc7c6c09f307c603a88804d203d57fa8", + "src/xtrace_sdk/x_vec/data_loaders/loader.py": "046cecae8a059a4c6a46d6e1402a4ccbf313d7c3e322e4987390818050b4af31", + "src/xtrace_sdk/x_vec/inference/__init__.py": "459b508c077f0c747972f5f45c6a676e1671f82d8c94b6ffbeefb0088772697c", + "src/xtrace_sdk/x_vec/inference/embedding.py": "fddcf46dbbb6801f9177ad844597b101c62d2ee614e45d29c834d4ab0829318a", + "src/xtrace_sdk/x_vec/inference/llm.py": "ce64bad375e0de5d133c6c65e659c8b3569ca8fe7b8f3790c0d00b22f81a1acc", + "src/xtrace_sdk/x_vec/retrievers/__init__.py": "27615f46e6b39bbfc8519c6fa33bb788f7d0030c534b653fcee81be1556b33d9", + "src/xtrace_sdk/x_vec/retrievers/retriever.py": "fb1f8a4b421638d7dc648ec81275a8e2eeef5cb3ff4a677fc96b88f17533cbf7", + "src/xtrace_sdk/x_vec/utils/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "src/xtrace_sdk/x_vec/utils/bench.py": "88c7bbda827c6f76bec4466bae7121e6bb36307d4a9c2b633a813b9ca364b85f", + "src/xtrace_sdk/x_vec/utils/execution_context.py": "200a4637f591cc93b905a14827d6f2d9e484873045434f697547141a87f911f7", + "src/xtrace_sdk/x_vec/utils/settings.py": "4b05198d71a5f5a7b9f3938bbd37477e980608725f98180765375462e7671d09", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "ee3f7371f92ba9fa3ebb8037906fc23c5f7832f96908d5879f1a4326847ffb53", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp": "7b48d92e526147d97d9401e868c8d3e9dbf0b8ba567a67e05047e01f2a371fc0", + "experiments/bfv_nitro/Dockerfile": "13b4c1d01f45e3c8eee0d8730bb2e21ea8d6248fbe613c5e2ae5d803b0077b33", + "experiments/bfv_nitro/Dockerfile.dockerignore": "f01f15b430132b53f818952d293369db366ce885765dd796e2ab9f7e5624cda2", + "experiments/bfv_nitro/Builder.Dockerfile": "0ee8911d63335ec9df990af7d1c595a8bd6bd523cee794e6a2ee0e3dd4a59c7b", + "experiments/bfv_nitro/Builder.Dockerfile.dockerignore": "06ab6a1a0df6a96cd59c3074b1b4a9ef9d4ef2b1d5832236358754d3767d220d", + "experiments/bfv_nitro/requirements.lock": "d79382cc3a4199d8c99d817a8c381d90287edd45065a305a3a3097c09326c0fb", + "experiments/bfv_nitro/nsmlib.Cargo.lock": "02dce31c049455647f78a2315ade3d015564fce6d3ea6f03750b7c24ae4c8fc6", + "experiments/bfv_nitro/__init__.py": "96f217ab43bd80b51d6b928cc128a7f7d6fb9c8bce318267ac157436bb49df75", + "experiments/bfv_nitro/service.py": "1354ca96c4c9ac65753dcfaf2cf86b9993a3cb35f4f9e7c3e165be56f50e3147", + "experiments/bfv_nitro/transport.py": "ede43317af73b53fa19fb95a84b147dcd9e47e3032386e67afa5daefed0e3b2a", + "LICENSE": "c8858a5a76440bbca484e134cf7df46385d090dd18b2c58e650f939258802e5b" + } +} diff --git a/experiments/bfv_nitro/build-record.json b/experiments/bfv_nitro/build-record.json new file mode 100644 index 0000000..475afa3 --- /dev/null +++ b/experiments/bfv_nitro/build-record.json @@ -0,0 +1,123 @@ +{ + "status": "Audited local image build and independent describe-eif check passed; not booted on Nitro hardware", + "baseline_commit": "46b637b604e1c2835653140610b7b2396f10b284", + "working_tree_changes": true, + "architecture": "amd64", + "build_time_utc": "2026-09-15T17:52:33.625586066+00:00", + "build_tool": "nitro-cli", + "build_tool_version": "1.4.5", + "kernel_version": "4.14.256", + "docker_image_id": "sha256:5879cf7051f2c60485cb7c1d1877e981969bab5d72ae65398196c8e0783e4fa8", + "eif_sha256": "25ddd36f47d70bb9a4ee1786fd868dfa5f8c52b8bef1981c5715bb172141c278", + "eif_bytes": 300336566, + "eif_version": 4, + "eif_signed": false, + "eif_crc_valid": true, + "Measurements": { + "HashAlgorithm": "Sha384 { ... }", + "PCR0": "4e2f9e0d1dac662bb3b243a0abc29362e28a1761aa1c6ab8648de9edfd8eb7602e2877cdd181b63350054c1fb3d64ed1", + "PCR1": "4b4d5b3661b3efc12920900c80e126e4ce783c522de6c02a2a5bf7af3a2b9327b86776f188e4be1c1c404a129dbda493", + "PCR2": "790eedb99bd35312d771074c577f50ae9a6d5a76c741b20ba32e864ab3d351c5214e7ed5fbddd1ba46aaef22ecc88d2e" + }, + "source_sha256": { + "src/xtrace_sdk/__init__.py": "6d4af37502706c09ea4b31926b9ef4f761ded7ee4a0bd9720de8055ad4318ea3", + "src/xtrace_sdk/cli/__init__.py": "3a3c2413d0459d993f48f930397eaf8922c7e436ff2b7d077bb8a5e481e6e9b0", + "src/xtrace_sdk/cli/commands/_utils/_admin_key.py": "c227b33ea39cf94d77446b6967c41aeeed6144d8d4a558be7942be9fb1c9d155", + "src/xtrace_sdk/cli/commands/_utils/_aio_http.py": "d8486db1486410a415129679ab4912bd960255df87170524fe39e3d833ef879f", + "src/xtrace_sdk/cli/commands/_utils/_disk_loader.py": "af94ecb29baf10fe96ef11dd7d77ee26fda33ef1e93c830411546b5b8cf2d1ab", + "src/xtrace_sdk/cli/commands/_utils/_errors.py": "f245d638e840f741e386b7088bde25b7d2754c0417feefe075d2ca1a93d9d2bb", + "src/xtrace_sdk/cli/commands/_utils/_getHash.py": "2cc4285f48d0100d43d11c1188ac9f9f9992ea80db2ffa2fcb5c3bd7a9c3babd", + "src/xtrace_sdk/cli/commands/_utils/_prompt.py": "0a6ee523acfb11b48a70ab6685422df9a3cc7d9729900219888294a3fe150879", + "src/xtrace_sdk/cli/commands/fetch.py": "7425beecc3180a72d3bed1a1887ed15e98034b7c5dc36132615b9be15956b056", + "src/xtrace_sdk/cli/commands/head.py": "e550813f462ef5773599cf0fd5580d1d13478e68b6786b05d71ea5fec0e9af08", + "src/xtrace_sdk/cli/commands/init.py": "04f356ab18513de12410088428aa85d4b2d1ced755cd7bff9f2b859c7d47fc02", + "src/xtrace_sdk/cli/commands/knowledgebase/__init__.py": "b538ac9e58fdd518c0131d9b44acf66ae0aa0190a1aa0a7f308d5aa20751daff", + "src/xtrace_sdk/cli/commands/knowledgebase/create.py": "fdf0ac9e6ff68107bf800fe6169bb2e861f86ab700ff4b02f7fb8dedf26556fc", + "src/xtrace_sdk/cli/commands/knowledgebase/delete.py": "34dd10f2d7354b5666b9171175112895ae08c4655756e4284f5b6f9bf5a72dc1", + "src/xtrace_sdk/cli/commands/knowledgebase/describe.py": "9f84a4804cafc4846bc7b8ce57ea6c04684517d3a360a13ce3d44014c59a5f69", + "src/xtrace_sdk/cli/commands/knowledgebase/list.py": "4e9d0edbc83fa870f69acf7a49bb09c94ca053d1c00704ad7e09ecc4a3381a51", + "src/xtrace_sdk/cli/commands/load.py": "f32511bdc1e5d3b9b28ba76d7edbadc03ca218fd669e468f1f803c17b24c1f8c", + "src/xtrace_sdk/cli/commands/retrieve.py": "9b5d245768fab15b217e8d9cac637c0e186888d372e6890af8265afd2ce4332f", + "src/xtrace_sdk/cli/commands/shell.py": "f1d5d8c26ce1e7c1a23f6135f2a9d0590a3ca3f671f66078365bf8343968d2e1", + "src/xtrace_sdk/cli/commands/upsert.py": "b1bada37a775e5336146edba757167ba39c0678f9de681ae35d037e9f6c1beac", + "src/xtrace_sdk/cli/commands/upsert_file.py": "90bbfb7af47384807c0a6afa07414f8d308206de79edf6b7f179e9bb0c95e7ca", + "src/xtrace_sdk/cli/commands/version.py": "412cbcb58c62a5cd20642b45f14c7d4abea5341866c543d3fda09e808be78fa3", + "src/xtrace_sdk/cli/main.py": "f6af42c342ffcd3e9d5b44959a04c1447f41b6cd3c0a3603684a9a8abdc8706d", + "src/xtrace_sdk/cli/shell.py": "efb3c27fb332741eff74644ce0b66c66fe6a459b4d4935c177222384bb0c06da", + "src/xtrace_sdk/cli/state.py": "f5d0afb300e091e443775d8a3540e3dedd479e7fb109af152ae2239319694786", + "src/xtrace_sdk/integrations/__init__.py": "75ef03361ac525c5793513a8a41792ad74a7f6fa552e3b268ca7f86f42108cd9", + "src/xtrace_sdk/integrations/xtrace.py": "20efa57daba4585b8c7efd4343c6c99e0682b67352d33293c97f14ba7feea8f8", + "src/xtrace_sdk/py.typed": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "src/xtrace_sdk/x_mem/__init__.py": "1eda9ebdbd9e6966eb7e98cf2da065dc69663d0f65f6ac36d291eaa9a53ae304", + "src/xtrace_sdk/x_vec/__init__.py": "1abee1815d691eaf2613565b221c3bff4f700dff6bed45ea326a462a1916aa90", + "src/xtrace_sdk/x_vec/crypto/__init__.py": "4e756987a9074daa86d19d451b12e49f9ffe77f7d2ed6d1c620aac8d74893cf5", + "src/xtrace_sdk/x_vec/crypto/bfv_assurance.py": "ba64de6db3f76031ba9c35627a0a3790244c816937a97a3b65c98b9c699bb33d", + "src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py": "d3bed7a596fd19f834ea308fc332f0a28109a1136c7dd63026819b4cc106c7de", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "ee3f7371f92ba9fa3ebb8037906fc23c5f7832f96908d5879f1a4326847ffb53", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/__init__.py": "5b403430d7bea9a571afeecf51f39d99b1360f162f33f6340dfca0b257d2b320", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.pyi": "11275c66776f344a015364d05af9d30534536c64dd82a08a6fd8de9f3a9fb098", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.pyi": "43f14ce1de6882836bed9f3a0afa98bc41501a9f0967d06e7a5c029af75b713a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp": "7b48d92e526147d97d9401e868c8d3e9dbf0b8ba567a67e05047e01f2a371fc0", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h": "66aceec7cc8452dabdde0061d347195e49a9500fba7bcbbe805c967b7171d767", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py": "2e684a8a7f2364a1134aa0271ee34fc6e2562367f26ae453f61a03fa80e31db3", + "src/xtrace_sdk/x_vec/crypto/bfv_nitro.py": "791107dd7daaea99b07cb9cc70116ec71fe932c0ece608a9bd65ce8a7aad6660", + "src/xtrace_sdk/x_vec/crypto/bfv_security.py": "17149c07384889f994f93bc9e7b209d965176cd916c97a0f6b7a138861a7204e", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "6a384b16c347d84f90e40ccc9903b1ae916ae277e6171bd668181366d56e88a0", + "src/xtrace_sdk/x_vec/crypto/commitment/__init__.py": "be7146c1fab4084a721cda5ab1063eb5b44912b22845b3d90236664dc220d37b", + "src/xtrace_sdk/x_vec/crypto/commitment/merkle.py": "2871ed5ed2739fea5687d868d535808306f00a170296f0b9402f1173602ca1ad", + "src/xtrace_sdk/x_vec/crypto/device.py": "8452899bc9aeb66e6556dfd5b4c876272e77379a90b2bd3320e93238cb773b9b", + "src/xtrace_sdk/x_vec/crypto/encryption/__init__.py": "261a50ed1758cee38c880cc625187f8acb051c6d5c96ba7a71ee956bdbea2cc5", + "src/xtrace_sdk/x_vec/crypto/encryption/aes.py": "2c833351db3b249c381e315307d858791ee4e5b633d3465127cad763b4d095d5", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py": "b1291e6d017be84638b054fb84e651734a44b2d54d601eb50dae2e9d2016bdfa", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/goldwasser_micali.py": "4b32296c6210663c098718fa49bd810c427b1f8102d026d12a7d734f58e8e3d1", + "src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py": "915da9e500d0a142651243bb13de7631dc96103fea2ec9e5b9288bee498010e7", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier.py": "525a1944caef38d5d216ae84fcac982627d2a2fddea50b747e47ef05e2e91c9f", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier_lookup.py": "eed0be7e0e2c1f6e8e8c0138262ec071fef73648e334a948ef1f82df6e9b9712", + "src/xtrace_sdk/x_vec/crypto/goldwasser_micali_client.py": "0982ad7532d07c8e26a701359ce8f4f7d4c1a3beb714050e741f3b534d59935f", + "src/xtrace_sdk/x_vec/crypto/hamming_client_base.py": "039fb7eb1f15347b8e79303cd4c80eec59e8271954815b6e69268cb6378ea57a", + "src/xtrace_sdk/x_vec/crypto/key_provider.py": "d55e1ac740172ecda6f025ae7b31e3e917a02b05ffe92b91ed7da3f8e97adca2", + "src/xtrace_sdk/x_vec/crypto/paillier_client.py": "40582d7acb86ee7dd2060e2d795c76829f4879db3b1eca9518be7a9599de4788", + "src/xtrace_sdk/x_vec/crypto/paillier_gpu_ext/Makefile": "9cce872d0be783b083f2ded192fcff1a047df4f2f0e6e8afe7d715e8362aa915", + "src/xtrace_sdk/x_vec/crypto/paillier_gpu_ext/__init__.py": "c79c1f793ec8686baa73d3b3e45840b8a9920d1a1fa68aa5aabf0eeba27b8f98", + "src/xtrace_sdk/x_vec/crypto/paillier_gpu_ext/paillier_GPU_client.cu": "b9868d8ccc40022deb41344e72a4816ff3dcbbbc6633a0c0267e82ae4d9b8bc6", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_client.py": "4faa0dd71d03b3e9bc9e49c4e60e8be7b173ff514b52cf1d94a3b1a2426bf699", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_gpu_ext/Makefile": "5830a894e36da0fc8cefeb3fac9bb4e1a416984812f36c968ba7db5f21261b52", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_gpu_ext/__init__.py": "e710a08f9a703f1cabf87021e3ceba821947d1179acb52cb9e1b49aa0b181779", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_gpu_ext/paillier_GPU_lookup_client.cu": "8e9321e2eef96cb31e3ab329280ff0ddaf820fd4f23061e75a7cf26e20eedc74", + "src/xtrace_sdk/x_vec/crypto/signature/__init__.py": "ea3e1c42bf2e9d6ab548f2eda28346f898b977016c2883a6e805573bf2f55d59", + "src/xtrace_sdk/x_vec/data_loaders/__init__.py": "6144396d539308208f8467840aa6a454fc7c6c09f307c603a88804d203d57fa8", + "src/xtrace_sdk/x_vec/data_loaders/loader.py": "046cecae8a059a4c6a46d6e1402a4ccbf313d7c3e322e4987390818050b4af31", + "src/xtrace_sdk/x_vec/inference/__init__.py": "459b508c077f0c747972f5f45c6a676e1671f82d8c94b6ffbeefb0088772697c", + "src/xtrace_sdk/x_vec/inference/embedding.py": "fddcf46dbbb6801f9177ad844597b101c62d2ee614e45d29c834d4ab0829318a", + "src/xtrace_sdk/x_vec/inference/llm.py": "ce64bad375e0de5d133c6c65e659c8b3569ca8fe7b8f3790c0d00b22f81a1acc", + "src/xtrace_sdk/x_vec/retrievers/__init__.py": "27615f46e6b39bbfc8519c6fa33bb788f7d0030c534b653fcee81be1556b33d9", + "src/xtrace_sdk/x_vec/retrievers/retriever.py": "fb1f8a4b421638d7dc648ec81275a8e2eeef5cb3ff4a677fc96b88f17533cbf7", + "src/xtrace_sdk/x_vec/utils/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "src/xtrace_sdk/x_vec/utils/bench.py": "88c7bbda827c6f76bec4466bae7121e6bb36307d4a9c2b633a813b9ca364b85f", + "src/xtrace_sdk/x_vec/utils/execution_context.py": "200a4637f591cc93b905a14827d6f2d9e484873045434f697547141a87f911f7", + "src/xtrace_sdk/x_vec/utils/settings.py": "4b05198d71a5f5a7b9f3938bbd37477e980608725f98180765375462e7671d09", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962", + "experiments/bfv_nitro/Dockerfile": "13b4c1d01f45e3c8eee0d8730bb2e21ea8d6248fbe613c5e2ae5d803b0077b33", + "experiments/bfv_nitro/Dockerfile.dockerignore": "f01f15b430132b53f818952d293369db366ce885765dd796e2ab9f7e5624cda2", + "experiments/bfv_nitro/Builder.Dockerfile": "0ee8911d63335ec9df990af7d1c595a8bd6bd523cee794e6a2ee0e3dd4a59c7b", + "experiments/bfv_nitro/Builder.Dockerfile.dockerignore": "06ab6a1a0df6a96cd59c3074b1b4a9ef9d4ef2b1d5832236358754d3767d220d", + "experiments/bfv_nitro/requirements.lock": "d79382cc3a4199d8c99d817a8c381d90287edd45065a305a3a3097c09326c0fb", + "experiments/bfv_nitro/nsmlib.Cargo.lock": "02dce31c049455647f78a2315ade3d015564fce6d3ea6f03750b7c24ae4c8fc6", + "experiments/bfv_nitro/__init__.py": "96f217ab43bd80b51d6b928cc128a7f7d6fb9c8bce318267ac157436bb49df75", + "experiments/bfv_nitro/service.py": "a4b322e678d9f49a16d44f76f53e9ea3fe2094ed213cb6f133aef21643c9e701", + "experiments/bfv_nitro/transport.py": "1fbb2767aa59209d0bdadd17b7aaafbfeb8f11a72e64cbb5f7997368a1bb36d5", + "LICENSE": "c8858a5a76440bbca484e134cf7df46385d090dd18b2c58e650f939258802e5b" + } +} diff --git a/experiments/bfv_nitro/client.py b/experiments/bfv_nitro/client.py new file mode 100644 index 0000000..a75c1b7 --- /dev/null +++ b/experiments/bfv_nitro/client.py @@ -0,0 +1,85 @@ +"""Run a synthetic owner-side search against the real Nitro enclave service. + +Run on the owner machine with trusted EIF measurements. No private material or +decryption feedback is sent to the relay. Use a fresh, empty enclave for each +invocation because registration is deliberately immutable. +""" + +import argparse +import json +import random +import secrets +from pathlib import Path + +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy +from xtrace_sdk.x_vec.crypto.bfv_attested_client import BFVAttestedClient +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_nitro import NitroAttestationPolicy + +from .transport import NitroRemote + + +def load_measurements(path: Path) -> NitroAttestationPolicy: + """Read owner-trusted nitro-cli build/describe-eif JSON; never fetch peer pins.""" + with path.open("rb") as stream: + raw = stream.read(16 * 1024 + 1) + if len(raw) > 16 * 1024: + raise ValueError("Measurement file is too large") + record = json.loads(raw) + measurements = record["Measurements"] + # nitro-cli 1.4.5 emits the Rust Debug spelling, including literal braces. + # Accept that documented output and the plain spelling, never a prefix match. + if measurements.get("HashAlgorithm", "").lower() not in ("sha384", "sha384 { ... }"): + raise ValueError("Expected SHA-384 EIF measurements") + pins = { + int(name[3:]): bytes.fromhex(value) + for name, value in measurements.items() + if name.startswith("PCR") and name[3:].isdigit() + } + return NitroAttestationPolicy(pins) + + +def main(): + """Create synthetic keys/data locally, verify attestation, then decrypt results.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--pins", type=Path, required=True, help="Trusted nitro-cli EIF measurement JSON" + ) + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=9000) + parser.add_argument("--vectors", type=int, default=32) + parser.add_argument("--queries", type=int, default=3) + parser.add_argument("--epoch", type=int, default=1) + args = parser.parse_args() + policy = bfv_review_policy() + if not 1 <= args.vectors <= policy.max_vectors or not 1 <= args.queries <= policy.max_queries: + parser.error("Synthetic workload exceeds the local BFV policy") + pins = load_measurements(args.pins) + rng = random.Random(1337) # Only synthetic plaintext uses a deterministic seed. + vectors = [[rng.randrange(2) for _ in range(512)] for _ in range(args.vectors)] + client = BFVAttestedClient( + BFVClient(**policy.config()), + secrets.token_bytes(32), + pins, + index_epoch=args.epoch, + policy=policy, + ) + remote = NitroRemote(args.host, args.port, policy=policy) + setup = client.prepare_index(vectors) + remote.register(client.registration_packet(setup)) + client.accept_attestation(*remote.attest(client.begin_attestation())) + print("AWS evidence, approved measurements and receipt-key possession verified.") + for _ in range(args.queries): + query = [rng.randrange(2) for _ in range(512)] + distances = client.finish_query(*remote.search(client.begin_query(query))) + expected = [sum(a != b for a, b in zip(query, vector, strict=True)) for vector in vectors] + if distances != expected: + raise RuntimeError("Synthetic Hamming result mismatch") + nearest = sorted(range(len(distances)), key=lambda i: (distances[i], i))[:3] + print( + "Verified local top results:", [(client.vector_ids[i], distances[i]) for i in nearest] + ) + + +if __name__ == "__main__": + main() diff --git a/experiments/bfv_nitro/nsmlib.Cargo.lock b/experiments/bfv_nitro/nsmlib.Cargo.lock new file mode 100644 index 0000000..c17f48f --- /dev/null +++ b/experiments/bfv_nitro/nsmlib.Cargo.lock @@ -0,0 +1,419 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "aws-nitro-enclaves-nsm-api" +version = "0.5.2" +dependencies = [ + "libc", + "log", + "nix", + "serde", + "serde_bytes", + "serde_cbor", +] + +[[package]] +name = "bitflags" +version = "2.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" + +[[package]] +name = "cbindgen" +version = "0.24.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b922faaf31122819ec80c4047cc684c6979a087366c069611e33649bf98e18d" +dependencies = [ + "heck", + "indexmap", + "log", + "proc-macro2", + "quote", + "serde", + "serde_json", + "syn 1.0.109", + "tempfile", + "toml", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "fastrand" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "half" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b43ede17f21864e81be2fa654110bf1e793774238d86ef8555c37e6519c0403" + +[[package]] +name = "hashbrown" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" + +[[package]] +name = "heck" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8" + +[[package]] +name = "hermit-abi" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e17592d60ebacc7d5e169f4663c5f84f9161cc90328abcfe8456f41e4dfcb284" + +[[package]] +name = "indexmap" +version = "1.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99" +dependencies = [ + "autocfg", + "hashbrown", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "log" +version = "0.4.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "memoffset" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a" +dependencies = [ + "autocfg", +] + +[[package]] +name = "nix" +version = "0.31.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", + "memoffset", +] + +[[package]] +name = "nsm-lib" +version = "0.5.2" +dependencies = [ + "aws-nitro-enclaves-nsm-api", + "cbindgen", + "serde_bytes", +] + +[[package]] +name = "nsm-test" +version = "0.5.2" +dependencies = [ + "aws-nitro-enclaves-nsm-api", + "nix", + "nsm-lib", + "serde_bytes", + "serde_cbor", + "signal-hook", + "threadpool", + "vsock", +] + +[[package]] +name = "num_cpus" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b" +dependencies = [ + "hermit-abi", + "libc", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_bytes" +version = "0.11.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" +dependencies = [ + "serde", + "serde_core", +] + +[[package]] +name = "serde_cbor" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2bef2ebfde456fb76bbcf9f59315333decc4fda0b2b44b420243c11e0f5ec1f5" +dependencies = [ + "half", + "serde", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.5", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "signal-hook" +version = "0.3.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" +dependencies = [ + "libc", + "signal-hook-registry", +] + +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom", + "once_cell", + "rustix", + "windows-sys", +] + +[[package]] +name = "threadpool" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d050e60b33d41c19108b32cea32164033a9013fe3b46cbd4457559bfbf77afaa" +dependencies = [ + "num_cpus", +] + +[[package]] +name = "toml" +version = "0.5.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f4f7f0dd8d50a853a531c426359045b1998f04219d88799810762cd4ad314234" +dependencies = [ + "serde", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "vsock" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ba782755fc073877e567c2253c0be48e4aa9a254c232d36d3985dfae0bd5205" +dependencies = [ + "libc", + "nix", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/experiments/bfv_nitro/relay.py b/experiments/bfv_nitro/relay.py new file mode 100644 index 0000000..8f6f33e --- /dev/null +++ b/experiments/bfv_nitro/relay.py @@ -0,0 +1,66 @@ +"""Untrusted, bounded TCP-to-vsock relay for the Nitro BFV experiment. + +Bind localhost by default and reach it through an authenticated SSH tunnel or a +deployment-managed TLS endpoint. This process has no BFV or signing secret and +cannot create accepted results. It can observe traffic and deny service. +""" + +import argparse +import select +import socket +import threading +import time +from concurrent.futures import ThreadPoolExecutor + + +def relay(stream, cid, port, admission): + """Relay one RPC with bounded buffers and a five-minute connection lifetime.""" + try: + with stream, socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM) as enclave: + enclave.settimeout(60) + enclave.connect((cid, port)) + deadline = time.monotonic() + 300 + while time.monotonic() < deadline: + readable, _, _ = select.select( + [stream, enclave], [], [], min(60, deadline - time.monotonic()) + ) + if not readable: + return + for source in readable: + data = source.recv(65536) + if not data: + return + target = enclave if source is stream else stream + target.settimeout(60) + target.sendall(data) + except (OSError, ValueError): + pass + finally: + admission.release() + + +def main(): + """Expose a selected enclave CID without accepting unbounded worker queues.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cid", type=int, required=True) + parser.add_argument("--enclave-port", type=int, default=5000) + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=9000) + args = parser.parse_args() + if args.cid < 3 or not 1 <= args.port <= 65535 or not 1 <= args.enclave_port <= 65535: + parser.error("Use a valid enclave CID and TCP/vsock ports") + admission = threading.BoundedSemaphore(4) + with ( + socket.create_server((args.host, args.port), backlog=4) as listener, + ThreadPoolExecutor(max_workers=4) as workers, + ): + while True: + stream, _ = listener.accept() + if admission.acquire(blocking=False): + workers.submit(relay, stream, args.cid, args.enclave_port, admission) + else: + stream.close() + + +if __name__ == "__main__": + main() diff --git a/experiments/bfv_nitro/requirements.lock b/experiments/bfv_nitro/requirements.lock new file mode 100644 index 0000000..81f2e14 --- /dev/null +++ b/experiments/bfv_nitro/requirements.lock @@ -0,0 +1,1203 @@ +# This file was autogenerated by uv via the following command: +# uv --cache-dir /tmp/xtrace-nitro-uv-cache export --frozen --no-dev --no-emit-project --extra bfv-nitro --output-file experiments/bfv_nitro/requirements.lock +aiohappyeyeballs==2.6.1 \ + --hash=sha256:c3f9d0113123803ccadfdf3f0faa505bc78e6a72d1cc4806cbd719826e943558 \ + --hash=sha256:f349ba8f4b75cb25c99c5c2d84e997e485204d2902a9597802b0371f09331fb8 + # via aiohttp +aiohttp==3.13.3 \ + --hash=sha256:01ad2529d4b5035578f5081606a465f3b814c542882804e2e8cda61adf5c71bf \ + --hash=sha256:042e9e0bcb5fba81886c8b4fbb9a09d6b8a00245fd8d88e4d989c1f96c74164c \ + --hash=sha256:05861afbbec40650d8a07ea324367cb93e9e8cc7762e04dd4405df99fa65159c \ + --hash=sha256:084911a532763e9d3dd95adf78a78f4096cd5f58cdc18e6fdbc1b58417a45423 \ + --hash=sha256:0add0900ff220d1d5c5ebbf99ed88b0c1bbf87aa7e4262300ed1376a6b13414f \ + --hash=sha256:10b47b7ba335d2e9b1239fa571131a87e2d8ec96b333e68b2a305e7a98b0bae2 \ + --hash=sha256:1449ceddcdbcf2e0446957863af03ebaaa03f94c090f945411b61269e2cb5daf \ + --hash=sha256:1cb93e166e6c28716c8c6aeb5f99dfb6d5ccf482d29fe9bf9a794110e6d0ab64 \ + --hash=sha256:2712039939ec963c237286113c68dbad80a82a4281543f3abf766d9d73228998 \ + --hash=sha256:27234ef6d85c914f9efeb77ff616dbf4ad2380be0cda40b4db086ffc7ddd1b7d \ + --hash=sha256:28e027cf2f6b641693a09f631759b4d9ce9165099d2b5d92af9bd4e197690eea \ + --hash=sha256:2b8d8ddba8f95ba17582226f80e2de99c7a7948e66490ef8d947e272a93e9463 \ + --hash=sha256:2be0e9ccf23e8a94f6f0650ce06042cefc6ac703d0d7ab6c7a917289f2539ad4 \ + --hash=sha256:2e41b18a58da1e474a057b3d35248d8320029f61d70a37629535b16a0c8f3767 \ + --hash=sha256:2eb752b102b12a76ca02dff751a801f028b4ffbbc478840b473597fc91a9ed43 \ + --hash=sha256:2fc82186fadc4a8316768d61f3722c230e2c1dcab4200d52d2ebdf2482e47592 \ + --hash=sha256:2fff83cfc93f18f215896e3a190e8e5cb413ce01553901aca925176e7568963a \ + --hash=sha256:34749271508078b261c4abb1767d42b8d0c0cc9449c73a4df494777dc55f0687 \ + --hash=sha256:34bac00a67a812570d4a460447e1e9e06fae622946955f939051e7cc895cfab8 \ + --hash=sha256:37239e9f9a7ea9ac5bf6b92b0260b01f8a22281996da609206a84df860bc1261 \ + --hash=sha256:3b61b7169ababd7802f9568ed96142616a9118dd2be0d1866e920e77ec8fa92a \ + --hash=sha256:3d9908a48eb7416dc1f4524e69f1d32e5d90e3981e4e37eb0aa1cd18f9cfa2a4 \ + --hash=sha256:3dd4dce1c718e38081c8f35f323209d4c1df7d4db4bab1b5c88a6b4d12b74587 \ + --hash=sha256:4021b51936308aeea0367b8f006dc999ca02bc118a0cc78c303f50a2ff6afb91 \ + --hash=sha256:425c126c0dc43861e22cb1c14ba4c8e45d09516d0a3ae0a3f7494b79f5f233a3 \ + --hash=sha256:44531a36aa2264a1860089ffd4dce7baf875ee5a6079d5fb42e261c704ef7344 \ + --hash=sha256:48e377758516d262bde50c2584fc6c578af272559c409eecbdd2bae1601184d6 \ + --hash=sha256:49a03727c1bba9a97d3e93c9f93ca03a57300f484b6e935463099841261195d3 \ + --hash=sha256:568f416a4072fbfae453dcf9a99194bbb8bdeab718e08ee13dfa2ba0e4bebf29 \ + --hash=sha256:5b179331a481cb5529fca8b432d8d3c7001cb217513c94cd72d668d1248688a3 \ + --hash=sha256:5b6073099fb654e0a068ae678b10feff95c5cae95bbfcbfa7af669d361a8aa6b \ + --hash=sha256:5d2d94f1f5fcbe40838ac51a6ab5704a6f9ea42e72ceda48de5e6b898521da51 \ + --hash=sha256:5dff64413671b0d3e7d5918ea490bdccb97a4ad29b3f311ed423200b2203e01c \ + --hash=sha256:5e1d8c8b8f1d91cd08d8f4a3c2b067bfca6ec043d3ff36de0f3a715feeedf926 \ + --hash=sha256:5f8ca7f2bb6ba8348a3614c7918cc4bb73268c5ac2a207576b7afea19d3d9f64 \ + --hash=sha256:642f752c3eb117b105acbd87e2c143de710987e09860d674e068c4c2c441034f \ + --hash=sha256:65d2ccb7eabee90ce0503c17716fc77226be026dcc3e65cce859a30db715025b \ + --hash=sha256:693781c45a4033d31d4187d2436f5ac701e7bbfe5df40d917736108c1cc7436e \ + --hash=sha256:694976222c711d1d00ba131904beb60534f93966562f64440d0c9d41b8cdb440 \ + --hash=sha256:697753042d57f4bf7122cab985bf15d0cef23c770864580f5af4f52023a56bd6 \ + --hash=sha256:6de499a1a44e7de70735d0b39f67c8f25eb3d91eb3103be99ca0fa882cdd987d \ + --hash=sha256:6fc0e2337d1a4c3e6acafda6a78a39d4c14caea625124817420abceed36e2415 \ + --hash=sha256:7a4a94eb787e606d0a09404b9c38c113d3b099d508021faa615d70a0131907ce \ + --hash=sha256:7b5e8fe4de30df199155baaf64f2fcd604f4c678ed20910db8e2c66dc4b11603 \ + --hash=sha256:7bfdc049127717581866fa4708791220970ce291c23e28ccf3922c700740fdc0 \ + --hash=sha256:7f9120f7093c2a32d9647abcaf21e6ad275b4fbec5b55969f978b1a97c7c86bf \ + --hash=sha256:8057c98e0c8472d8846b9c79f56766bcc57e3e8ac7bfd510482332366c56c591 \ + --hash=sha256:80dd4c21b0f6237676449c6baaa1039abae86b91636b6c91a7f8e61c87f89540 \ + --hash=sha256:82611aeec80eb144416956ec85b6ca45a64d76429c1ed46ae1b5f86c6e0c9a26 \ + --hash=sha256:8542f41a62bcc58fc7f11cf7c90e0ec324ce44950003feb70640fc2a9092c32a \ + --hash=sha256:87797e645d9d8e222e04160ee32aa06bc5c163e8499f24db719e7852ec23093a \ + --hash=sha256:87b9aab6d6ed88235aa2970294f496ff1a1f9adcd724d800e9b952395a80ffd9 \ + --hash=sha256:90455115e5da1c3c51ab619ac57f877da8fd6d73c05aacd125c5ae9819582aba \ + --hash=sha256:90751b8eed69435bac9ff4e3d2f6b3af1f57e37ecb0fbeee59c0174c9e2d41df \ + --hash=sha256:96d604498a7c782cb15a51c406acaea70d8c027ee6b90c569baa6e7b93073679 \ + --hash=sha256:9ae8dd55c8e6c4257eae3a20fd2c8f41edaea5992ed67156642493b8daf3cecc \ + --hash=sha256:9af5e68ee47d6534d36791bbe9b646d2a7c7deb6fc24d7943628edfbb3581f29 \ + --hash=sha256:9bf9f7a65e7aa20dd764151fb3d616c81088f91f8df39c3893a536e279b4b984 \ + --hash=sha256:9d4c940f02f49483b18b079d1c27ab948721852b281f8b015c058100e9421dd1 \ + --hash=sha256:a19884d2ee70b06d9204b2727a7b9f983d0c684c650254679e716b0b77920632 \ + --hash=sha256:a1e53262fd202e4b40b70c3aff944a8155059beedc8a89bba9dc1f9ef06a1b56 \ + --hash=sha256:a2212ad43c0833a873d0fb3c63fa1bacedd4cf6af2fee62bf4b739ceec3ab239 \ + --hash=sha256:a45530014d7a1e09f4a55f4f43097ba0fd155089372e105e4bff4ca76cb1b168 \ + --hash=sha256:a949eee43d3782f2daae4f4a2819b2cb9b0c5d3b7f7a927067cc84dafdbb9f88 \ + --hash=sha256:add1da70de90a2569c5e15249ff76a631ccacfe198375eead4aadf3b8dc849dc \ + --hash=sha256:b04be762396457bef43f3597c991e192ee7da460a4953d7e647ee4b1c28e7046 \ + --hash=sha256:b0d95340658b9d2f11d9697f59b3814a9d3bb4b7a7c20b131df4bcef464037c0 \ + --hash=sha256:b1a6102b4d3ebc07dad44fbf07b45bb600300f15b552ddf1851b5390202ea2e3 \ + --hash=sha256:b556c85915d8efaed322bf1bdae9486aa0f3f764195a0fb6ee962e5c71ef5ce1 \ + --hash=sha256:b903a4dfee7d347e2d87697d0713be59e0b87925be030c9178c5faa58ea58d5c \ + --hash=sha256:b928f30fe49574253644b1ca44b1b8adbd903aa0da4b9054a6c20fc7f4092a25 \ + --hash=sha256:bb4f7475e359992b580559e008c598091c45b5088f28614e855e42d39c2f1033 \ + --hash=sha256:bbe7d4cecacb439e2e2a8a1a7b935c25b812af7a5fd26503a66dadf428e79ec1 \ + --hash=sha256:c014c7ea7fb775dd015b2d3137378b7be0249a448a1612268b5a90c2d81de04d \ + --hash=sha256:c19b90316ad3b24c69cd78d5c9b4f3aa4497643685901185b65166293d36a00f \ + --hash=sha256:c685f2d80bb67ca8c3837823ad76196b3694b0159d232206d1e461d3d434666f \ + --hash=sha256:d32764c6c9aafb7fb55366a224756387cd50bfa720f32b88e0e6fa45b27dcf29 \ + --hash=sha256:d60ac9663f44168038586cab2157e122e46bdef09e9368b37f2d82d354c23f72 \ + --hash=sha256:e3531d63d3bdfa7e3ac5e9b27b2dd7ec9df3206a98e0b3445fa906f233264c57 \ + --hash=sha256:e636b3c5f61da31a92bf0d91da83e58fdfa96f178ba682f11d24f31944cdd28c \ + --hash=sha256:ea37047c6b367fd4bd632bff8077449b8fa034b69e812a18e0132a00fae6e808 \ + --hash=sha256:f33ed1a2bf1997a36661874b017f5c4b760f41266341af36febaf271d179f6d7 \ + --hash=sha256:f76c1e3fe7d7c8afad7ed193f89a292e1999608170dcc9751a7462a87dfd5bc0 \ + --hash=sha256:f9444f105664c4ce47a2a7171a2418bce5b7bae45fb610f4e2c36045d85911d3 \ + --hash=sha256:fc290605db2a917f6e81b0e1e0796469871f5af381ce15c604a3c5c7e51cb730 \ + --hash=sha256:fc353029f176fd2b3ec6cfc71be166aba1936fe5d73dd1992ce289ca6647a9aa + # via xtrace-ai-sdk +aiosignal==1.4.0 \ + --hash=sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e \ + --hash=sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7 + # via aiohttp +annotated-types==0.7.0 \ + --hash=sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53 \ + --hash=sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89 + # via pydantic +anyio==4.12.1 \ + --hash=sha256:41cfcc3a4c85d3f05c932da7c26d0201ac36f72abd4435ba90d0464a3ffed703 \ + --hash=sha256:d405828884fc140aa80a3c667b8beed277f1dfedec42ba031bd6ac3db606ab6c + # via + # httpx + # openai +attrs==25.4.0 \ + --hash=sha256:16d5969b87f0859ef33a48b35d55ac1be6e42ae49d5e853b597db70c35c57e11 \ + --hash=sha256:adcf7e2a1fb3b36ac48d97835bb6d8ade15b8dcce26aba8bf1d14847b57a3373 + # via aiohttp +boto3==1.42.68 \ + --hash=sha256:3f349f967ab38c23425626d130962bcb363e75f042734fe856ea8c5a00eef03c \ + --hash=sha256:dbff353eb7dc93cbddd7926ed24793e0174c04adbe88860dfa639568442e4962 + # via xtrace-ai-sdk +botocore==1.42.68 \ + --hash=sha256:3951c69e12ac871dda245f48dac5c7dd88ea1bfdd74a8879ec356cf2874b806a \ + --hash=sha256:9df7da26374601f890e2f115bfa573d65bf15b25fe136bb3aac809f6145f52ab + # via + # boto3 + # s3transfer +cbor2==6.1.4 \ + --hash=sha256:01ecc79a28f33d17331943ce508fc1e21f4b06553c73f874f4c77120d72b2ef9 \ + --hash=sha256:0859a0837e6e2d4fe5f5b849f6475797e4db545da98c19db4b1d3487bd47aa22 \ + --hash=sha256:164ca22b509408435b2d8236c80c964e4fc77c085ab034569cd04c40d5cc8883 \ + --hash=sha256:179a794bf4be1d46ff190695929f65f0b42019c156919846ae539d2a7ec42e54 \ + --hash=sha256:1fc15061553e4494dc10883237501e3402c645fe509248dd698e1faf2460d68b \ + --hash=sha256:2310f07db3f9ba26f2a623774ff9f3dc7185af54f732ea119785a6b1bf7e1e7e \ + --hash=sha256:239db0f92d537fd29eaec4e40195fc3b2b48bc34a5887059658162489a9eb6ae \ + --hash=sha256:24da0a481294ac416e1e369e2d204b2b1d993cbd082d0d99fa3d6f5f27ae5e69 \ + --hash=sha256:28fa5db05a7eae8fd80709959988d8a7f12838c6d4e5c58ec951414058641195 \ + --hash=sha256:310f3dfb296ba48fe9b63c5cf26e691e3548a1eae6901d2f0c18e941d151f220 \ + --hash=sha256:316e217a496640418d3137483279d0e70053b000cdd4b52a4dbf20ea478bc40a \ + --hash=sha256:32a4663425fbca4a4a7aa918eb5789d844c406439e58424cf34511f79f559242 \ + --hash=sha256:36ae16d64b1f7b620c1af748e7b6947e20069ef80eee56871c5fbb84cc635905 \ + --hash=sha256:3f4a434c36bb0d33aeb48ddae8e8b673ca7e1f14545ee7cf4a4c7c39380ea9a2 \ + --hash=sha256:4903f24e0f9087275a0b6606c8b0aa586277001d51e4844fcdbc5b7211330aa8 \ + --hash=sha256:4bd29f21529e279d50fc14f1a811f7b05b4d8e66a7969163cce98983b6817245 \ + --hash=sha256:553a46bda7d09552631a714e22b91e6ff2c867ecd91511596ce290d8879b8d5b \ + --hash=sha256:598710183daae69cbdeb177a870ec64aa601de8138a61491fd256826d15a860f \ + --hash=sha256:5b99305d4013867e059f147752b95f728680682ab03d75a3f4dcfbb270d8dfe9 \ + --hash=sha256:5e6c76004d674ad1c620660cb0bc5a8a0b72a5d8c7b70926d8e09e6d7e87332f \ + --hash=sha256:69978901302ecbc8cda57b520487c5c5240ed217de783eb7728fceb258311d76 \ + --hash=sha256:6abcf072b8c0fdc8ad7902ee26a906cafbf3427d026b662ff21166a253f85e18 \ + --hash=sha256:71fcf4f237d68bf4445bf45070f36f82b333f2e6a62612aa2c256683b51378a9 \ + --hash=sha256:7336ff4cb7d161ec43b65eef43bf3e9bcab44bd152efb54dd637b7afe711254f \ + --hash=sha256:7deccc50fd0b55c4c7dd265b144c5358a645121e457c0ae3722b5ad59832b257 \ + --hash=sha256:8156fdeb73c3ff6c8cf67ad414fb5c887cd708ff0af6d61f62629f41cb4c17b2 \ + --hash=sha256:855764e02dc60ab9413acd044e997c3170000fdea6155d6c43a923a1d966dbe6 \ + --hash=sha256:8f1019494b0ec81a3df3ebb01b6acb446d5b946fe35845b1726379abd66a71da \ + --hash=sha256:9b904b8d0f4ddac9259197d21d121fae4cb8b555700d65bc12c5d46a2e6c2025 \ + --hash=sha256:ad4efa23fee6447e56a269191044e06eb39e809458bcd674e164fe9445feafd0 \ + --hash=sha256:bd20ecc5c8ece24db952e48a91c8c47319eaa6358af707c85ac2bb388a79abc8 \ + --hash=sha256:c08b9c7d2ea013e24a0cb819b872b0119dde404f64a1182c0b24095b7bba781f \ + --hash=sha256:c0f5f2d6d3b58e44146860c049f3c082207a4005588b8926d51bf937ab66773c \ + --hash=sha256:c204a75f91f8cd9ed0881f6b88ec395c59aeac9fcf4d08155e7f899db2a1c46e \ + --hash=sha256:c48a7c938fc5fa5300ff82b5df09068dcb4838685ae8556b5ee8279d74f97ab4 \ + --hash=sha256:c6b28b928c5f2dbf47dffa12dce9c8e36fe6ac1c1358bc326499c0736263b66f \ + --hash=sha256:cc8cd300e236e9797b2e1ce306109dc481fcccf78bfa2682bf36d99e6eab1ec6 \ + --hash=sha256:d2560c2ba6a95904ba2a0ca257af878c4344409d9b46d8e646d8ebb617b1e0dd \ + --hash=sha256:d9ada5a6ccfbb8ea7a3aa2aeb028421b52d8e0cd9323f0a2aeaa9c09d25fbce2 \ + --hash=sha256:e1fe2d62c50df290576280b18247ec63486f78be73e285bae269c2456c6ddff0 \ + --hash=sha256:f3fc7d15cba4174373df2496070faa4a927fe3ed772130d281808120aec7b61c + # via xtrace-ai-sdk +certifi==2026.2.25 \ + --hash=sha256:027692e4402ad994f1c42e52a4997a9763c646b73e4096e4d5d6db8af1d6f0fa \ + --hash=sha256:e887ab5cee78ea814d3472169153c2d12cd43b14bd03329a39a9c6e2e80bfba7 + # via + # httpcore + # httpx + # xtrace-ai-sdk +cffi==2.1.1 ; platform_python_implementation != 'PyPy' \ + --hash=sha256:046bfc24911b37851ee1b51aab8bffe713d89c68c6a057b09484ce9fd5f69b4e \ + --hash=sha256:06c72bb76605a4b0cd0aad6930b69d4baf7dd5d806cfc409b824191099700e66 \ + --hash=sha256:0beceaabe56af686895136a2de78db54ecd8e4046b236b8fd6d6cb61389e9bf2 \ + --hash=sha256:154852545011f779917b11c78db2358d095da62a9a172b78ad0a583ee5adc0d0 \ + --hash=sha256:19ee6127ee34de7d83ce3d371ebc5ed91addbdcc39f9ab15ce4eb35a4e534971 \ + --hash=sha256:1a18a57b58cfb21fc28d72e876acf10eaed67a1ed96226f92af4df681d571c4c \ + --hash=sha256:1aa5645c30469b09530c4ebca77ebf8f17618293c58f8549cb1a543a50236e7d \ + --hash=sha256:208f941bb9d18e768138677f0a6d2ce01f590df56043dda1df1535ac57c88517 \ + --hash=sha256:210019b6c7cf07f081b4c54635c8cf744377001350e29cc0f81c4377b4797735 \ + --hash=sha256:246fa40ce8645a614ff682e0b70f37134e460eaf93a775e0cbe3cca585a67a80 \ + --hash=sha256:25792eac27877609e7bb06d42ff88278a6624fff2ba9bbb523c09616b117e80f \ + --hash=sha256:27350daa11d4f10c540e6e89dada4c54feb7256ad03e9a4dc075ebad7ba360d1 \ + --hash=sha256:28907ab9bfb6aa13184cfc17c6b8e1023c5ab6fd7076d8c20a35e59fe04f8f29 \ + --hash=sha256:2ae64be792b8966f2c69538199728b290e34726562896df1e5dc8ffd8d8188e8 \ + --hash=sha256:31348097ff5bbe827ccc41795d4dd099d9f0625e7def00ee653c137a490c2a6c \ + --hash=sha256:3143d81e29e1e20a9ce10901ec369012947876596f75a222235965f2b7ae832e \ + --hash=sha256:3222ba5d678f80a030e6afbcc33dc1ae5cb45facabb61cee2c7016b8432fde48 \ + --hash=sha256:3311ed60d36f83378794e1009ac6258bafbf81f7888b4caa7b35a521e3f95813 \ + --hash=sha256:334644fbac4eff73d985a17a91226df55d0f394160c4cfb880e084c8f7161cac \ + --hash=sha256:34e261f78cb6ceaaa36f42f2613f4380d94d9c759a9c73c769ee6e0247364632 \ + --hash=sha256:363e05fa78e15116c3c32c210ee36884fd6b9afa6d440e47112c3bd511d64cb6 \ + --hash=sha256:398aff33cee2767e3e781d2554c54bd0dff386bb437581e0d8011fde1a942ec1 \ + --hash=sha256:42a494cee34437f05546455144f2b5d9ac09b1face62bcfce597d2e521066688 \ + --hash=sha256:42f6930c31dc7f50732c9ae793c2786c7b6b044195967bbdde40bb9be81c4cc0 \ + --hash=sha256:456a61fa52d579ebf9df2e9552ead5129855dbaff6c1e5a9b1bc408809bdc062 \ + --hash=sha256:471cee653ae88de62096552e6d24ccb4a5adb8c8c9f10b5054d0122c15bf2779 \ + --hash=sha256:49cbc70e6542d4ccccb936558d1064a8012541e78f821f955cff24e357776c94 \ + --hash=sha256:4a7c934f7360e8cd64fe9efadcbd10c7c6364f531e432b9a4bf5ccbc9e0e8b50 \ + --hash=sha256:4be96343e422f2dfcd12ab5c9f5aebe03f82f737c6bffeca6830b3875cb44aab \ + --hash=sha256:4f42141fc14250de6dde5ee7ea4432be017252d91f19c5ad043c084cea629cac \ + --hash=sha256:507a24c282e0f42f8ed737cf048572cbf580468da5555764a8331735e9c736b6 \ + --hash=sha256:51b31d1c98274844cfd7838ce00bfc27c7423a4dc00fc0772fc3331c2cc90676 \ + --hash=sha256:58acb8ab8e295e6c5ea12f888cbb13cf21511ef2a3303a23f4325c29d17fe5c1 \ + --hash=sha256:5c58fe613dc5e5336357eff555824a314d8e43282600435c8d1cb6a7a2fedd13 \ + --hash=sha256:5e7cecbaadb83884793e05828cee59b210b24583b9c7425d0ba6a754fe22eb4e \ + --hash=sha256:616f097f2fe415bc92a247f02e11f634e1f9e9a83d327e3c915c15089c87869e \ + --hash=sha256:63bbfd5ded17c4840ac07cd8f1c21ba9d9708141f840b324f422f41b207e3973 \ + --hash=sha256:64faea20f4e2613363a1a9b9c7dd73058f3ecd00133a511e72ad7c511658f527 \ + --hash=sha256:661c298b4821edebead0c91edd2b00374d67ad7c5a1f7a91d4442633b79d6a72 \ + --hash=sha256:68e62fe11f30d5ca8289242866f0a5291402d8529ca2178ab8afc5c9694ae890 \ + --hash=sha256:6a8dddef476fab96d066d578fc88526767b836ab5ab21754e1d5bf3879c31c7c \ + --hash=sha256:6e192623c49c94421616a5778fba35cf0d5a8d000650c1967ef4448ee5cdd990 \ + --hash=sha256:7225e4514edb64eb6740324353e0da0711954fd8d7da4576755b1c6e09b697cd \ + --hash=sha256:770de9db11e84213beec501cfcaa013b019820ca881e03344dea5844f7876d94 \ + --hash=sha256:7750c6449dff7864bb9bb27ddfb0267756189201a3afc911d82b3caacd70dfc3 \ + --hash=sha256:7bde5e4cc5c10140859842b9d383af292b22639a4dffb725314baf45968cef80 \ + --hash=sha256:7da0c5eff80f0197f3b3d1232ec5a682a9325f4ae9016a78f5f5ca35f9ced1f5 \ + --hash=sha256:7dbb61fe3a7699468030f71bbe5f8a0e326a151daa91beb11a6fc1f980c55e1c \ + --hash=sha256:811bd1e21d32de12efca32393a0ab3f5133b54fce9bd44b8bd77ab07da14bf6a \ + --hash=sha256:8ef53b2de9bcb9197d31854256575d59dbac0cba72ac627bb291ef5eceb74be4 \ + --hash=sha256:937c0052c05a31ca1daf18de3158eed4dbfcb9cc107adbea227728d647be701e \ + --hash=sha256:9d2055050ea716bd38b7f7f1579c275386646b4894c155a3e2f3cd62ed41b7c6 \ + --hash=sha256:a2d7755bef5a12ed488f4ef1f1b69ee9191d7396083b755a5d2295f6edb4768b \ + --hash=sha256:a4f00aa42f75d6e4595e8866e748cc1705adc0cddfeb2ca86d0d03993d63ba03 \ + --hash=sha256:a6e721d4b0e45d5b65e87534470e67b18dcd092c83f68fba09f152b9cbc061af \ + --hash=sha256:a730a083190634c65cca36ba5f489531576ebd79bcd5c8e172130f6453127231 \ + --hash=sha256:a931079504ecc49efed7744c476a5c343a92fabf66dec2db95edb1b2fdc770e2 \ + --hash=sha256:aa9511c62d14da7aacc9b4bf51f3f697a621e83b2d6919008243c3aad168eea3 \ + --hash=sha256:ab36d55f9ed2d067327667c2fea18dda018eb628dd6347aa01dda6cf1f5d3836 \ + --hash=sha256:ad2c86c495b899d862ea0f4b42891b8713a3bd45dd4105c7fd51c2a72f39f3a5 \ + --hash=sha256:aeae0e330c9f6acd681f647d46cefd30c29f93e3392882e792e82080c9691399 \ + --hash=sha256:b0431303acaea1089ad4b3e9ce4e6518193def1118d4073ca848635ee4ea2e96 \ + --hash=sha256:b5bdfd1c873d4e093aabc0ca84c4ca6dbc4f752afb5c86f146d9742580c9da2e \ + --hash=sha256:c1453022f490d2459a11819d83ad1d586e9ff65a12ac3e705ffebd46d3685dcf \ + --hash=sha256:c26608d2222fb1e94487e4a387d85f13eb55d5ed725cb25a0c589ac4ee60e7bc \ + --hash=sha256:c7659f22557c5a0bc4855cd635f55edec690cc008a40768527762cb9fb263455 \ + --hash=sha256:c8c69575568085ba0b1b10c0249d779a214aea6f6522e949a0fc9fb0fcb449d0 \ + --hash=sha256:c8d2c9fd1f2d16f780d15127abb050d13d1a76c03a4bd87d7e4980e45e511e12 \ + --hash=sha256:cc572dace3f60ef98d7b12ff411d20f5362feb31a0439eab0085bbfd349982d7 \ + --hash=sha256:d18e5ac0f2f03f4f518d3e23db0f0cad7faa1da8620e9c09461d443bbf6e6692 \ + --hash=sha256:d28630f5854ab07ab1fd4aba756de52326c82e6be15d414b12793f1975048b54 \ + --hash=sha256:d9c275eaacd24aa73f94ffd6de08fc3f932424d8b6c376f4bed7cde376fe7bc3 \ + --hash=sha256:da0e573f9f97159390c89d9f1a9e41908b66d408cc5b58d08cf3847d844c531b \ + --hash=sha256:dd31f52ea1086513bb9df30f8fcee9b8918323ae067a3d5b78bc826a000712be \ + --hash=sha256:dddad92b554513a31f272570678ba307fb9f618f05e3d4a5eacafff9eae03e1d \ + --hash=sha256:df423d40ee8654634421812bc3b196da3f9bd7d32929da813f8394c4348a5358 \ + --hash=sha256:df913725b79db7bcf03448f36b7bf8815363417d5b58deecf9305e3e30f0f21a \ + --hash=sha256:e0bcb7e0f677f543555d2adff3bf19c05f66cdb4796e5ff602442ab2fe3c4ef7 \ + --hash=sha256:e2d65b31f36619cda3999b78b2aa9632e76b78448e7a56fc4240824200e7c4fc \ + --hash=sha256:e6e8cff14d6fb0be70a09c0bdc58096f501952d04624ebf867e0e56da2df8960 \ + --hash=sha256:f16c709686a78c727bbbf059f92b0bf41c6fc60deec706d2dc19f529175a6125 \ + --hash=sha256:f24fb43132a4c6b4cb4eb029492919b2db645be6808d738f244fd146c03c32cb \ + --hash=sha256:f53e442b08449d42821fa4a4fba000095af9f62742a500f978a9f557ec44339a \ + --hash=sha256:f5cfbc5fe74540d335175b656c725d74d90e3730c626d92575eea35029d9afaa \ + --hash=sha256:f81b3b8f3d4e343550fa4baa0e479bba9f2d29ce9c2e9b51d1ce1718d7442fcf \ + --hash=sha256:f8ec5e643a9a937f64e1999eb9f75d072263751912dc5cd06d3c85f8f44be7c3 \ + --hash=sha256:fb92203a88b3d3053034db775110081c49d28be6551923805e039924093761e4 \ + --hash=sha256:fcd22650c908d7b7da162bbfaab594a1227a15d1643a98c68b122ac642fa2264 + # via cryptography +colorama==0.4.6 ; sys_platform == 'win32' \ + --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ + --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 + # via tqdm +cryptography==50.0.1 \ + --hash=sha256:01f41478cf33fc605a6a089cd56d28b45c6c0b45a1928b61797f2621a04bac71 \ + --hash=sha256:05ba322c4da95b262a212c345af888ef2c37c88c0509756ea00a0e6d68850f23 \ + --hash=sha256:16c5ecd954b3330ebfb6605eca4fd952da8bef376551d5cc264534e3770a9ee6 \ + --hash=sha256:2a93d05e34d5f67fba6f891fe85d929999baa7195e853923ea6d7576c9e68c5e \ + --hash=sha256:2b34d76a652ea2b6faf777c35df230c5637842cd904e04f16230c3f9f03e4361 \ + --hash=sha256:2ebbfb0f1fed745e91796e3e1080a1440423fdae8ece1b995a1d80883a409054 \ + --hash=sha256:30a125032e5642a21ff816e021152bd4e7e94f03eff3f4b7fca41cd22bc3110f \ + --hash=sha256:330fbb252391c596f1ae42c5754449dc924e6ad012dca8efe0d703f9f2d12ec6 \ + --hash=sha256:359e62deae718bce96170e223fdcb6357e4fbd3bb7a3a75f4430763532560e49 \ + --hash=sha256:407fe2b6db00939c05c0e945e9914238f2f0a430974839429dafc82b1ee6bee5 \ + --hash=sha256:42be3bb70596b3abe4ac097b75be223e8b3ab614a0e5de068e3dcc54d71d6149 \ + --hash=sha256:4c4188f7c0cf655be5c06342b817ed0f9595b69ffa2b12026e5353eed29dea88 \ + --hash=sha256:51593d180cf6d179bde5c5d065bed81386b1f381656ae7d042b7ffc87a9895ad \ + --hash=sha256:51afcfceb15597cf2635068e4ac9a56b2abde622edde17f37d85fd7b5306497a \ + --hash=sha256:53e279950892dc102c6b4e52af03ae5ea92fac572a1ddab78ca73a997f62b69f \ + --hash=sha256:55d16b1ef3ee0958d893a977b19777887e546c9954ea81b200c3301a864013f2 \ + --hash=sha256:5dd9bda1c12b4162f6ff568eeb5e0ff956c28d14406e875cfe8a63a2d414ff20 \ + --hash=sha256:5fe002589592ed749ce77fe0695fcbd3500dd61d7d6db5858a7544c612fa8e45 \ + --hash=sha256:5fe939deeb161024a6be98229c953b6591fef1f41214497a78fe793a244c017f \ + --hash=sha256:693c99b49bd37d0d096e4334c10232c77248c415b98d35236094cdf96d57258b \ + --hash=sha256:76de83fbd91ac49c0feaaa983d0748fd7a53176afac5fb3bf7478d244f0eb527 \ + --hash=sha256:79bf008d1f9af6071c797ad133e39915dfee7614f18f18f4db9072eb715064a3 \ + --hash=sha256:804728ce710890870f3aaa344b2e161172d258d768ac139d02cfd9092d0d94e6 \ + --hash=sha256:8921d58f426793c5f1b47f0b59575780de9a095214958d0eb37d909593db8367 \ + --hash=sha256:8df2de9102026855887e4587084f6eabd80ed0f345b8ad8a7ac27ab9bf4723e0 \ + --hash=sha256:9cb3cb952cf5a8abd50c782a98a89d71699715e802fe349704b47f2425b42a94 \ + --hash=sha256:9dde0a357190eb3b1da1bb9ab750e9c85cba82ca5977aa0836cbb94e92611239 \ + --hash=sha256:9ebcdd5519be9b652a46f507817a74591774fc3d6923ac364e4dfa64e36b291b \ + --hash=sha256:a0b1a59e3a089064a0ec309e9428c8e3ae4e161419d20ac33600767e83fc658a \ + --hash=sha256:a255449073358275b64b67d3f595f268bbef70e72b6edb65e0c70c735bf739c9 \ + --hash=sha256:a8f40ea47330e71b594a7e246898f93177c259490c63183dbaf9e571d71ed9a5 \ + --hash=sha256:ac02b07824d4d1001bd4367599f839c19cb171924c796e52c23508ac14c2c0cc \ + --hash=sha256:aed8db4f6d71c51efb89530e12d9464e7bf2923d46c3205dc794a2a93f8c0648 \ + --hash=sha256:b8f852c65863251b9e3a1b8c150ce21e59b522dbb6a7d4bc80e680d38388e986 \ + --hash=sha256:be224a65493ec5b74a158ff22a5522ce4a5ca1e543c647a3a4730d4a09e5f959 \ + --hash=sha256:ca83d00d9e69cd5eb63f2e69c3a5a59e0cecae5ae14c6ae0b35830fe3b37bad0 \ + --hash=sha256:cbf74a81765ee67413503ca6e26dcc4f6f5a519822436cc0a1b97aab6c1b8a17 \ + --hash=sha256:d63ae8f6481fec907ac0f588eee8a90aefde112c633131fe540e5711ddbb5a4e \ + --hash=sha256:e22dfed744bd4002e909464cb23d2f0b05c6f3113a79ef2e9864a53db737c733 \ + --hash=sha256:e2ca8fd1b6b4b82a1c4cb02841d0837e3c12336c2e24b520ab8ab3b969733d8f \ + --hash=sha256:e74591e283fe6eb956416c929eb58262a719fe0311fd9054c62c3350ed8760d8 \ + --hash=sha256:f74455bb086a85d5e81246412602aaa97ed095e504cd40dd261ef50be42205bf \ + --hash=sha256:fb4b9672d389c738b175c4166e78310f8a70358886aacd9173ee03a85ffdc671 \ + --hash=sha256:fc3ed7ebd2a8c96f5b166de0ab9b624996bef3b07bbeb19364dfb78222c22c80 \ + --hash=sha256:fd3718b960d0b5dd213cdf03f3bcb7000e69dda0de8b956061947ff6bcff5558 \ + --hash=sha256:ff838d62ec1bfce4f9ba7fa16f4a7b554cd8d0c299e6be37502161a660c84eef + # via + # pyopenssl + # xtrace-ai-sdk +distro==1.9.0 \ + --hash=sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed \ + --hash=sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2 + # via openai +frozenlist==1.8.0 \ + --hash=sha256:0325024fe97f94c41c08872db482cf8ac4800d80e79222c6b0b7b162d5b13686 \ + --hash=sha256:032efa2674356903cd0261c4317a561a6850f3ac864a63fc1583147fb05a79b0 \ + --hash=sha256:03ae967b4e297f58f8c774c7eabcce57fe3c2434817d4385c50661845a058121 \ + --hash=sha256:06be8f67f39c8b1dc671f5d83aaefd3358ae5cdcf8314552c57e7ed3e6475bdd \ + --hash=sha256:073f8bf8becba60aa931eb3bc420b217bb7d5b8f4750e6f8b3be7f3da85d38b7 \ + --hash=sha256:07cdca25a91a4386d2e76ad992916a85038a9b97561bf7a3fd12d5d9ce31870c \ + --hash=sha256:09474e9831bc2b2199fad6da3c14c7b0fbdd377cce9d3d77131be28906cb7d84 \ + --hash=sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d \ + --hash=sha256:0f96534f8bfebc1a394209427d0f8a63d343c9779cda6fc25e8e121b5fd8555b \ + --hash=sha256:102e6314ca4da683dca92e3b1355490fed5f313b768500084fbe6371fddfdb79 \ + --hash=sha256:11847b53d722050808926e785df837353bd4d75f1d494377e59b23594d834967 \ + --hash=sha256:119fb2a1bd47307e899c2fac7f28e85b9a543864df47aa7ec9d3c1b4545f096f \ + --hash=sha256:154e55ec0655291b5dd1b8731c637ecdb50975a2ae70c606d100750a540082f7 \ + --hash=sha256:168c0969a329b416119507ba30b9ea13688fafffac1b7822802537569a1cb0ef \ + --hash=sha256:17c883ab0ab67200b5f964d2b9ed6b00971917d5d8a92df149dc2c9779208ee9 \ + --hash=sha256:1a7fa382a4a223773ed64242dbe1c9c326ec09457e6b8428efb4118c685c3dfd \ + --hash=sha256:21900c48ae04d13d416f0e1e0c4d81f7931f73a9dfa0b7a8746fb2fe7dd970ed \ + --hash=sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b \ + --hash=sha256:2552f44204b744fba866e573be4c1f9048d6a324dfe14475103fd51613eb1d1f \ + --hash=sha256:27c6e8077956cf73eadd514be8fb04d77fc946a7fe9f7fe167648b0b9085cc25 \ + --hash=sha256:28bd570e8e189d7f7b001966435f9dac6718324b5be2990ac496cf1ea9ddb7fe \ + --hash=sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143 \ + --hash=sha256:29548f9b5b5e3460ce7378144c3010363d8035cea44bc0bf02d57f5a685e084e \ + --hash=sha256:2c5dcbbc55383e5883246d11fd179782a9d07a986c40f49abe89ddf865913930 \ + --hash=sha256:2dc43a022e555de94c3b68a4ef0b11c4f747d12c024a520c7101709a2144fb37 \ + --hash=sha256:2f05983daecab868a31e1da44462873306d3cbfd76d1f0b5b69c473d21dbb128 \ + --hash=sha256:33139dc858c580ea50e7e60a1b0ea003efa1fd42e6ec7fdbad78fff65fad2fd2 \ + --hash=sha256:33f48f51a446114bc5d251fb2954ab0164d5be02ad3382abcbfe07e2531d650f \ + --hash=sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746 \ + --hash=sha256:342c97bf697ac5480c0a7ec73cd700ecfa5a8a40ac923bd035484616efecc2df \ + --hash=sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8 \ + --hash=sha256:39ecbc32f1390387d2aa4f5a995e465e9e2f79ba3adcac92d68e3e0afae6657c \ + --hash=sha256:3e0761f4d1a44f1d1a47996511752cf3dcec5bbdd9cc2b4fe595caf97754b7a0 \ + --hash=sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad \ + --hash=sha256:3ef2d026f16a2b1866e1d86fc4e1291e1ed8a387b2c333809419a2f8b3a77b82 \ + --hash=sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29 \ + --hash=sha256:4314debad13beb564b708b4a496020e5306c7333fa9a3ab90374169a20ffab30 \ + --hash=sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf \ + --hash=sha256:44389d135b3ff43ba8cc89ff7f51f5a0bb6b63d829c8300f79a2fe4fe61bcc62 \ + --hash=sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383 \ + --hash=sha256:4970ece02dbc8c3a92fcc5228e36a3e933a01a999f7094ff7c23fbd2beeaa67c \ + --hash=sha256:4e0c11f2cc6717e0a741f84a527c52616140741cd812a50422f83dc31749fb52 \ + --hash=sha256:50066c3997d0091c411a66e710f4e11752251e6d2d73d70d8d5d4c76442a199d \ + --hash=sha256:517279f58009d0b1f2e7c1b130b377a349405da3f7621ed6bfae50b10adf20c1 \ + --hash=sha256:54b2077180eb7f83dd52c40b2750d0a9f175e06a42e3213ce047219de902717a \ + --hash=sha256:5500ef82073f599ac84d888e3a8c1f77ac831183244bfd7f11eaa0289fb30714 \ + --hash=sha256:581ef5194c48035a7de2aefc72ac6539823bb71508189e5de01d60c9dcd5fa65 \ + --hash=sha256:5c1c8e78426e59b3f8005e9b19f6ff46e5845895adbde20ece9218319eca6506 \ + --hash=sha256:5d63a068f978fc69421fb0e6eb91a9603187527c86b7cd3f534a5b77a592b888 \ + --hash=sha256:6da155091429aeba16851ecb10a9104a108bcd32f6c1642867eadaee401c1c41 \ + --hash=sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608 \ + --hash=sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa \ + --hash=sha256:778a11b15673f6f1df23d9586f83c4846c471a8af693a22e066508b77d201ec8 \ + --hash=sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1 \ + --hash=sha256:8009897cdef112072f93a0efdce29cd819e717fd2f649ee3016efd3cd885a7ed \ + --hash=sha256:8585e3bb2cdea02fc88ffa245069c36555557ad3609e83be0ec71f54fd4abb52 \ + --hash=sha256:878be833caa6a3821caf85eb39c5ba92d28e85df26d57afb06b35b2efd937231 \ + --hash=sha256:8b7b94a067d1c504ee0b16def57ad5738701e4ba10cec90529f13fa03c833496 \ + --hash=sha256:8d92f1a84bb12d9e56f818b3a746f3efba93c1b63c8387a73dde655e1e42282a \ + --hash=sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3 \ + --hash=sha256:92db2bf818d5cc8d9c1f1fc56b897662e24ea5adb36ad1f1d82875bd64e03c24 \ + --hash=sha256:957e7c38f250991e48a9a73e6423db1bb9dd14e722a10f6b8bb8e16a0f55f695 \ + --hash=sha256:96153e77a591c8adc2ee805756c61f59fef4cf4073a9275ee86fe8cba41241f7 \ + --hash=sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4 \ + --hash=sha256:97260ff46b207a82a7567b581ab4190bd4dfa09f4db8a8b49d1a958f6aa4940e \ + --hash=sha256:974b28cf63cc99dfb2188d8d222bc6843656188164848c4f679e63dae4b0708e \ + --hash=sha256:ac913f8403b36a2c8610bbfd25b8013488533e71e62b4b4adce9c86c8cea905b \ + --hash=sha256:b2a095d45c5d46e5e79ba1e5b9cb787f541a8dee0433836cea4b96a2c439dcd8 \ + --hash=sha256:b3210649ee28062ea6099cfda39e147fa1bc039583c8ee4481cb7811e2448c51 \ + --hash=sha256:b4dec9482a65c54a5044486847b8a66bf10c9cb4926d42927ec4e8fd5db7fed8 \ + --hash=sha256:b6db2185db9be0a04fecf2f241c70b63b1a242e2805be291855078f2b404dd6b \ + --hash=sha256:bac9c42ba2ac65ddc115d930c78d24ab8d4f465fd3fc473cdedfccadb9429806 \ + --hash=sha256:bf0a7e10b077bf5fb9380ad3ae8ce20ef919a6ad93b4552896419ac7e1d8e042 \ + --hash=sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b \ + --hash=sha256:c8d1634419f39ea6f5c427ea2f90ca85126b54b50837f31497f3bf38266e853d \ + --hash=sha256:c9a63152fe95756b85f31186bddf42e4c02c6321207fd6601a1c89ebac4fe567 \ + --hash=sha256:cb89a7f2de3602cfed448095bab3f178399646ab7c61454315089787df07733a \ + --hash=sha256:cba69cb73723c3f329622e34bdbf5ce1f80c21c290ff04256cff1cd3c2036ed2 \ + --hash=sha256:cee686f1f4cadeb2136007ddedd0aaf928ab95216e7691c63e50a8ec066336d0 \ + --hash=sha256:cf253e0e1c3ceb4aaff6df637ce033ff6535fb8c70a764a8f46aafd3d6ab798e \ + --hash=sha256:d1eaff1d00c7751b7c6662e9c5ba6eb2c17a2306ba5e2a37f24ddf3cc953402b \ + --hash=sha256:d3bb933317c52d7ea5004a1c442eef86f426886fba134ef8cf4226ea6ee1821d \ + --hash=sha256:d4d3214a0f8394edfa3e303136d0575eece0745ff2b47bd2cb2e66dd92d4351a \ + --hash=sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52 \ + --hash=sha256:db1e72ede2d0d7ccb213f218df6a078a9c09a7de257c2fe8fcef16d5925230b1 \ + --hash=sha256:e25ac20a2ef37e91c1b39938b591457666a0fa835c7783c3a8f33ea42870db94 \ + --hash=sha256:eaa352d7047a31d87dafcacbabe89df0aa506abb5b1b85a2fb91bc3faa02d822 \ + --hash=sha256:eab8145831a0d56ec9c4139b6c3e594c7a83c2c8be25d5bcf2d86136a532287a \ + --hash=sha256:ec3cc8c5d4084591b4237c0a272cc4f50a5b03396a47d9caaf76f5d7b38a4f11 \ + --hash=sha256:edee74874ce20a373d62dc28b0b18b93f645633c2943fd90ee9d898550770581 \ + --hash=sha256:eefdba20de0d938cec6a89bd4d70f346a03108a19b9df4248d3cf0d88f1b0f51 \ + --hash=sha256:f21f00a91358803399890ab167098c131ec2ddd5f8f5fd5fe9c9f2c6fcd91e40 \ + --hash=sha256:f4be2e3d8bc8aabd566f8d5b8ba7ecc09249d74ba3c9ed52e54dc23a293f0b92 \ + --hash=sha256:f6292f1de555ffcc675941d65fffffb0a5bcd992905015f85d0592201793e0e5 \ + --hash=sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4 \ + --hash=sha256:fa47e444b8ba08fffd1c18e8cdb9a75db1b6a27f17507522834ad13ed5922b93 \ + --hash=sha256:fb30f9626572a76dfe4293c7194a09fb1fe93ba94c7d4f720dfae3b646b45027 \ + --hash=sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd + # via + # aiohttp + # aiosignal +gmpy2==2.3.0 \ + --hash=sha256:04112c8c572499877aa785e91a1b06a90ac65138638d8d331b78032b2fe76dd9 \ + --hash=sha256:089229ef18b8d804a76fec9bd7e7d653f598a977e8354f7de8850731a48adb37 \ + --hash=sha256:0a42ced3bc8a679daa46fba0d7b46f5549c5c1400adc005da76a9420821ad959 \ + --hash=sha256:122e37c3f2ced5ae30a904d3ba5cc9c9e65e752560911eac44c6e9f3a8a240a1 \ + --hash=sha256:17770d7a8fb06159563d63a69b95e9c935f94d078ec5b1876e64ae2f233cc97b \ + --hash=sha256:19022e0103aa76803b666720f107d8ab1941c597fd3fe70fadf7c49bac82a097 \ + --hash=sha256:23235c78f75f5d5349c5a28ba747abdabf8e43cb6c3e2d7296d9be07d5ee150c \ + --hash=sha256:2792ec96b2c4ee5af9f72409cd5b786edaf8277321f7022ce80ddff265815b01 \ + --hash=sha256:28422e9a137ac1ae60f906b7c1a9b99fc7191e6a8ced3c4222dd514b33d5556f \ + --hash=sha256:2d943cc9051fcd6b15b2a09369e2f7e18c526bc04c210782e4da61b62495eb4a \ + --hash=sha256:2f045c824c7ac28bb026e55b8a8c97a41ebbcdfbe1f065e47debb5d842baf667 \ + --hash=sha256:2f6b38e1b6d2aeb553c936c136c3a12cf983c9f9ce3e211b8632744a15f2bce7 \ + --hash=sha256:392d0560526dfa377c54c5c001d507fbbdea6cf54574895b90a97fc3587fa51e \ + --hash=sha256:399d8ed2168075837c9b8817808532d5d3fc197cd7b121822e97e8add3e60f7f \ + --hash=sha256:3c13f1078b30bbb95472f38e6213fb269be55ef37ede94fb9695fb1173e568a4 \ + --hash=sha256:404f1c46e9e3871c482d485f661b8b94e7b8bb7837e582d2dad961a3c8a9aa81 \ + --hash=sha256:4623e700423396ef3d1658efa83b6feb0615fb68cb0b850e9ac0cba966db34c8 \ + --hash=sha256:488a370c232b314c353982920fded3c275e44a4f55d48d690462ad31cbf971ae \ + --hash=sha256:51051b7327f384cbd6e3fecdb31280c3aab5648cc5f49a79d447b9851840eb65 \ + --hash=sha256:51dd9999a2cbfc4cd1d6166478c2e5749aa9cf5cf384349d2d8f9169463190f1 \ + --hash=sha256:566861a586dabffc4913b3a99f611240c966e9d5fab71cdb3147e759a035ebd2 \ + --hash=sha256:5779f6cf498dfea2dd486c6acff0b61b834d3c3a18724d09f297a495a2ba4794 \ + --hash=sha256:63203c8a31f2afb282e0535e517e8f53a7f63c490b51bb3e1e3dbad67533ca63 \ + --hash=sha256:65d8bf129b50b42644602e241e04ba3a27dafef0b07904964d8b4689704d97d7 \ + --hash=sha256:67606f3da678b3c2379dc606186867cde5061e44f96bf5a114cc6a08ea0d2fc7 \ + --hash=sha256:692289a37442468856328986e0fab7e7e71c514bc470e1abae82d3bc54ca4cd2 \ + --hash=sha256:713ba9b7a0a9098591f202e8f24f27ac5dd5001baf088ece1762852608a04b95 \ + --hash=sha256:71dc3734104fa1f300d35ac6f55c7e98f7b0e1c7fd96f27b409110ed1c0c47d2 \ + --hash=sha256:77d4d5b580339f31f89db3640453472d3c8c9308233db35bdccd2e8a93ae61fd \ + --hash=sha256:800d6bbb5ad1dac5ab13000c4a693332c8e27f169723e4b43f98ec8e15d41277 \ + --hash=sha256:884b200343db4eca4a918cff602ecd1686a8dfc44f4b0b7e7df21747edf4e168 \ + --hash=sha256:8b6399d2dd1229242b0e24ebf525f70fee0efa98f366990c7efafd32ce90855c \ + --hash=sha256:8d087b262a0356c318a56fbb5c718e4e56762d861b2f9d581adc90a180264db9 \ + --hash=sha256:9cedc4e27c543fa811175a00765bd284114fc6f813fcca2dbb24b383173115e2 \ + --hash=sha256:9d135dcef824e26e1b3af544004d8f98564d090e7cf1001c50cc93d9dc1dc047 \ + --hash=sha256:9d7cae51c2613884192901b3b7cf6546e4c70548589b58ef9eec7d3aaea88783 \ + --hash=sha256:9dcbb628f9c806f0e6789f2c5e056e67e949b317af0e9ea0c3f0e0488c56e2a8 \ + --hash=sha256:a12f5b508b940000b54dec3a2fdd3523e09f20b87093a190a55d9a139bb5eece \ + --hash=sha256:a5713e51e244470e986051ac2be4aa4d0594067216f07d76cd07234f8c25a2c6 \ + --hash=sha256:ab62738afef1d65e96d170efdd8e982caa9338591e8faa0ac6999bf574b5f441 \ + --hash=sha256:b308af421eadb1736befdde0f9ab9927d4a674596eef88a99faa98d75bba8f16 \ + --hash=sha256:baf62ccecb76a0c922af2f01c7fbb0b0f7606ed41fd7d59458656898a0dad5c9 \ + --hash=sha256:bb379412033b52c3ec6bc44c6eaa134c88a068b6f1f360e6c13ca962082478ee \ + --hash=sha256:bc09a8fddecf2eed7b6da1b877d96ec9582285f0bbc3bdf57f3dedfcfbef01f7 \ + --hash=sha256:bf9ac69c1346a6dc21e62261d916325be74d1aee97e87e14c90eedbb0171feba \ + --hash=sha256:cd5b92fa675dde5151ebe8d89814c78d573e5210cdc162016080782778f15654 \ + --hash=sha256:d2ed7b6d557b5d47068e889e2db204321ac855e001316a12928e4e7435f98637 \ + --hash=sha256:e900f41cc46700a5f49a4fbdcd5cd895e00bd0c2b9889fb2504ac1d594c21ac2 \ + --hash=sha256:f1843f2ca5a1643fac7563a12a6a7d68e539d93de4afe5812355d32fb1613891 \ + --hash=sha256:f35d6b1a8f067323a0a0d7034699284baebef498b030bbb29ab31d2ec13d1068 \ + --hash=sha256:f3770aa5e44c5650d18232a0b8b8ed3d12db530d8278d4c800e4de5eef24cac5 \ + --hash=sha256:f9b4cee1fa3647505f53b81dc3b60ac49034768117f6295a04aaf4d3f216b821 \ + --hash=sha256:fc6329b18bdbac682452ae403edf373be52794400643578863b5a751cc606e4f \ + --hash=sha256:fd9f4124d7dc39d50896ba08820049a95f9f3952dcd6e072cc3a9d07361b7f1f + # via xtrace-ai-sdk +h11==0.16.0 \ + --hash=sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1 \ + --hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86 + # via httpcore +httpcore==1.0.9 \ + --hash=sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55 \ + --hash=sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8 + # via httpx +httpx==0.28.1 \ + --hash=sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc \ + --hash=sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad + # via openai +idna==3.11 \ + --hash=sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea \ + --hash=sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902 + # via + # anyio + # httpx + # yarl +jiter==0.13.0 \ + --hash=sha256:04670992b576fa65bd056dbac0c39fe8bd67681c380cb2b48efa885711d9d726 \ + --hash=sha256:0733312953b909688ae3c2d58d043aa040f9f1a6a75693defed7bc2cc4bf2654 \ + --hash=sha256:07b75fe09a4ee8e0c606200622e571e44943f47254f95e2436c8bdcaceb36d7d \ + --hash=sha256:0a2bd69fc1d902e89925fc34d1da51b2128019423d7b339a45d9e99c894e0663 \ + --hash=sha256:0a8d76c7524087272c8ae913f5d9d608bd839154b62c4322ef65723d2e5bb0b8 \ + --hash=sha256:0b34c519e17658ed88d5047999a93547f8889f3c1824120c26ad6be5f27b6cf5 \ + --hash=sha256:0bf670e3b1445fc4d31612199f1744f67f889ee1bbae703c4b54dc097e5dd394 \ + --hash=sha256:0c365005b05505a90d1c47856420980d0237adf82f70c4aff7aebd3c1cc143ad \ + --hash=sha256:0e3a5f0cde8ff433b8e88e41aa40131455420fb3649a3c7abdda6145f8cb7202 \ + --hash=sha256:0f0c065695f616a27c920a56ad0d4fc46415ef8b806bf8fc1cacf25002bd24e1 \ + --hash=sha256:1211427574b17b633cfceba5040de8081e5abf114f7a7602f73d2e16f9fdaa59 \ + --hash=sha256:1317fdffd16f5873e46ce27d0e0f7f4f90f0cdf1d86bf6abeaea9f63ca2c401d \ + --hash=sha256:15db60e121e11fe186c0b15236bd5d18381b9ddacdcf4e659feb96fc6c969c92 \ + --hash=sha256:1f4748aad1b4a93c8bdd70f604d0f748cdc0e8744c5547798acfa52f10e79228 \ + --hash=sha256:1f8a55b848cbabf97d861495cd65f1e5c590246fabca8b48e1747c4dfc8f85bf \ + --hash=sha256:24ab43126d5e05f3d53a36a8e11eb2f23304c6c1117844aaaf9a0aa5e40b5018 \ + --hash=sha256:24dc96eca9f84da4131cdf87a95e6ce36765c3b156fc9ae33280873b1c32d5f6 \ + --hash=sha256:2b4972c6df33731aac0742b64fd0d18e0a69bc7d6e03108ce7d40c85fd9e3e6d \ + --hash=sha256:2c26cf47e2cad140fa23b6d58d435a7c0161f5c514284802f25e87fddfe11024 \ + --hash=sha256:2d08c9475d48b92892583df9da592a0e2ac49bcd41fae1fec4f39ba6cf107820 \ + --hash=sha256:3097d665a27bc96fd9bbf7f86178037db139f319f785e4757ce7ccbf390db6c2 \ + --hash=sha256:36ebfbcffafb146d0e6ffb3e74d51e03d9c35ce7c625c8066cdbfc7b953bdc72 \ + --hash=sha256:3b3fb8c2053acaef8580809ac1d1f7481a0a0bdc012fd7f5d8b18fb696a5a089 \ + --hash=sha256:3d744a6061afba08dd7ae375dcde870cffb14429b7477e10f67e9e6d68772a0a \ + --hash=sha256:41f92313d17989102f3cb5dd533a02787cdb99454d494344b0361355da52fcb9 \ + --hash=sha256:45f6f8efb2f3b0603092401dc2df79fa89ccbc027aaba4174d2d4133ed661434 \ + --hash=sha256:47455245307e4debf2ce6c6e65a717550a0244231240dcf3b8f7d64e4c2f22f4 \ + --hash=sha256:5467696f6b827f1116556cb0db620440380434591e93ecee7fd14d1a491b6daa \ + --hash=sha256:57aab48f40be1db920a582b30b116fe2435d184f77f0e4226f546794cedd9cf0 \ + --hash=sha256:597245258e6ad085d064780abfb23a284d418d3e61c57362d9449c6c7317ee2d \ + --hash=sha256:5a1aff1fbdb803a376d4d22a8f63f8e7ccbce0b4890c26cc7af9e501ab339ef0 \ + --hash=sha256:5d9b34ad56761b3bf0fbe8f7e55468704107608512350962d3317ffd7a4382d5 \ + --hash=sha256:632bf7c1d28421c00dd8bbb8a3bac5663e1f57d5cd5ed962bce3c73bf62608e6 \ + --hash=sha256:66aa3e663840152d18cc8ff1e4faad3dd181373491b9cfdc6004b92198d67911 \ + --hash=sha256:682161a67adea11e3aae9038c06c8b4a9a71023228767477d683f69903ebc607 \ + --hash=sha256:6c26a424569a59140fb51160a56df13f438a2b0967365e987889186d5fc2f6f9 \ + --hash=sha256:701a1e77d1e593c1b435315ff625fd071f0998c5f02792038a5ca98899261b7d \ + --hash=sha256:775e10de3849d0631a97c603f996f518159272db00fdda0a780f81752255ee9d \ + --hash=sha256:7772115877c53f62beeb8fd853cab692dbc04374ef623b30f997959a4c0e7e95 \ + --hash=sha256:7b88d649135aca526da172e48083da915ec086b54e8e73a425ba50999468cc08 \ + --hash=sha256:7bb00b6d26db67a05fe3e12c76edc75f32077fb51deed13822dc648fa373bc19 \ + --hash=sha256:7beae3a3d3b5212d3a55d2961db3c292e02e302feb43fce6a3f7a31b90ea6dfe \ + --hash=sha256:879e768938e7b49b5e90b7e3fecc0dbec01b8cb89595861fb39a8967c5220d09 \ + --hash=sha256:87ce0f14c6c08892b610686ae8be350bf368467b6acd5085a5b65441e2bf36d2 \ + --hash=sha256:8d76029f077379374cf0dbc78dbe45b38dec4a2eb78b08b5194ce836b2517afc \ + --hash=sha256:964538479359059a35fb400e769295d4b315ae61e4105396d355a12f7fef09f0 \ + --hash=sha256:9776ebe51713acf438fd9b4405fcd86893ae5d03487546dae7f34993217f8a91 \ + --hash=sha256:98fbafb6e88256f4454de33c1f40203d09fc33ed19162a68b3b257b29ca7f663 \ + --hash=sha256:9950290340acc1adaded363edd94baebcee7dabdfa8bee4790794cd5cfad2af6 \ + --hash=sha256:9d01ecc3a8cbdb6f25a37bd500510550b64ddf9f7d64a107d92f3ccb25035d0f \ + --hash=sha256:9da38b4fedde4fb528c740c2564628fbab737166a0e73d6d46cb4bb5463ff411 \ + --hash=sha256:a13b68cd1cd8cc9de8f244ebae18ccb3e4067ad205220ef324c39181e23bbf66 \ + --hash=sha256:ab44b178f7981fcaea7e0a5df20e773c663d06ffda0198f1a524e91b2fde7e59 \ + --hash=sha256:ade8cb6ff5632a62b7dbd4757d8c5573f7a2e9ae285d6b5b841707d8363205ef \ + --hash=sha256:aed40e099404721d7fcaf5b89bd3b4568a4666358bcac7b6b15c09fb6252ab68 \ + --hash=sha256:b1cbfa133241d0e6bdab48dcdc2604e8ba81512f6bbd68ec3e8e1357dd3c316c \ + --hash=sha256:bb7613e1a427cfcb6ea4544f9ac566b93d5bf67e0d48c787eca673ff9c9dff2b \ + --hash=sha256:bdaba7d87e66f26a2c45d8cbadcbfc4bf7884182317907baf39cfe9775bb4d93 \ + --hash=sha256:c05b450d37ba0c9e21c77fef1f205f56bcee2330bddca68d344baebfc55ae0df \ + --hash=sha256:c1e2b199f446d3e82246b4fd9236d7cb502dc2222b18698ba0d986d2fecc6152 \ + --hash=sha256:c3524798e70655ff19aec58c7d05adb1f074fecff62da857ea9be2b908b6d701 \ + --hash=sha256:cc5223ab19fe25e2f0bf2643204ad7318896fe3729bf12fde41b77bfc4fafff0 \ + --hash=sha256:d2a6394e6af690d462310a86b53c47ad75ac8c21dc79f120714ea449979cb1d3 \ + --hash=sha256:db367d8be9fad6e8ebbac4a7578b7af562e506211036cba2c06c3b998603c3d2 \ + --hash=sha256:e104da1db1c0991b3eaed391ccd650ae8d947eab1480c733e5a3fb28d4313e40 \ + --hash=sha256:e404ea551d35438013c64b4f357b0474c7abf9f781c06d44fcaf7a14c69ff9e2 \ + --hash=sha256:e5562a0f0e90a6223b704163ea28e831bd3a9faa3512a711f031611e6b06c939 \ + --hash=sha256:ea026e70a9a28ebbdddcbcf0f1323128a8db66898a06eaad3a4e62d2f554d096 \ + --hash=sha256:ec7e287d7fbd02cb6e22f9a00dd9c9cd504c40a61f2c61e7e1f9690a82726b4c \ + --hash=sha256:ed9bbc30f5d60a3bdf63ae76beb3f9db280d7f195dfcfa61af792d6ce912d159 \ + --hash=sha256:ee9da221dca6e0429c2704c1b3655fe7b025204a71d4d9b73390c759d776d165 \ + --hash=sha256:f22ef501c3f87ede88f23f9b11e608581c14f04db59b6a801f354397ae13739f \ + --hash=sha256:f2839f9c2c7e2dffc1bc5929a510e14ce0a946be9365fd1219e7ef342dae14f4 \ + --hash=sha256:f556aa591c00f2c45eb1b89f68f52441a016034d18b65da60e2d2875bbbf344a \ + --hash=sha256:f7e1d61da332ec412350463891923f960c3073cf1aae93b538f0bb4c8cd46efb \ + --hash=sha256:f917a04240ef31898182f76a332f508f2cc4b57d2b4d7ad2dbfebbfe167eb505 \ + --hash=sha256:fa476ab5dd49f3bf3a168e05f89358c75a17608dbabb080ef65f96b27c19ab10 \ + --hash=sha256:ff732bd0a0e778f43d5009840f20b935e79087b4dc65bd36f1cd0f9b04b8ff7f + # via openai +jmespath==1.1.0 \ + --hash=sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d \ + --hash=sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64 + # via + # boto3 + # botocore +msgpack==1.1.2 \ + --hash=sha256:04fb995247a6e83830b62f0b07bf36540c213f6eac8e851166d8d86d83cbd014 \ + --hash=sha256:180759d89a057eab503cf62eeec0aa61c4ea1200dee709f3a8e9397dbb3b6931 \ + --hash=sha256:1d1418482b1ee984625d88aa9585db570180c286d942da463533b238b98b812b \ + --hash=sha256:1de460f0403172cff81169a30b9a92b260cb809c4cb7e2fc79ae8d0510c78b6b \ + --hash=sha256:1fdf7d83102bf09e7ce3357de96c59b627395352a4024f6e2458501f158bf999 \ + --hash=sha256:1fff3d825d7859ac888b0fbda39a42d59193543920eda9d9bea44d958a878029 \ + --hash=sha256:283ae72fc89da59aa004ba147e8fc2f766647b1251500182fac0350d8af299c0 \ + --hash=sha256:2929af52106ca73fcb28576218476ffbb531a036c2adbcf54a3664de124303e9 \ + --hash=sha256:2e86a607e558d22985d856948c12a3fa7b42efad264dca8a3ebbcfa2735d786c \ + --hash=sha256:372839311ccf6bdaf39b00b61288e0557916c3729529b301c52c2d88842add42 \ + --hash=sha256:3b60763c1373dd60f398488069bcdc703cd08a711477b5d480eecc9f9626f47e \ + --hash=sha256:42eefe2c3e2af97ed470eec850facbe1b5ad1d6eacdbadc42ec98e7dcf68b4b7 \ + --hash=sha256:446abdd8b94b55c800ac34b102dffd2f6aa0ce643c55dfc017ad89347db3dbdb \ + --hash=sha256:454e29e186285d2ebe65be34629fa0e8605202c60fbc7c4c650ccd41870896ef \ + --hash=sha256:4efd7b5979ccb539c221a4c4e16aac1a533efc97f3b759bb5a5ac9f6d10383bf \ + --hash=sha256:5559d03930d3aa0f3aacb4c42c776af1a2ace2611871c84a75afe436695e6245 \ + --hash=sha256:5928604de9b032bc17f5099496417f113c45bc6bc21b5c6920caf34b3c428794 \ + --hash=sha256:59415c6076b1e30e563eb732e23b994a61c159cec44deaf584e5cc1dd662f2af \ + --hash=sha256:5a46bf7e831d09470ad92dff02b8b1ac92175ca36b087f904a0519857c6be3ff \ + --hash=sha256:602b6740e95ffc55bfb078172d279de3773d7b7db1f703b2f1323566b878b90e \ + --hash=sha256:61c8aa3bd513d87c72ed0b37b53dd5c5a0f58f2ff9f26e1555d3bd7948fb7296 \ + --hash=sha256:6c15b7d74c939ebe620dd8e559384be806204d73b4f9356320632d783d1f7939 \ + --hash=sha256:70a0dff9d1f8da25179ffcf880e10cf1aad55fdb63cd59c9a49a1b82290062aa \ + --hash=sha256:70c5a7a9fea7f036b716191c29047374c10721c389c21e9ffafad04df8c52c90 \ + --hash=sha256:7bc8813f88417599564fafa59fd6f95be417179f76b40325b500b3c98409757c \ + --hash=sha256:80a0ff7d4abf5fecb995fcf235d4064b9a9a8a40a3ab80999e6ac1e30b702717 \ + --hash=sha256:86f8136dfa5c116365a8a651a7d7484b65b13339731dd6faebb9a0242151c406 \ + --hash=sha256:897c478140877e5307760b0ea66e0932738879e7aa68144d9b78ea4c8302a84a \ + --hash=sha256:8e22ab046fa7ede9e36eeb4cfad44d46450f37bb05d5ec482b02868f451c95e2 \ + --hash=sha256:99e2cb7b9031568a2a5c73aa077180f93dd2e95b4f8d3b8e14a73ae94a9e667e \ + --hash=sha256:9ade919fac6a3e7260b7f64cea89df6bec59104987cbea34d34a2fa15d74310b \ + --hash=sha256:a465f0dceb8e13a487e54c07d04ae3ba131c7c5b95e2612596eafde1dccf64a9 \ + --hash=sha256:a668204fa43e6d02f89dbe79a30b0d67238d9ec4c5bd8a940fc3a004a47b721b \ + --hash=sha256:a7787d353595c7c7e145e2331abf8b7ff1e6673a6b974ded96e6d4ec09f00c8c \ + --hash=sha256:bafca952dc13907bdfdedfc6a5f579bf4f292bdd506fadb38389afa3ac5b208e \ + --hash=sha256:be52a8fc79e45b0364210eef5234a7cf8d330836d0a64dfbb878efa903d84620 \ + --hash=sha256:be5980f3ee0e6bd44f3a9e9dea01054f175b50c3e6cdb692bc9424c0bbb8bf69 \ + --hash=sha256:c63eea553c69ab05b6747901b97d620bb2a690633c77f23feb0c6a947a8a7b8f \ + --hash=sha256:d198d275222dc54244bf3327eb8cbe00307d220241d9cec4d306d49a44e85f68 \ + --hash=sha256:d62ce1f483f355f61adb5433ebfd8868c5f078d1a52d042b0a998682b4fa8c27 \ + --hash=sha256:d99ef64f349d5ec3293688e91486c5fdb925ed03807f64d98d205d2713c60b46 \ + --hash=sha256:e23ce8d5f7aa6ea6d2a2b326b4ba46c985dbb204523759984430db7114f8aa00 \ + --hash=sha256:e69b39f8c0aa5ec24b57737ebee40be647035158f14ed4b40e6f150077e21a84 \ + --hash=sha256:f2cb069d8b981abc72b41aea1c580ce92d57c673ec61af4c500153a626cb9e20 \ + --hash=sha256:fac4be746328f90caa3cd4bc67e6fe36ca2bf61d5c6eb6d895b6527e3f05071e \ + --hash=sha256:fffee09044073e69f2bad787071aeec727183e7580443dfeb8556cbf1978d162 + # via xtrace-ai-sdk +multidict==6.7.1 \ + --hash=sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9 \ + --hash=sha256:0458c978acd8e6ea53c81eefaddbbee9c6c5e591f41b3f5e8e194780fe026581 \ + --hash=sha256:0b38ebffd9be37c1170d33bc0f36f4f262e0a09bc1aac1c34c7aa51a7293f0b3 \ + --hash=sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43 \ + --hash=sha256:0e161ddf326db5577c3a4cc2d8648f81456e8a20d40415541587a71620d7a7d1 \ + --hash=sha256:10ae39c9cfe6adedcdb764f5e8411d4a92b055e35573a2eaa88d3323289ef93c \ + --hash=sha256:128441d052254f42989ef98b7b6a6ecb1e6f708aa962c7984235316db59f50fa \ + --hash=sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6 \ + --hash=sha256:14525a5f61d7d0c94b368a42cff4c9a4e7ba2d52e2672a7b23d84dc86fb02b0c \ + --hash=sha256:17307b22c217b4cf05033dabefe68255a534d637c6c9b0cc8382718f87be4262 \ + --hash=sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd \ + --hash=sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d \ + --hash=sha256:1e3a8bb24342a8201d178c3b4984c26ba81a577c80d4d525727427460a50c22d \ + --hash=sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3 \ + --hash=sha256:233b398c29d3f1b9676b4b6f75c518a06fcb2ea0b925119fb2c1bc35c05e1601 \ + --hash=sha256:25167cc263257660290fba06b9318d2026e3c910be240a146e1f66dd114af2b0 \ + --hash=sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292 \ + --hash=sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed \ + --hash=sha256:283ddac99f7ac25a4acadbf004cb5ae34480bbeb063520f70ce397b281859362 \ + --hash=sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511 \ + --hash=sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23 \ + --hash=sha256:2bbd113e0d4af5db41d5ebfe9ccaff89de2120578164f86a5d17d5a576d1e5b2 \ + --hash=sha256:2e1425e2f99ec5bd36c15a01b690a1a2456209c5deed58f95469ffb46039ccbb \ + --hash=sha256:2e2d2ed645ea29f31c4c7ea1552fcfd7cb7ba656e1eafd4134a6620c9f5fdd9e \ + --hash=sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582 \ + --hash=sha256:38fb49540705369bab8484db0689d86c0a33a0a9f2c1b197f506b71b4b6c19b0 \ + --hash=sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e \ + --hash=sha256:3ab8b9d8b75aef9df299595d5388b14530839f6422333357af1339443cff777d \ + --hash=sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65 \ + --hash=sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a \ + --hash=sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd \ + --hash=sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d \ + --hash=sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108 \ + --hash=sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177 \ + --hash=sha256:439cbebd499f92e9aa6793016a8acaa161dfa749ae86d20960189f5398a19144 \ + --hash=sha256:4885cb0e817aef5d00a2e8451d4665c1808378dc27c2705f1bf4ef8505c0d2e5 \ + --hash=sha256:497394b3239fc6f0e13a78a3e1b61296e72bf1c5f94b4c4eb80b265c37a131cd \ + --hash=sha256:497bde6223c212ba11d462853cfa4f0ae6ef97465033e7dc9940cdb3ab5b48e5 \ + --hash=sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060 \ + --hash=sha256:538cec1e18c067d0e6103aa9a74f9e832904c957adc260e61cd9d8cf0c3b3d37 \ + --hash=sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56 \ + --hash=sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df \ + --hash=sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963 \ + --hash=sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118 \ + --hash=sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84 \ + --hash=sha256:5e01429a929600e7dab7b166062d9bb54a5eed752384c7384c968c2afab8f50f \ + --hash=sha256:5fa6a95dfee63893d80a34758cd0e0c118a30b8dcb46372bf75106c591b77889 \ + --hash=sha256:619e5a1ac57986dbfec9f0b301d865dddf763696435e2962f6d9cf2fdff2bb71 \ + --hash=sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7 \ + --hash=sha256:6b10359683bd8806a200fd2909e7c8ca3a7b24ec1d8132e483d58e791d881048 \ + --hash=sha256:6b83cabdc375ffaaa15edd97eb7c0c672ad788e2687004990074d7d6c9b140c8 \ + --hash=sha256:6d3bc717b6fe763b8be3f2bee2701d3c8eb1b2a8ae9f60910f1b2860c82b6c49 \ + --hash=sha256:7a7e590ff876a3eaf1c02a4dfe0724b6e69a9e9de6d8f556816f29c496046e59 \ + --hash=sha256:7eee46ccb30ff48a1e35bb818cc90846c6be2b68240e42a78599166722cea709 \ + --hash=sha256:7ff981b266af91d7b4b3793ca3382e53229088d193a85dfad6f5f4c27fc73e5d \ + --hash=sha256:841189848ba629c3552035a6a7f5bf3b02eb304e9fea7492ca220a8eda6b0e5c \ + --hash=sha256:844c5bca0b5444adb44a623fb0a1310c2f4cd41f402126bb269cd44c9b3f3e1e \ + --hash=sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2 \ + --hash=sha256:8be1802715a8e892c784c0197c2ace276ea52702a0ede98b6310c8f255a5afb3 \ + --hash=sha256:8f333ec9c5eb1b7105e3b84b53141e66ca05a19a605368c55450b6ba208cb9ee \ + --hash=sha256:9004d8386d133b7e6135679424c91b0b854d2d164af6ea3f289f8f2761064609 \ + --hash=sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c \ + --hash=sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445 \ + --hash=sha256:93b1818e4a6e0930454f0f2af7dfce69307ca03cdcfb3739bf4d91241967b6c1 \ + --hash=sha256:95922cee9a778659e91db6497596435777bd25ed116701a4c034f8e46544955a \ + --hash=sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5 \ + --hash=sha256:97231140a50f5d447d3164f994b86a0bed7cd016e2682f8650d6a9158e14fd31 \ + --hash=sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33 \ + --hash=sha256:98655c737850c064a65e006a3df7c997cd3b220be4ec8fe26215760b9697d4d7 \ + --hash=sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca \ + --hash=sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733 \ + --hash=sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429 \ + --hash=sha256:9f9af11306994335398293f9958071019e3ab95e9a707dc1383a35613f6abcb9 \ + --hash=sha256:a0543217a6a017692aa6ae5cc39adb75e587af0f3a82288b1492eb73dd6cc2a4 \ + --hash=sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6 \ + --hash=sha256:a407f13c188f804c759fc6a9f88286a565c242a76b27626594c133b82883b5c2 \ + --hash=sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172 \ + --hash=sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52 \ + --hash=sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7 \ + --hash=sha256:b26684587228afed0d50cf804cc71062cc9c1cdf55051c4c6345d372947b268c \ + --hash=sha256:b4938326284c4f1224178a560987b6cf8b4d38458b113d9b8c1db1a836e640a2 \ + --hash=sha256:b8c990b037d2fff2f4e33d3f21b9b531c5745b33a49a7d6dbe7a177266af44f6 \ + --hash=sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf \ + --hash=sha256:bdbf9f3b332abd0cdb306e7c2113818ab1e922dc84b8f8fd06ec89ed2a19ab8b \ + --hash=sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961 \ + --hash=sha256:c0abd12629b0af3cf590982c0b413b1e7395cd4ec026f30986818ab95bfaa94a \ + --hash=sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3 \ + --hash=sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b \ + --hash=sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1 \ + --hash=sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c \ + --hash=sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53 \ + --hash=sha256:ce1bbd7d780bb5a0da032e095c951f7014d6b0a205f8318308140f1a6aba159e \ + --hash=sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8 \ + --hash=sha256:d62b7f64ffde3b99d06b707a280db04fb3855b55f5a06df387236051d0668f4a \ + --hash=sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a \ + --hash=sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32 \ + --hash=sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3 \ + --hash=sha256:e628ef0e6859ffd8273c69412a2465c4be4a9517d07261b33334b5ec6f3c7489 \ + --hash=sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23 \ + --hash=sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34 \ + --hash=sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75 \ + --hash=sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8 \ + --hash=sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d \ + --hash=sha256:f2a0a924d4c2e9afcd7ec64f9de35fcd96915149b2216e1cb2c10a56df483855 \ + --hash=sha256:f33dc2a3abe9249ea5d8360f969ec7f4142e7ac45ee7014d8f8d5acddf178b7b \ + --hash=sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4 \ + --hash=sha256:f99fe611c312b3c1c0ace793f92464d8cd263cc3b26b5721950d977b006b6c4d \ + --hash=sha256:fa263a02f4f2dd2d11a7b1bb4362aa7cb1049f84a9235d31adf63f30143469a0 \ + --hash=sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba \ + --hash=sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19 + # via + # aiohttp + # yarl +numpy==2.4.3 \ + --hash=sha256:0200b25c687033316fb39f0ff4e3e690e8957a2c3c8d22499891ec58c37a3eb5 \ + --hash=sha256:0448e7f9caefb34b4b7dd2b77f21e8906e5d6f0365ad525f9f4f530b13df2afc \ + --hash=sha256:0a195f4216be9305a73c0e91c9b026a35f2161237cf1c6de9b681637772ea657 \ + --hash=sha256:0a60e17a14d640f49146cb38e3f105f571318db7826d9b6fef7e4dce758faecd \ + --hash=sha256:120df8c0a81ebbf5b9020c91439fccd85f5e018a927a39f624845be194a2be02 \ + --hash=sha256:148d59127ac95979d6f07e4d460f934ebdd6eed641db9c0db6c73026f2b2101a \ + --hash=sha256:1ec84fd7c8e652b0f4aaaf2e6e9cc8eaa9b1b80a537e06b2e3a2fb176eedcb26 \ + --hash=sha256:22654fe6be0e5206f553a9250762c653d3698e46686eee53b399ab90da59bd92 \ + --hash=sha256:22c31dc07025123aedf7f2db9e91783df13f1776dc52c6b22c620870dc0fab22 \ + --hash=sha256:23b46bb6d8ecb68b58c09944483c135ae5f0e9b8d8858ece5e4ead783771d2a9 \ + --hash=sha256:2629289168f4897a3c4e23dc98d6f1731f0fc0fe52fb9db19f974041e4cc12b9 \ + --hash=sha256:26952e18d82a1dbbc2f008d402021baa8d6fc8e84347a2072a25e08b46d698b9 \ + --hash=sha256:29363fbfa6f8ee855d7569c96ce524845e3d726d6c19b29eceec7dd555dab152 \ + --hash=sha256:297837823f5bc572c5f9379b0c9f3a3365f08492cbdc33bcc3af174372ebb168 \ + --hash=sha256:2abad5c7fef172b3377502bde47892439bae394a71bc329f31df0fd829b41a9e \ + --hash=sha256:2b3f8d2c4589b1a2028d2a770b0fc4d1f332fb5e01521f4de3199a896d158ddd \ + --hash=sha256:2ddb7919366ee468342b91dea2352824c25b55814a987847b6c52003a7c97f15 \ + --hash=sha256:2e03c05abaee1f672e9d67bc858f300b5ccba1c21397211e8d77d98350972093 \ + --hash=sha256:32e3bef222ad6b052280311d1d60db8e259e4947052c3ae7dd6817451fc8a4c5 \ + --hash=sha256:33b3bf58ee84b172c067f56aeadc7ee9ab6de69c5e800ab5b10295d54c581adb \ + --hash=sha256:45f003dbdffb997a03da2d1d0cb41fbd24a87507fb41605c0420a3db5bd4667b \ + --hash=sha256:483a201202b73495f00dbc83796c6ae63137a9bdade074f7648b3e32613412dd \ + --hash=sha256:48da3a4ee1336454b07497ff7ec83903efa5505792c4e6d9bf83d99dc07a1e18 \ + --hash=sha256:4b42639cdde6d24e732ff823a3fa5b701d8acad89c4142bc1d0bd6dc85200ba5 \ + --hash=sha256:4bd4741a6a676770e0e97fe9ab2e51de01183df3dcbcec591d26d331a40de950 \ + --hash=sha256:4d382735cecd7bcf090172489a525cd7d4087bc331f7df9f60ddc9a296cf208e \ + --hash=sha256:52077feedeff7c76ed7c9f1a0428558e50825347b7545bbb8523da2cd55c547a \ + --hash=sha256:54f29b877279d51e210e0c80709ee14ccbbad647810e8f3d375561c45ef613dd \ + --hash=sha256:5884ce5c7acfae1e4e1b6fde43797d10aa506074d25b531b4f54bde33c0c31d4 \ + --hash=sha256:5e10da9e93247e554bb1d22f8edc51847ddd7dde52d85ce31024c1b4312bfba0 \ + --hash=sha256:61b0cbabbb6126c8df63b9a3a0c4b1f44ebca5e12ff6997b80fcf267fb3150ef \ + --hash=sha256:65f3c2455188f09678355f5cae1f959a06b778bc66d535da07bf2ef20cd319d5 \ + --hash=sha256:679f2a834bae9020f81534671c56fd0cc76dd7e5182f57131478e23d0dc59e24 \ + --hash=sha256:6bd06731541f89cdc01b261ba2c9e037f1543df7472517836b78dfb15bd6e476 \ + --hash=sha256:715de7f82e192e8cae5a507a347d97ad17598f8e026152ca97233e3666daaa71 \ + --hash=sha256:737f630a337364665aba3b5a77e56a68cc42d350edd010c345d65a3efa3addcc \ + --hash=sha256:7395e69ff32526710748f92cd8c9849b361830968ea3e24a676f272653e8983e \ + --hash=sha256:76dbb9d4e43c16cf9aa711fcd8de1e2eeb27539dcefb60a1d5e9f12fae1d1ed8 \ + --hash=sha256:76f0f283506c28b12bba319c0fab98217e9f9b54e6160e9c79e9f7348ba32e9c \ + --hash=sha256:77e76d932c49a75617c6d13464e41203cd410956614d0a0e999b25e9e8d27eec \ + --hash=sha256:7aa4e54f6469300ebca1d9eb80acd5253cdfa36f2c03d79a35883687da430875 \ + --hash=sha256:7d1ce23cce91fcea443320a9d0ece9b9305d4368875bab09538f7a5b4131938a \ + --hash=sha256:7e58765ad74dcebd3ef0208a5078fba32dc8ec3578fe84a604432950cd043d79 \ + --hash=sha256:7f3408ff897f8ab07a07fbe2823d7aee6ff644c097cc1f90382511fe982f647f \ + --hash=sha256:8ba7b51e71c05aa1f9bc3641463cd82308eab40ce0d5c7e1fd4038cbf9938147 \ + --hash=sha256:8e236dbda4e1d319d681afcbb136c0c4a8e0f1a5c58ceec2adebb547357fe857 \ + --hash=sha256:94f3c4a151a2e529adf49c1d54f0f57ff8f9b233ee4d44af623a81553ab86368 \ + --hash=sha256:9684823a78a6cd6ad7511fc5e25b07947d1d5b5e2812c93fe99d7d4195130720 \ + --hash=sha256:a016db5c5dba78fa8fe9f5d80d6708f9c42ab087a739803c0ac83a43d686a470 \ + --hash=sha256:a111698b4a3f8dcbe54c64a7708f049355abd603e619013c346553c1fd4ca90b \ + --hash=sha256:a1988292870c7cb9d0ebb4cc96b4d447513a9644801de54606dc7aabf2b7d920 \ + --hash=sha256:a315e5234d88067f2d97e1f2ef670a7569df445d55400f1e33d117418d008d52 \ + --hash=sha256:a749547700de0a20a6718293396ec237bb38218049cfce788e08fcb716e8cf73 \ + --hash=sha256:a97cbf7e905c435865c2d939af3d93f99d18eaaa3cabe4256f4304fb51604349 \ + --hash=sha256:abdce0f71dcb4a00e4e77f3faf05e4616ceccfe72ccaa07f47ee79cda3b7b0f4 \ + --hash=sha256:b346845443716c8e542d54112966383b448f4a3ba5c66409771b8c0889485dd3 \ + --hash=sha256:b44fd60341c4d9783039598efadd03617fa28d041fc37d22b62d08f2027fa0e7 \ + --hash=sha256:bb2e3cf95854233799013779216c57e153c1ee67a0bf92138acca0e429aefaee \ + --hash=sha256:bc71942c789ef415a37f0d4eab90341425a00d538cd0642445d30b41023d3395 \ + --hash=sha256:be3b8487d725a77acccc9924f65fd8bce9af7fac8c9820df1049424a2115af6c \ + --hash=sha256:c59020932feb24ed49ffd03704fbab89f22aa9c0d4b180ff45542fe8918f5611 \ + --hash=sha256:c6b124bfcafb9e8d3ed09130dbee44848c20b3e758b6bbf006e641778927c028 \ + --hash=sha256:c9619741e9da2059cd9c3f206110b97583c7152c1dc9f8aafd4beb450ac1c89d \ + --hash=sha256:cd32fbacb9fd1bf041bf8e89e4576b6f00b895f06d00914820ae06a616bdfef7 \ + --hash=sha256:d1b90d840b25874cf5cd20c219af10bac3667db3876d9a495609273ebe679070 \ + --hash=sha256:d213c7e6e8d211888cc359bab7199670a00f5b82c0978b9d1c75baf1eddbeac0 \ + --hash=sha256:d5f51900414fc9204a0e0da158ba2ac52b75656e7dce7e77fb9f84bfa343b4cc \ + --hash=sha256:d71e379452a2f670ccb689ec801b1218cd3983e253105d6e83780967e899d687 \ + --hash=sha256:d84f0f881cb2225c2dfd7f78a10a5645d487a496c6668d6cc39f0f114164f3d0 \ + --hash=sha256:decb0eb8a53c3b009b0962378065589685d66b23467ef5dac16cbe818afde27f \ + --hash=sha256:e7dd01a46700b1967487141a66ac1a3cf0dd8ebf1f08db37d46389401512ca97 \ + --hash=sha256:eb610595dd91560905c132c709412b512135a60f1851ccbd2c959e136431ff67 + # via xtrace-ai-sdk +openai==2.28.0 \ + --hash=sha256:79aa5c45dba7fef84085701c235cf13ba88485e1ef4f8dfcedc44fc2a698fc1d \ + --hash=sha256:bb7fdff384d2a787fa82e8822d1dd3c02e8cf901d60f1df523b7da03cbb6d48d + # via xtrace-ai-sdk +propcache==0.4.1 \ + --hash=sha256:0013cb6f8dde4b2a2f66903b8ba740bdfe378c943c4377a200551ceb27f379e4 \ + --hash=sha256:005f08e6a0529984491e37d8dbc3dd86f84bd78a8ceb5fa9a021f4c48d4984be \ + --hash=sha256:031dce78b9dc099f4c29785d9cf5577a3faf9ebf74ecbd3c856a7b92768c3df3 \ + --hash=sha256:05674a162469f31358c30bcaa8883cb7829fa3110bf9c0991fe27d7896c42d85 \ + --hash=sha256:060b16ae65bc098da7f6d25bf359f1f31f688384858204fe5d652979e0015e5b \ + --hash=sha256:120c964da3fdc75e3731aa392527136d4ad35868cc556fd09bb6d09172d9a367 \ + --hash=sha256:15932ab57837c3368b024473a525e25d316d8353016e7cc0e5ba9eb343fbb1cf \ + --hash=sha256:17612831fda0138059cc5546f4d12a2aacfb9e47068c06af35c400ba58ba7393 \ + --hash=sha256:204483131fb222bdaaeeea9f9e6c6ed0cac32731f75dfc1d4a567fc1926477c1 \ + --hash=sha256:2ad890caa1d928c7c2965b48f3a3815c853180831d0e5503d35cf00c472f4717 \ + --hash=sha256:2bb07ffd7eaad486576430c89f9b215f9e4be68c4866a96e97db9e97fead85dc \ + --hash=sha256:333ddb9031d2704a301ee3e506dc46b1fe5f294ec198ed6435ad5b6a085facfe \ + --hash=sha256:35c3277624a080cc6ec6f847cbbbb5b49affa3598c4535a0a4682a697aaa5c75 \ + --hash=sha256:364426a62660f3f699949ac8c621aad6977be7126c5807ce48c0aeb8e7333ea6 \ + --hash=sha256:381914df18634f5494334d201e98245c0596067504b9372d8cf93f4bb23e025e \ + --hash=sha256:3d902a36df4e5989763425a8ab9e98cd8ad5c52c823b34ee7ef307fd50582566 \ + --hash=sha256:3f7124c9d820ba5548d431afb4632301acf965db49e666aa21c305cbe8c6de12 \ + --hash=sha256:405aac25c6394ef275dee4c709be43745d36674b223ba4eb7144bf4d691b7367 \ + --hash=sha256:41a89040cb10bd345b3c1a873b2bf36413d48da1def52f268a055f7398514874 \ + --hash=sha256:43eedf29202c08550aac1d14e0ee619b0430aaef78f85864c1a892294fbc28cf \ + --hash=sha256:473c61b39e1460d386479b9b2f337da492042447c9b685f28be4f74d3529e566 \ + --hash=sha256:49a2dc67c154db2c1463013594c458881a069fcf98940e61a0569016a583020a \ + --hash=sha256:4c3c70630930447f9ef1caac7728c8ad1c56bc5015338b20fed0d08ea2480b3a \ + --hash=sha256:4d3df5fa7e36b3225954fba85589da77a0fe6a53e3976de39caf04a0db4c36f1 \ + --hash=sha256:4d7af63f9f93fe593afbf104c21b3b15868efb2c21d07d8732c0c4287e66b6a6 \ + --hash=sha256:501d20b891688eb8e7aa903021f0b72d5a55db40ffaab27edefd1027caaafa61 \ + --hash=sha256:521a463429ef54143092c11a77e04056dd00636f72e8c45b70aaa3140d639726 \ + --hash=sha256:5558992a00dfd54ccbc64a32726a3357ec93825a418a401f5cc67df0ac5d9e49 \ + --hash=sha256:55c72fd6ea2da4c318e74ffdf93c4fe4e926051133657459131a95c846d16d44 \ + --hash=sha256:564d9f0d4d9509e1a870c920a89b2fec951b44bf5ba7d537a9e7c1ccec2c18af \ + --hash=sha256:580e97762b950f993ae618e167e7be9256b8353c2dcd8b99ec100eb50f5286aa \ + --hash=sha256:5a103c3eb905fcea0ab98be99c3a9a5ab2de60228aa5aceedc614c0281cf6153 \ + --hash=sha256:5c3310452e0d31390da9035c348633b43d7e7feb2e37be252be6da45abd1abcc \ + --hash=sha256:60a8fda9644b7dfd5dece8c61d8a85e271cb958075bfc4e01083c148b61a7caf \ + --hash=sha256:671538c2262dadb5ba6395e26c1731e1d52534bfe9ae56d0b5573ce539266aa8 \ + --hash=sha256:678ae89ebc632c5c204c794f8dab2837c5f159aeb59e6ed0539500400577298c \ + --hash=sha256:67fad6162281e80e882fb3ec355398cf72864a54069d060321f6cd0ade95fe85 \ + --hash=sha256:6918ecbd897443087a3b7cd978d56546a812517dcaaca51b49526720571fa93e \ + --hash=sha256:6f6ff873ed40292cd4969ef5310179afd5db59fdf055897e282485043fc80ad0 \ + --hash=sha256:6f8b465489f927b0df505cbe26ffbeed4d6d8a2bbc61ce90eb074ff129ef0ab1 \ + --hash=sha256:74c1fb26515153e482e00177a1ad654721bf9207da8a494a0c05e797ad27b992 \ + --hash=sha256:824e908bce90fb2743bd6b59db36eb4f45cd350a39637c9f73b1c1ea66f5b75f \ + --hash=sha256:8326e144341460402713f91df60ade3c999d601e7eb5ff8f6f7862d54de0610d \ + --hash=sha256:8873eb4460fd55333ea49b7d189749ecf6e55bf85080f11b1c4530ed3034cba1 \ + --hash=sha256:89eb3fa9524f7bec9de6e83cf3faed9d79bffa560672c118a96a171a6f55831e \ + --hash=sha256:8e57061305815dfc910a3634dcf584f08168a8836e6999983569f51a8544cd89 \ + --hash=sha256:929d7cbe1f01bb7baffb33dc14eb5691c95831450a26354cd210a8155170c93a \ + --hash=sha256:92d1935ee1f8d7442da9c0c4fa7ac20d07e94064184811b685f5c4fada64553b \ + --hash=sha256:981333cb2f4c1896a12f4ab92a9cc8f09ea664e9b7dbdc4eff74627af3a11c0f \ + --hash=sha256:990f6b3e2a27d683cb7602ed6c86f15ee6b43b1194736f9baaeb93d0016633b1 \ + --hash=sha256:9a0bd56e5b100aef69bd8562b74b46254e7c8812918d3baa700c8a8009b0af66 \ + --hash=sha256:9f302f4783709a78240ebc311b793f123328716a60911d667e0c036bc5dcbded \ + --hash=sha256:a78372c932c90ee474559c5ddfffd718238e8673c340dc21fe45c5b8b54559a0 \ + --hash=sha256:a9695397f85973bb40427dedddf70d8dc4a44b22f1650dd4af9eedf443d45165 \ + --hash=sha256:ab08df6c9a035bee56e31af99be621526bd237bea9f32def431c656b29e41778 \ + --hash=sha256:ab2943be7c652f09638800905ee1bab2c544e537edb57d527997a24c13dc1455 \ + --hash=sha256:ab4c29b49d560fe48b696cdcb127dd36e0bc2472548f3bf56cc5cb3da2b2984f \ + --hash=sha256:af223b406d6d000830c6f65f1e6431783fc3f713ba3e6cc8c024d5ee96170a4b \ + --hash=sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237 \ + --hash=sha256:bcc9aaa5d80322bc2fb24bb7accb4a30f81e90ab8d6ba187aec0744bc302ad81 \ + --hash=sha256:c07fda85708bc48578467e85099645167a955ba093be0a2dcba962195676e859 \ + --hash=sha256:c0d4b719b7da33599dfe3b22d3db1ef789210a0597bc650b7cee9c77c2be8c5c \ + --hash=sha256:c0ef0aaafc66fbd87842a3fe3902fd889825646bc21149eafe47be6072725835 \ + --hash=sha256:c2b5e7db5328427c57c8e8831abda175421b709672f6cfc3d630c3b7e2146393 \ + --hash=sha256:c30b53e7e6bda1d547cabb47c825f3843a0a1a42b0496087bb58d8fedf9f41b5 \ + --hash=sha256:c80ee5802e3fb9ea37938e7eecc307fb984837091d5fd262bb37238b1ae97641 \ + --hash=sha256:c9b822a577f560fbd9554812526831712c1436d2c046cedee4c3796d3543b144 \ + --hash=sha256:cae65ad55793da34db5f54e4029b89d3b9b9490d8abe1b4c7ab5d4b8ec7ebf74 \ + --hash=sha256:cb2d222e72399fcf5890d1d5cc1060857b9b236adff2792ff48ca2dfd46c81db \ + --hash=sha256:cd547953428f7abb73c5ad82cbb32109566204260d98e41e5dfdc682eb7f8403 \ + --hash=sha256:cfc27c945f422e8b5071b6e93169679e4eb5bf73bbcbf1ba3ae3a83d2f78ebd9 \ + --hash=sha256:d472aeb4fbf9865e0c6d622d7f4d54a4e101a89715d8904282bb5f9a2f476c3f \ + --hash=sha256:d62cdfcfd89ccb8de04e0eda998535c406bf5e060ffd56be6c586cbcc05b3311 \ + --hash=sha256:d8f353eb14ee3441ee844ade4277d560cdd68288838673273b978e3d6d2c8f36 \ + --hash=sha256:dee69d7015dc235f526fe80a9c90d65eb0039103fe565776250881731f06349f \ + --hash=sha256:e153e9cd40cc8945138822807139367f256f89c6810c2634a4f6902b52d3b4e2 \ + --hash=sha256:e35b88984e7fa64aacecea39236cee32dd9bd8c55f57ba8a75cf2399553f9bd7 \ + --hash=sha256:e53f3a38d3510c11953f3e6a33f205c6d1b001129f972805ca9b42fc308bc239 \ + --hash=sha256:e9b0d8d0845bbc4cfcdcbcdbf5086886bc8157aa963c31c777ceff7846c77757 \ + --hash=sha256:ec17c65562a827bba85e3872ead335f95405ea1674860d96483a02f5c698fa72 \ + --hash=sha256:ecef2343af4cc68e05131e45024ba34f6095821988a9d0a02aa7c73fcc448aa9 \ + --hash=sha256:ed5a841e8bb29a55fb8159ed526b26adc5bdd7e8bd7bf793ce647cb08656cdf4 \ + --hash=sha256:ee17f18d2498f2673e432faaa71698032b0127ebf23ae5974eeaf806c279df24 \ + --hash=sha256:f048da1b4f243fc44f205dfd320933a951b8d89e0afd4c7cacc762a8b9165207 \ + --hash=sha256:f10207adf04d08bec185bae14d9606a1444715bc99180f9331c9c02093e1959e \ + --hash=sha256:f1d2f90aeec838a52f1c1a32fe9a619fefd5e411721a9117fbf82aea638fe8a1 \ + --hash=sha256:f48107a8c637e80362555f37ecf49abe20370e557cc4ab374f04ec4423c97c3d \ + --hash=sha256:f7ee0e597f495cf415bcbd3da3caa3bd7e816b74d0d52b8145954c5e6fd3ff37 \ + --hash=sha256:f95393b4d66bfae908c3ca8d169d5f79cd65636ae15b5e7a4f6e67af675adb0e \ + --hash=sha256:fc38cba02d1acba4e2869eef1a57a43dfbd3d49a59bf90dda7444ec2be6a5570 \ + --hash=sha256:fd0858c20f078a32cf55f7e81473d96dcf3b93fd2ccdb3d40fdf54b8573df3af \ + --hash=sha256:fd6f30fdcf9ae2a70abd34da54f18da086160e4d7d9251f81f3da0ff84fc5a48 + # via + # aiohttp + # yarl +pycparser==3.0 ; implementation_name != 'PyPy' and platform_python_implementation != 'PyPy' \ + --hash=sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29 \ + --hash=sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992 + # via cffi +pycryptodome==3.23.0 \ + --hash=sha256:0011f7f00cdb74879142011f95133274741778abba114ceca229adbf8e62c3e4 \ + --hash=sha256:11eeeb6917903876f134b56ba11abe95c0b0fd5e3330def218083c7d98bbcb3c \ + --hash=sha256:156df9667ad9f2ad26255926524e1c136d6664b741547deb0a86a9acf5ea631f \ + --hash=sha256:187058ab80b3281b1de11c2e6842a357a1f71b42cb1e15bce373f3d238135c27 \ + --hash=sha256:257bb3572c63ad8ba40b89f6fc9d63a2a628e9f9708d31ee26560925ebe0210a \ + --hash=sha256:447700a657182d60338bab09fdb27518f8856aecd80ae4c6bdddb67ff5da44ef \ + --hash=sha256:4764e64b269fc83b00f682c47443c2e6e85b18273712b98aa43bcb77f8570477 \ + --hash=sha256:507dbead45474b62b2bbe318eb1c4c8ee641077532067fec9c1aa82c31f84886 \ + --hash=sha256:53ecbafc2b55353edcebd64bf5da94a2a2cdf5090a6915bcca6eca6cc452585a \ + --hash=sha256:6501790c5b62a29fcb227bd6b62012181d886a767ce9ed03b303d1f22eb5c625 \ + --hash=sha256:67bd81fcbe34f43ad9422ee8fd4843c8e7198dd88dd3d40e6de42ee65fbe1490 \ + --hash=sha256:763d1d74f56f031788e5d307029caef067febf890cd1f8bf61183ae142f1a77b \ + --hash=sha256:90460fc9e088ce095f9ee8356722d4f10f86e5be06e2354230a9880b9c549aae \ + --hash=sha256:954af0e2bd7cea83ce72243b14e4fb518b18f0c1649b576d114973e2073b273d \ + --hash=sha256:9a53a4fe5cb075075d515797d6ce2f56772ea7e6a1e5e4b96cf78a14bac3d265 \ + --hash=sha256:9a77627a330ab23ca43b48b130e202582e91cc69619947840ea4d2d1be21eb39 \ + --hash=sha256:aa0698f65e5b570426fc31b8162ed4603b0c2841cbb9088e2b01641e3065915b \ + --hash=sha256:c75b52aacc6c0c260f204cbdd834f76edc9fb0d8e0da9fbf8352ef58202564e2 \ + --hash=sha256:c8987bd3307a39bc03df5c8e0e3d8be0c4c3518b7f044b0f4c15d1aa78f52575 \ + --hash=sha256:cfb5cd445280c5b0a4e6187a7ce8de5a07b5f3f897f235caa11f1f435f182843 \ + --hash=sha256:d97618c9c6684a97ef7637ba43bdf6663a2e2e77efe0f863cce97a76af396446 \ + --hash=sha256:dea827b4d55ee390dc89b2afe5927d4308a8b538ae91d9c6f7a5090f397af1aa \ + --hash=sha256:eb8f24adb74984aa0e5d07a2368ad95276cf38051fe2dc6605cbcf482e04f2a7 + # via xtrace-ai-sdk +pydantic==2.12.5 \ + --hash=sha256:4d351024c75c0f085a9febbb665ce8c0c6ec5d30e903bdb6394b7ede26aebb49 \ + --hash=sha256:e561593fccf61e8a20fc46dfc2dfe075b8be7d0188df33f221ad1f0139180f9d + # via openai +pydantic-core==2.41.5 \ + --hash=sha256:0177272f88ab8312479336e1d777f6b124537d47f2123f89cb37e0accea97f90 \ + --hash=sha256:01a3d0ab748ee531f4ea6c3e48ad9dac84ddba4b0d82291f87248f2f9de8d740 \ + --hash=sha256:03b77d184b9eb40240ae9fd676ca364ce1085f203e1b1256f8ab9984dca80a84 \ + --hash=sha256:03ca43e12fab6023fc79d28ca6b39b05f794ad08ec2feccc59a339b02f2b3d33 \ + --hash=sha256:070259a8818988b9a84a449a2a7337c7f430a22acc0859c6b110aa7212a6d9c0 \ + --hash=sha256:08daa51ea16ad373ffd5e7606252cc32f07bc72b28284b6bc9c6df804816476e \ + --hash=sha256:0cbaad15cb0c90aa221d43c00e77bb33c93e8d36e0bf74760cd00e732d10a6a0 \ + --hash=sha256:112e305c3314f40c93998e567879e887a3160bb8689ef3d2c04b6cc62c33ac34 \ + --hash=sha256:1746d4a3d9a794cacae06a5eaaccb4b8643a131d45fbc9af23e353dc0a5ba5c3 \ + --hash=sha256:1962293292865bca8e54702b08a4f26da73adc83dd1fcf26fbc875b35d81c815 \ + --hash=sha256:1d1d9764366c73f996edd17abb6d9d7649a7eb690006ab6adbda117717099b14 \ + --hash=sha256:22f0fb8c1c583a3b6f24df2470833b40207e907b90c928cc8d3594b76f874375 \ + --hash=sha256:239edca560d05757817c13dc17c50766136d21f7cd0fac50295499ae24f90fdf \ + --hash=sha256:25e1c2af0fce638d5f1988b686f3b3ea8cd7de5f244ca147c777769e798a9cd1 \ + --hash=sha256:266fb4cbf5e3cbd0b53669a6d1b039c45e3ce651fd5442eff4d07c2cc8d66808 \ + --hash=sha256:2782c870e99878c634505236d81e5443092fba820f0373997ff75f90f68cd553 \ + --hash=sha256:287dad91cfb551c363dc62899a80e9e14da1f0e2b6ebde82c806612ca2a13ef1 \ + --hash=sha256:2a5e06546e19f24c6a96a129142a75cee553cc018ffee48a460059b1185f4470 \ + --hash=sha256:2b761d210c9ea91feda40d25b4efe82a1707da2ef62901466a42492c028553a2 \ + --hash=sha256:2c010c6ded393148374c0f6f0bf89d206bf3217f201faa0635dcd56bd1520f6b \ + --hash=sha256:2ff4321e56e879ee8d2a879501c8e469414d948f4aba74a2d4593184eb326660 \ + --hash=sha256:3006c3dd9ba34b0c094c544c6006cc79e87d8612999f1a5d43b769b89181f23c \ + --hash=sha256:34a64bc3441dc1213096a20fe27e8e128bd3ff89921706e83c0b1ac971276594 \ + --hash=sha256:35b44f37a3199f771c3eaa53051bc8a70cd7b54f333531c59e29fd4db5d15008 \ + --hash=sha256:378bec5c66998815d224c9ca994f1e14c0c21cb95d2f52b6021cc0b2a58f2a5a \ + --hash=sha256:3f37a19d7ebcdd20b96485056ba9e8b304e27d9904d233d7b1015db320e51f0a \ + --hash=sha256:3f84d5c1b4ab906093bdc1ff10484838aca54ef08de4afa9de0f5f14d69639cd \ + --hash=sha256:4009935984bd36bd2c774e13f9a09563ce8de4abaa7226f5108262fa3e637284 \ + --hash=sha256:406bf18d345822d6c21366031003612b9c77b3e29ffdb0f612367352aab7d586 \ + --hash=sha256:4819fa52133c9aa3c387b3328f25c1facc356491e6135b459f1de698ff64d869 \ + --hash=sha256:482c982f814460eabe1d3bb0adfdc583387bd4691ef00b90575ca0d2b6fe2294 \ + --hash=sha256:4bc36bbc0b7584de96561184ad7f012478987882ebf9f9c389b23f432ea3d90f \ + --hash=sha256:506d766a8727beef16b7adaeb8ee6217c64fc813646b424d0804d67c16eddb66 \ + --hash=sha256:56121965f7a4dc965bff783d70b907ddf3d57f6eba29b6d2e5dabfaf07799c51 \ + --hash=sha256:58133647260ea01e4d0500089a8c4f07bd7aa6ce109682b1426394988d8aaacc \ + --hash=sha256:5cb1b2f9742240e4bb26b652a5aeb840aa4b417c7748b6f8387927bc6e45e40d \ + --hash=sha256:634e8609e89ceecea15e2d61bc9ac3718caaaa71963717bf3c8f38bfde64242c \ + --hash=sha256:63510af5e38f8955b8ee5687740d6ebf7c2a0886d15a6d65c32814613681bc07 \ + --hash=sha256:650ae77860b45cfa6e2cdafc42618ceafab3a2d9a3811fcfbd3bbf8ac3c40d36 \ + --hash=sha256:6561e94ba9dacc9c61bce40e2d6bdc3bfaa0259d3ff36ace3b1e6901936d2e3e \ + --hash=sha256:65840751b72fbfd82c3c640cff9284545342a4f1eb1586ad0636955b261b0b05 \ + --hash=sha256:6cb58b9c66f7e4179a2d5e0f849c48eff5c1fca560994d6eb6543abf955a149e \ + --hash=sha256:72f6c8b11857a856bcfa48c86f5368439f74453563f951e473514579d44aa612 \ + --hash=sha256:76d0819de158cd855d1cbb8fcafdf6f5cf1eb8e470abe056d5d161106e38062b \ + --hash=sha256:76ee27c6e9c7f16f47db7a94157112a2f3a00e958bc626e2f4ee8bec5c328fbe \ + --hash=sha256:79ec52ec461e99e13791ec6508c722742ad745571f234ea6255bed38c6480f11 \ + --hash=sha256:7da7087d756b19037bc2c06edc6c170eeef3c3bafcb8f532ff17d64dc427adfd \ + --hash=sha256:7f3bf998340c6d4b0c9a2f02d6a400e51f123b59565d74dc60d252ce888c260b \ + --hash=sha256:80aa89cad80b32a912a65332f64a4450ed00966111b6615ca6816153d3585a8c \ + --hash=sha256:8566def80554c3faa0e65ac30ab0932b9e3a5cd7f8323764303d468e5c37595a \ + --hash=sha256:88942d3a3dff3afc8288c21e565e476fc278902ae4d6d134f1eeda118cc830b1 \ + --hash=sha256:8e7c86f27c585ef37c35e56a96363ab8de4e549a95512445b85c96d3e2f7c1bf \ + --hash=sha256:915c3d10f81bec3a74fbd4faebe8391013ba61e5a1a8d48c4455b923bdda7858 \ + --hash=sha256:93e8740d7503eb008aa2df04d3b9735f845d43ae845e6dcd2be0b55a2da43cd2 \ + --hash=sha256:941103c9be18ac8daf7b7adca8228f8ed6bb7a1849020f643b3a14d15b1924d9 \ + --hash=sha256:97aeba56665b4c3235a0e52b2c2f5ae9cd071b8a8310ad27bddb3f7fb30e9aa2 \ + --hash=sha256:a39455728aabd58ceabb03c90e12f71fd30fa69615760a075b9fec596456ccc3 \ + --hash=sha256:a3a52f6156e73e7ccb0f8cced536adccb7042be67cb45f9562e12b319c119da6 \ + --hash=sha256:a668ce24de96165bb239160b3d854943128f4334822900534f2fe947930e5770 \ + --hash=sha256:aabf5777b5c8ca26f7824cb4a120a740c9588ed58df9b2d196ce92fba42ff8dc \ + --hash=sha256:aec5cf2fd867b4ff45b9959f8b20ea3993fc93e63c7363fe6851424c8a7e7c23 \ + --hash=sha256:b2379fa7ed44ddecb5bfe4e48577d752db9fc10be00a6b7446e9663ba143de26 \ + --hash=sha256:b4ececa40ac28afa90871c2cc2b9ffd2ff0bf749380fbdf57d165fd23da353aa \ + --hash=sha256:b74557b16e390ec12dca509bce9264c3bbd128f8a2c376eaa68003d7f327276d \ + --hash=sha256:b80aa5095cd3109962a298ce14110ae16b8c1aece8b72f9dafe81cf597ad80b3 \ + --hash=sha256:b93590ae81f7010dbe380cdeab6f515902ebcbefe0b9327cc4804d74e93ae69d \ + --hash=sha256:b96d5f26b05d03cc60f11a7761a5ded1741da411e7fe0909e27a5e6a0cb7b034 \ + --hash=sha256:bd3d54f38609ff308209bd43acea66061494157703364ae40c951f83ba99a1a9 \ + --hash=sha256:bfea2a5f0b4d8d43adf9d7b8bf019fb46fdd10a2e5cde477fbcb9d1fa08c68e1 \ + --hash=sha256:c007fe8a43d43b3969e8469004e9845944f1a80e6acd47c150856bb87f230c56 \ + --hash=sha256:c1df3d34aced70add6f867a8cf413e299177e0c22660cc767218373d0779487b \ + --hash=sha256:c23e27686783f60290e36827f9c626e63154b82b116d7fe9adba1fda36da706c \ + --hash=sha256:c9e19dd6e28fdcaa5a1de679aec4141f691023916427ef9bae8584f9c2fb3b0e \ + --hash=sha256:d0d2568a8c11bf8225044aa94409e21da0cb09dcdafe9ecd10250b2baad531a9 \ + --hash=sha256:d38548150c39b74aeeb0ce8ee1d8e82696f4a4e16ddc6de7b1d8823f7de4b9b5 \ + --hash=sha256:dc799088c08fa04e43144b164feb0c13f9a0bc40503f8df3e9fde58a3c0c101e \ + --hash=sha256:e536c98a7626a98feb2d3eaf75944ef6f3dbee447e1f841eae16f2f0a72d8ddc \ + --hash=sha256:e56ba91f47764cc14f1daacd723e3e82d1a89d783f0f5afe9c364b8bb491ccdb \ + --hash=sha256:e672ba74fbc2dc8eea59fb6d4aed6845e6905fc2a8afe93175d94a83ba2a01a0 \ + --hash=sha256:e7b576130c69225432866fe2f4a469a85a54ade141d96fd396dffcf607b558f8 \ + --hash=sha256:e96cea19e34778f8d59fe40775a7a574d95816eb150850a85a7a4c8f4b94ac69 \ + --hash=sha256:eceb81a8d74f9267ef4081e246ffd6d129da5d87e37a77c9bde550cb04870c1c \ + --hash=sha256:ed2e99c456e3fadd05c991f8f437ef902e00eedf34320ba2b0842bd1c3ca3a75 \ + --hash=sha256:f14f8f046c14563f8eb3f45f499cc658ab8d10072961e07225e507adb700e93f \ + --hash=sha256:f15489ba13d61f670dcc96772e733aad1a6f9c429cc27574c6cdaed82d0146ad \ + --hash=sha256:f31d95a179f8d64d90f6831d71fa93290893a33148d890ba15de25642c5d075b \ + --hash=sha256:f41a7489d32336dbf2199c8c0a215390a751c5b014c2c1c5366e817202e9cdf7 + # via pydantic +pyopenssl==26.4.0 \ + --hash=sha256:28dfcce0162b9211413e26dfbfdf1d24317fbeba18fc93c12400a1856b2a0bc7 \ + --hash=sha256:f0eb0cb2d581d3ad2b9c489468485e7f2ab6727d08401bcf9d824c3caddf3c1c + # via xtrace-ai-sdk +python-dateutil==2.9.0.post0 \ + --hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \ + --hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427 + # via botocore +python-dotenv==1.2.2 \ + --hash=sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a \ + --hash=sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3 + # via xtrace-ai-sdk +s3transfer==0.16.0 \ + --hash=sha256:18e25d66fed509e3868dc1572b3f427ff947dd2c56f844a5bf09481ad3f3b2fe \ + --hash=sha256:8e990f13268025792229cd52fa10cb7163744bf56e719e0b9cb925ab79abf920 + # via boto3 +six==1.17.0 \ + --hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \ + --hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81 + # via python-dateutil +sniffio==1.3.1 \ + --hash=sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2 \ + --hash=sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc + # via openai +tqdm==4.67.3 \ + --hash=sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb \ + --hash=sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf + # via + # openai + # xtrace-ai-sdk +typing-extensions==4.15.0 \ + --hash=sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466 \ + --hash=sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548 + # via + # aiosignal + # anyio + # openai + # pydantic + # pydantic-core + # pyopenssl + # typing-inspection + # xtrace-ai-sdk +typing-inspection==0.4.2 \ + --hash=sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7 \ + --hash=sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464 + # via pydantic +urllib3==2.6.3 \ + --hash=sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed \ + --hash=sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4 + # via botocore +yarl==1.23.0 \ + --hash=sha256:03214408cfa590df47728b84c679ae4ef00be2428e11630277be0727eba2d7cc \ + --hash=sha256:041b1a4cefacf65840b4e295c6985f334ba83c30607441ae3cf206a0eed1a2e4 \ + --hash=sha256:0793e2bd0cf14234983bbb371591e6bea9e876ddf6896cdcc93450996b0b5c85 \ + --hash=sha256:0e40111274f340d32ebcc0a5668d54d2b552a6cca84c9475859d364b380e3222 \ + --hash=sha256:115136c4a426f9da976187d238e84139ff6b51a20839aa6e3720cd1026d768de \ + --hash=sha256:13a563739ae600a631c36ce096615fe307f131344588b0bc0daec108cdb47b25 \ + --hash=sha256:16c6994ac35c3e74fb0ae93323bf8b9c2a9088d55946109489667c510a7d010e \ + --hash=sha256:170e26584b060879e29fac213e4228ef063f39128723807a312e5c7fec28eff2 \ + --hash=sha256:17235362f580149742739cc3828b80e24029d08cbb9c4bda0242c7b5bc610a8e \ + --hash=sha256:1932b6b8bba8d0160a9d1078aae5838a66039e8832d41d2992daa9a3a08f7860 \ + --hash=sha256:1b6b572edd95b4fa8df75de10b04bc81acc87c1c7d16bcdd2035b09d30acc957 \ + --hash=sha256:1c3a3598a832590c5a3ce56ab5576361b5688c12cb1d39429cf5dba30b510760 \ + --hash=sha256:1c57676bdedc94cd3bc37724cf6f8cd2779f02f6aba48de45feca073e714fe52 \ + --hash=sha256:1dc702e42d0684f42d6519c8d581e49c96cefaaab16691f03566d30658ee8788 \ + --hash=sha256:21d1b7305a71a15b4794b5ff22e8eef96ff4a6d7f9657155e5aa419444b28912 \ + --hash=sha256:23f371bd662cf44a7630d4d113101eafc0cfa7518a2760d20760b26021454719 \ + --hash=sha256:2569b67d616eab450d262ca7cb9f9e19d2f718c70a8b88712859359d0ab17035 \ + --hash=sha256:2803ed8b21ca47a43da80a6fd1ed3019d30061f7061daa35ac54f63933409412 \ + --hash=sha256:2e27c8841126e017dd2a054a95771569e6070b9ee1b133366d8b31beb5018a41 \ + --hash=sha256:34b6cf500e61c90f305094911f9acc9c86da1a05a7a3f5be9f68817043f486e4 \ + --hash=sha256:3650dc2480f94f7116c364096bc84b1d602f44224ef7d5c7208425915c0475dd \ + --hash=sha256:39004f0ad156da43e86aa71f44e033de68a44e5a31fc53507b36dd253970054a \ + --hash=sha256:394906945aa8b19fc14a61cf69743a868bb8c465efe85eee687109cc540b98f4 \ + --hash=sha256:3ceb13c5c858d01321b5d9bb65e4cf37a92169ea470b70fec6f236b2c9dd7e34 \ + --hash=sha256:411225bae281f114067578891bc75534cfb3d92a3b4dfef7a6ca78ba354e6069 \ + --hash=sha256:44bb7bef4ea409384e3f8bc36c063d77ea1b8d4a5b2706956c0d6695f07dcc25 \ + --hash=sha256:4503053d296bc6e4cbd1fad61cf3b6e33b939886c4f249ba7c78b602214fabe2 \ + --hash=sha256:4764a6a7588561a9aef92f65bda2c4fb58fe7c675c0883862e6df97559de0bfb \ + --hash=sha256:4966242ec68afc74c122f8459abd597afd7d8a60dc93d695c1334c5fd25f762f \ + --hash=sha256:4a42e651629dafb64fd5b0286a3580613702b5809ad3f24934ea87595804f2c5 \ + --hash=sha256:4a59ba56f340334766f3a4442e0efd0af895fae9e2b204741ef885c446b3a1a8 \ + --hash=sha256:4c41e021bc6d7affb3364dc1e1e5fa9582b470f283748784bd6ea0558f87f42c \ + --hash=sha256:5023346c4ee7992febc0068e7593de5fa2bf611848c08404b35ebbb76b1b0512 \ + --hash=sha256:50f9d8d531dfb767c565f348f33dd5139a6c43f5cbdf3f67da40d54241df93f6 \ + --hash=sha256:51430653db848d258336cfa0244427b17d12db63d42603a55f0d4546f50f25b5 \ + --hash=sha256:531ef597132086b6cf96faa7c6c1dcd0361dd5f1694e5cc30375907b9b7d3ea9 \ + --hash=sha256:53ad387048f6f09a8969631e4de3f1bf70c50e93545d64af4f751b2498755072 \ + --hash=sha256:53b1ea6ca88ebd4420379c330aea57e258408dd0df9af0992e5de2078dc9f5d5 \ + --hash=sha256:575aa4405a656e61a540f4a80eaa5260f2a38fff7bfdc4b5f611840d76e9e277 \ + --hash=sha256:578110dd426f0d209d1509244e6d4a3f1a3e9077655d98c5f22583d63252a08a \ + --hash=sha256:5ee586fb17ff8f90c91cf73c6108a434b02d69925f44f5f8e0d7f2f260607eae \ + --hash=sha256:609d3614d78d74ebe35f54953c5bbd2ac647a7ddb9c30a5d877580f5e86b22f2 \ + --hash=sha256:62694e275c93d54f7ccedcfef57d42761b2aad5234b6be1f3e3026cae4001cd4 \ + --hash=sha256:63e92247f383c85ab00dd0091e8c3fa331a96e865459f5ee80353c70a4a42d70 \ + --hash=sha256:682bae25f0a0dd23a056739f23a134db9f52a63e2afd6bfb37ddc76292bbd723 \ + --hash=sha256:6b41389c19b07c760c7e427a3462e8ab83c4bb087d127f0e854c706ce1b9215c \ + --hash=sha256:6e87a6e8735b44816e7db0b2fbc9686932df473c826b0d9743148432e10bb9b9 \ + --hash=sha256:6f0fd84de0c957b2d280143522c4f91a73aada1923caee763e24a2b3fda9f8a5 \ + --hash=sha256:70efd20be968c76ece7baa8dafe04c5be06abc57f754d6f36f3741f7aa7a208e \ + --hash=sha256:71d006bee8397a4a89f469b8deb22469fe7508132d3c17fa6ed871e79832691c \ + --hash=sha256:73309162a6a571d4cbd3b6a1dcc703c7311843ae0d1578df6f09be4e98df38d4 \ + --hash=sha256:76855800ac56f878847a09ce6dba727c93ca2d89c9e9d63002d26b916810b0a2 \ + --hash=sha256:7c6b9461a2a8b47c65eef63bb1c76a4f1c119618ffa99ea79bc5bb1e46c5821b \ + --hash=sha256:803a3c3ce4acc62eaf01eaca1208dcf0783025ef27572c3336502b9c232005e7 \ + --hash=sha256:8419ebd326430d1cbb7efb5292330a2cf39114e82df5cc3d83c9a0d5ebeaf2f2 \ + --hash=sha256:85610b4f27f69984932a7abbe52703688de3724d9f72bceb1cca667deff27474 \ + --hash=sha256:877b0738624280e34c55680d6054a307aa94f7d52fa0e3034a9cc6e790871da7 \ + --hash=sha256:88f9fb0116fbfcefcab70f85cf4b74a2b6ce5d199c41345296f49d974ddb4123 \ + --hash=sha256:93a784271881035ab4406a172edb0faecb6e7d00f4b53dc2f55919d6c9688595 \ + --hash=sha256:94f8575fbdf81749008d980c17796097e645574a3b8c28ee313931068dad14fe \ + --hash=sha256:95451e6ce06c3e104556d73b559f5da6c34a069b6b62946d3ad66afcd51642ea \ + --hash=sha256:99c8a9ed30f4164bc4c14b37a90208836cbf50d4ce2a57c71d0f52c7fb4f7598 \ + --hash=sha256:9a18d6f9359e45722c064c97464ec883eb0e0366d33eda61cb19a244bf222679 \ + --hash=sha256:9cbf44c5cb4a7633d078788e1b56387e3d3cf2b8139a3be38040b22d6c3221c8 \ + --hash=sha256:9ee33b875f0b390564c1fb7bc528abf18c8ee6073b201c6ae8524aca778e2d83 \ + --hash=sha256:a0e317df055958a0c1e79e5d2aa5a5eaa4a6d05a20d4b0c9c3f48918139c9fc6 \ + --hash=sha256:a2df6afe50dea8ae15fa34c9f824a3ee958d785fd5d089063d960bae1daa0a3f \ + --hash=sha256:a31de1613658308efdb21ada98cbc86a97c181aa050ba22a808120bb5be3ab94 \ + --hash=sha256:a3d2bff8f37f8d0f96c7ec554d16945050d54462d6e95414babaa18bfafc7f51 \ + --hash=sha256:a41bcf68efd19073376eb8cf948b8d9be0af26256403e512bb18f3966f1f9120 \ + --hash=sha256:a82836cab5f197a0514235aaf7ffccdc886ccdaa2324bc0aafdd4ae898103039 \ + --hash=sha256:a8d00f29b42f534cc8aa3931cfe773b13b23e561e10d2b26f27a8d309b0e82a1 \ + --hash=sha256:aafe5dcfda86c8af00386d7781d4c2181b5011b7be3f2add5e99899ea925df05 \ + --hash=sha256:ac09d42f48f80c9ee1635b2fcaa819496a44502737660d3c0f2ade7526d29144 \ + --hash=sha256:aecfed0b41aa72b7881712c65cf764e39ce2ec352324f5e0837c7048d9e6daaa \ + --hash=sha256:b2c6b50c7b0464165472b56b42d4c76a7b864597007d9c085e8b63e185cf4a7a \ + --hash=sha256:b35d13d549077713e4414f927cdc388d62e543987c572baee613bf82f11a4b99 \ + --hash=sha256:b39cb32a6582750b6cc77bfb3c49c0f8760dc18dc96ec9fb55fbb0f04e08b928 \ + --hash=sha256:b5405bb8f0e783a988172993cfc627e4d9d00432d6bbac65a923041edacf997d \ + --hash=sha256:baaf55442359053c7d62f6f8413a62adba3205119bcb6f49594894d8be47e5e3 \ + --hash=sha256:bd654fad46d8d9e823afbb4f87c79160b5a374ed1ff5bde24e542e6ba8f41434 \ + --hash=sha256:be61f6fff406ca40e3b1d84716fde398fc08bc63dd96d15f3a14230a0973ed86 \ + --hash=sha256:bf49a3ae946a87083ef3a34c8f677ae4243f5b824bfc4c69672e72b3d6719d46 \ + --hash=sha256:c4a80f77dc1acaaa61f0934176fccca7096d9b1ff08c8ba9cddf5ae034a24319 \ + --hash=sha256:c75eb09e8d55bceb4367e83496ff8ef2bc7ea6960efb38e978e8073ea59ecb67 \ + --hash=sha256:c7f8dc16c498ff06497c015642333219871effba93e4a2e8604a06264aca5c5c \ + --hash=sha256:c8aa34a5c864db1087d911a0b902d60d203ea3607d91f615acd3f3108ac32169 \ + --hash=sha256:cbb0fef01f0c6b38cb0f39b1f78fc90b807e0e3c86a7ff3ce74ad77ce5c7880c \ + --hash=sha256:cde9a2ecd91668bcb7f077c4966d8ceddb60af01b52e6e3e2680e4cf00ad1a59 \ + --hash=sha256:d1009abedb49ae95b136a8904a3f71b342f849ffeced2d3747bf29caeda218c4 \ + --hash=sha256:d38c1e8231722c4ce40d7593f28d92b5fc72f3e9774fe73d7e800ec32299f63a \ + --hash=sha256:d53834e23c015ee83a99377db6e5e37d8484f333edb03bd15b4bc312cc7254fb \ + --hash=sha256:d7504f2b476d21653e4d143f44a175f7f751cd41233525312696c76aa3dbb23f \ + --hash=sha256:dc52310451fc7c629e13c4e061cbe2dd01684d91f2f8ee2821b083c58bd72432 \ + --hash=sha256:dd00607bffbf30250fe108065f07453ec124dbf223420f57f5e749b04295e090 \ + --hash=sha256:e09fd068c2e169a7070d83d3bde728a4d48de0549f975290be3c108c02e499b4 \ + --hash=sha256:e0fd068364a6759bc794459f0a735ab151d11304346332489c7972bacbe9e72b \ + --hash=sha256:e5723c01a56c5028c807c701aa66722916d2747ad737a046853f6c46f4875543 \ + --hash=sha256:e7b0460976dc75cb87ad9cc1f9899a4b97751e7d4e77ab840fc9b6d377b8fd24 \ + --hash=sha256:e9d9a4d06d3481eab79803beb4d9bd6f6a8e781ec078ac70d7ef2dcc29d1bea5 \ + --hash=sha256:ead11956716a940c1abc816b7df3fa2b84d06eaed8832ca32f5c5e058c65506b \ + --hash=sha256:f2af5c81a1f124609d5f33507082fc3f739959d4719b56877ab1ee7e7b3d602b \ + --hash=sha256:f40e782d49630ad384db66d4d8b73ff4f1b8955dc12e26b09a3e3af064b3b9d6 \ + --hash=sha256:f514f6474e04179d3d33175ed3f3e31434d3130d42ec153540d5b157deefd735 \ + --hash=sha256:f69f57305656a4852f2a7203efc661d8c042e6cc67f7acd97d8667fb448a426e \ + --hash=sha256:fb1e8b8d66c278b21d13b0a7ca22c41dd757a7c209c6b12c313e445c31dd3b28 \ + --hash=sha256:fb4948814a2a98e3912505f09c9e7493b1506226afb1f881825368d6fb776ee3 \ + --hash=sha256:fda207c815b253e34f7e1909840fd14299567b1c0eb4908f8c2ce01a41265401 \ + --hash=sha256:fe8f8f5e70e6dbdfca9882cd9deaac058729bcf323cf7a58660901e55c9c94f6 \ + --hash=sha256:fffc45637bcd6538de8b85f51e3df3223e4ad89bccbfca0481c08c7fc8b7ed7d + # via aiohttp diff --git a/experiments/bfv_nitro/service.py b/experiments/bfv_nitro/service.py new file mode 100644 index 0000000..0a9fce9 --- /dev/null +++ b/experiments/bfv_nitro/service.py @@ -0,0 +1,94 @@ +"""Measured, single-owner Nitro evaluator; run INSIDE the enclave image. + +The entrypoint fixes the review BFV policy and residue backend. Parent messages +cannot select code, libraries, parameters, execution backends or output hashes. +There is no debug/mock-attestation mode, secret import, shell or signing endpoint. +Only one request is serviced at a time, bounding memory and concurrent parsing. +""" + +from __future__ import annotations + +import resource +import socket +import time + +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy +from xtrace_sdk.x_vec.crypto.bfv_attested_client import ( + BFVAttestedServer, + NITRO_HELLO_BYTES, + NITRO_QUERY_OVERHEAD, +) +from xtrace_sdk.x_vec.crypto.bfv_nitro import NitroNSM +from xtrace_sdk.x_vec.crypto.bfv_security import BFVProtocolError, _pack + +from .transport import REGISTER, ATTEST, SEARCH, ERROR, receive_header, receive_exact, send_frame + + +class EnclaveApplication: + """One immutable server, initialized once through its signed registration. + + Constructor arguments are trusted local code configuration for tests/builds. + The deployed main function supplies only the real NSM and fixed defaults. + """ + + def __init__(self, nsm, *, policy=None, backend="residue"): + self.nsm = nsm + self.policy = policy or bfv_review_policy() + self.backend = backend + self.server = None + + def handle(self, stream: socket.socket, *, timeout=60.0): + """Handle one bounded RPC, checking operation state before reading its body.""" + deadline = time.monotonic() + timeout + try: + kind, length = receive_header(stream, deadline) + limits = { + REGISTER: self.policy.max_setup_bytes + 1024, + ATTEST: NITRO_HELLO_BYTES, + SEARCH: self.policy.max_query_bytes + NITRO_QUERY_OVERHEAD, + } + if kind not in limits or length > limits[kind]: + raise BFVProtocolError("Invalid Nitro request length") + if (kind == REGISTER) != (self.server is None): + raise BFVProtocolError("Invalid Nitro service state") + payload = receive_exact(stream, length, deadline) + if kind == REGISTER: + self.server = BFVAttestedServer.from_registration( + payload, self.nsm, policy=self.policy, backend=self.backend + ) + result = b"registered" + elif kind == ATTEST: + result = _pack(self.server.attest(payload)) + else: + result = _pack(self.server.search(payload)) + stream.settimeout(timeout) + send_frame(stream, kind, result, self.policy.max_response_bytes + 256) + except (ValueError, TypeError, OverflowError, OSError): + # No decryption occurs here. Do not reflect exception strings, + # payloads, keys or arbitrary host-controlled content into logs. + try: + stream.settimeout(1.0) + send_frame(stream, ERROR, b"request rejected", 64) + except OSError: + pass + + +def main(): + """Start the fixed vsock service with real NSM; fail if hardware is absent.""" + resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) + nsm = NitroNSM() + try: + app = EnclaveApplication(nsm) + with socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM) as listener: + listener.bind((socket.VMADDR_CID_ANY, 5000)) + listener.listen(4) + while True: + stream, _ = listener.accept() + with stream: + app.handle(stream) + finally: + nsm.close() + + +if __name__ == "__main__": + main() diff --git a/experiments/bfv_nitro/transport.py b/experiments/bfv_nitro/transport.py new file mode 100644 index 0000000..2ad9a6a --- /dev/null +++ b/experiments/bfv_nitro/transport.py @@ -0,0 +1,127 @@ +"""Bounded request/response framing for an untrusted TCP/vsock transport. + +This module supplies no trust: the SDK verifies attestation and receipts. Frame +lengths and operation kinds are checked before buffering, with a deadline for +the entire read. Each RPC uses a new connection; no custom secure channel or +key-exchange scheme is introduced. +""" + +from __future__ import annotations + +import socket +import struct +import time + +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy +from xtrace_sdk.x_vec.crypto.bfv_attested_client import ( + NITRO_HELLO_BYTES, + NITRO_QUERY_OVERHEAD, + NITRO_RECEIPT_BYTES, +) +from xtrace_sdk.x_vec.crypto.bfv_nitro import MAX_NITRO_DOCUMENT_BYTES +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + BFVProtocolError, + _bytes, + _pack, + _record, + _unpack, +) + + +REGISTER, ATTEST, SEARCH, ERROR = 1, 2, 3, 0 +_HEADER = struct.Struct("!4sBI") +_MAGIC = b"XBN1" + + +def receive_exact(stream: socket.socket, length: int, deadline: float) -> bytes: + """Read a caller-bounded length; a stalled peer cannot renew the deadline.""" + result = bytearray() + while len(result) < length: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError("Nitro frame deadline exceeded") + stream.settimeout(remaining) + chunk = stream.recv(min(65536, length - len(result))) + if not chunk: + raise BFVProtocolError("Truncated Nitro frame") + result.extend(chunk) + return bytes(result) + + +def receive_header(stream: socket.socket, deadline: float) -> tuple[int, int]: + """Parse the fixed header without allocating space for its claimed body.""" + magic, kind, length = _HEADER.unpack(receive_exact(stream, _HEADER.size, deadline)) + if magic != _MAGIC or kind not in (REGISTER, ATTEST, SEARCH, ERROR): + raise BFVProtocolError("Invalid Nitro frame header") + return kind, length + + +def send_frame(stream: socket.socket, kind: int, payload: bytes, limit: int) -> None: + """Send a bounded frame; the caller configures the socket write timeout.""" + if type(payload) is not bytes or len(payload) > limit or len(payload) >= 1 << 32: + raise BFVProtocolError("Nitro frame exceeds its size limit") + stream.sendall(_HEADER.pack(_MAGIC, kind, len(payload))) + stream.sendall(payload) + + +class NitroRemote: + """Owner-side RPC helper for a relay, with no decryption or acceptance feedback. + + Connect through an authenticated tunnel/TLS deployment for transport privacy + and access control. The Nitro-required client independently authenticates + evidence and responses even if the relay is malicious. + """ + + def __init__( + self, + host: str = "127.0.0.1", + port: int = 9000, + *, + policy: BFVExecutionPolicy | None = None, + timeout: float = 60.0, + ) -> None: + if not 0 < timeout <= 300: + raise ValueError("Use a transport timeout in (0, 300] seconds") + self.host, self.port, self.timeout = host, port, timeout + self.policy = policy or bfv_review_policy() + + def _call(self, kind: int, payload: bytes, request_limit: int, response_limit: int) -> bytes: + with socket.create_connection((self.host, self.port), timeout=self.timeout) as stream: + send_frame(stream, kind, payload, request_limit) + deadline = time.monotonic() + self.timeout + returned, length = receive_header(stream, deadline) + if returned not in (kind, ERROR) or length > ( + 64 if returned == ERROR else response_limit + ): + raise BFVProtocolError("Unexpected Nitro response frame") + result = receive_exact(stream, length, deadline) + if returned == ERROR: + raise BFVProtocolError("Nitro service rejected the request") + return result + + def register(self, packet: bytes) -> None: + """Upload a signed public setup; acknowledgement does not establish trust.""" + if self._call(REGISTER, packet, self.policy.max_setup_bytes + 1024, 32) != b"registered": + raise BFVProtocolError("Invalid Nitro registration acknowledgement") + + def attest(self, hello: bytes) -> tuple[bytes, bytes]: + """Return bounded evidence and possession proof for the SDK to validate.""" + result = self._call(ATTEST, hello, NITRO_HELLO_BYTES, MAX_NITRO_DOCUMENT_BYTES + 128) + document, proof = _record(_unpack(result, limit=MAX_NITRO_DOCUMENT_BYTES + 128), 2) + if type(document) is not bytes or len(document) > MAX_NITRO_DOCUMENT_BYTES: + raise BFVProtocolError("Invalid Nitro evidence") + return document, _bytes(proof, 64) + + def search(self, query: bytes) -> tuple[bytes, bytes]: + """Return encrypted bytes and receipt; never perform private work here.""" + result = self._call( + SEARCH, + query, + self.policy.max_query_bytes + NITRO_QUERY_OVERHEAD, + self.policy.max_response_bytes + 256, + ) + response, receipt = _record(_unpack(result, limit=self.policy.max_response_bytes + 256), 2) + if type(response) is not bytes or len(response) > self.policy.max_response_bytes: + raise BFVProtocolError("Invalid Nitro response") + return response, _bytes(receipt, NITRO_RECEIPT_BYTES) diff --git a/pyproject.toml b/pyproject.toml index fae9b0b..514dbb9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -62,6 +62,11 @@ include = ["src/xtrace_sdk", "README.md", "LICENSE", "pyproject.toml", "hatch_bu exclude = ["tests", "docs", "benchmarks"] [project.optional-dependencies] +bfv-nitro = [ + "cbor2==6.1.4", + "pyOpenSSL==26.4.0", + "cryptography==50.0.1", +] cli = [ "typer>=0.12.0", "rich>=13.0.0", diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py b/src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py new file mode 100644 index 0000000..96b2158 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py @@ -0,0 +1,518 @@ +"""One attested public BFV evaluation, with mandatory approval before decryption. + +The owner retains BFV and request-signing secrets. The Nitro evaluator generates +its own ephemeral receipt key and computes each result itself. Its host knows +only public BFV data and the legacy transport MAC key. Raw and recomputing BFV +APIs remain separate; this protocol never falls back to them after rejection. +""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import secrets +import threading +import time +from collections.abc import Sequence +from dataclasses import asdict +from typing import Protocol + +from Crypto.Signature import eddsa + +from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy, hamming_noise_bound +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_guarded_client import _CIRCUIT +from xtrace_sdk.x_vec.crypto.bfv_nitro import ( + MAX_NITRO_DOCUMENT_BYTES, + NitroAttestationPolicy, + NitroIdentity, + verify_nitro_attestation, +) +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + BFVProtocolError, + _bytes, + _key, + _pack, + _record, + _unpack, + protect_bfv_state, + unprotect_bfv_state, +) +from xtrace_sdk.x_vec.crypto.bfv_verified_client import BFVVerifiedClient, BFVVerifiedServer +from xtrace_sdk.x_vec.crypto.encryption.bfv_evaluator import BFVServerBackend +from xtrace_sdk.x_vec.crypto.encryption.bfv_private import BFVPrivateBackend + + +_REGISTER = b"XBNI01" +_HELLO = b"XBNH01" +_QUERY = b"XBNQ01" +_RECEIPT = b"XBNR01" +_OWNER_CONTEXT = b"XTRACE-BFV-NITRO-OWNER-v1" +_POP_CONTEXT = b"XTRACE-BFV-NITRO-ENROLL-v1" +_RECEIPT_CONTEXT = b"XTRACE-BFV-NITRO-RECEIPT-v1" +_STATE_CONTEXT = b"XTRACE-BFV-NITRO-PRIVATE-STATE-v1" +# Shared with the bounded transport; changing a format requires a new version. +NITRO_HELLO_BYTES = 6 + 32 + 32 + 64 +NITRO_QUERY_OVERHEAD = 6 + 32 + 64 +NITRO_RECEIPT_BYTES = 6 + 4 * 32 + 64 +_SESSION_SECONDS = 300 +_MAX_ATTESTATIONS = 1024 + + +def _verifier(public_key: bytes, context: bytes) -> eddsa.EdDSASigScheme: + """Import an Ed25519 identity, rejecting small-order keys before use.""" + key = eddsa.import_public_key(_bytes(public_key, 32)) + if (key.pointQ * 8).is_point_at_infinity(): + raise BFVProtocolError("Invalid BFV signing identity") + return eddsa.new(key, "rfc8032", context=context) + + +def _epoch(index_epoch: int) -> bytes: + """Encode an owner-maintained index version without bool/integer coercion.""" + if type(index_epoch) is not int or not 0 <= index_epoch < 1 << 64: + raise ValueError("BFV index epoch must be an unsigned 64-bit integer") + return index_epoch.to_bytes(8, "big") + + +def _policy_digest(policy: BFVExecutionPolicy) -> bytes: + """Commit to local arithmetic AND admission limits, independent of map order.""" + return hashlib.sha256( + json.dumps(asdict(policy), sort_keys=True, separators=(",", ":")).encode() + ).digest() + + +def _context( + setup_digest: bytes, owner: bytes, index_epoch: int, policy: BFVExecutionPolicy +) -> bytes: + """Bind the circuit, actual setup, epoch, owner and execution policy together.""" + return hashlib.sha256( + b"XTRACE-BFV-NITRO-CONTEXT-v1\0" + + _CIRCUIT + + _policy_digest(policy) + + _bytes(setup_digest, 32) + + _bytes(owner, 32) + + _epoch(index_epoch) + ).digest() + + +def _session_id(context: bytes, nonce: bytes, public_key: bytes) -> bytes: + """Separate renewals and enclave restarts, even for an unchanged BFV index.""" + return hashlib.sha256(b"XTRACE-BFV-NITRO-SESSION-v1\0" + context + nonce + public_key).digest() + + +class _Attestor(Protocol): + """Protected-runtime evidence source; the remote client still verifies AWS PKI.""" + + def random_seed(self) -> bytes: ... + def attest(self, public_key: bytes, nonce: bytes, context: bytes) -> bytes: ... + + +class BFVAttestedServer: + """Public-only evaluator intended to execute INSIDE the measured Nitro image. + + A local instance is not a TEE. The real service passes NitroNSM as attestor; + a synthetic provider cannot produce evidence accepted under the AWS root. + Every signing path has a fixed purpose: enrollment or an internally computed + search result. No arbitrary signing, host-result or external-cache API exists. + One owner, one index and one active attestation session are supported. + """ + + def __init__( + self, + setup: bytes, + authentication_key: bytes, + owner_public_key: bytes, + index_epoch: int, + attestor: _Attestor, + *, + policy: BFVExecutionPolicy | None = None, + backend: BFVServerBackend = "residue", + ) -> None: + self.policy = policy or bfv_review_policy() + if type(setup) is not bytes or len(setup) > self.policy.max_setup_bytes: + raise BFVProtocolError("BFV setup exceeds the local size policy") + self._owner = _verifier(owner_public_key, _OWNER_CONTEXT) + self._context = _context( + hashlib.sha256(setup).digest(), owner_public_key, index_epoch, self.policy + ) + self._server = BFVVerifiedServer( + setup, authentication_key, policy=self.policy, backend=backend + ) + if not hamming_noise_bound(self.policy, self._server._count).sufficient: + raise BFVProtocolError("BFV circuit exceeds the conservative correctness bound") + self._attestor = attestor + signing_key = eddsa.import_private_key(_key(attestor.random_seed())) + self._public_key = signing_key.public_key().export_key(format="raw") + self._public_spki = signing_key.public_key().export_key(format="DER") + self._signer = eddsa.new(signing_key, "rfc8032", context=_RECEIPT_CONTEXT) + self._enrollment_signer = eddsa.new(signing_key, "rfc8032", context=_POP_CONTEXT) + self._session: bytes | None = None + self._deadline = 0.0 + self._seen_nonces: set[bytes] = set() + self._lock = threading.Lock() + + @classmethod + def from_registration( + cls, + packet: bytes, + attestor: _Attestor, + *, + policy: BFVExecutionPolicy | None = None, + backend: BFVServerBackend = "residue", + ) -> BFVAttestedServer: + """Verify the enrolling owner's request before importing BFV key JSON. + + Enrollment is self-authorized: the first registrant can claim this empty + service. The client accepts only attestation bound to ITS owner key and + setup. Hosting/account access control is a separate deployment concern. + A hostile parent can claim or stop the service, causing denial of service. + """ + policy = policy or bfv_review_policy() + setup, auth, owner, epoch, signature = _record( + _unpack(packet, limit=policy.max_setup_bytes + 1024), 5 + ) + if type(setup) is not bytes or len(setup) > policy.max_setup_bytes: + raise BFVProtocolError("BFV registration exceeds its size limit") + context = _context(hashlib.sha256(setup).digest(), owner, epoch, policy) + _verifier(owner, _OWNER_CONTEXT).verify( + _REGISTER + context + hashlib.sha256(_key(auth)).digest(), _bytes(signature, 64) + ) + return cls(setup, auth, owner, epoch, attestor, policy=policy, backend=backend) + + def attest(self, hello: bytes) -> tuple[bytes, bytes]: + """Authenticate an owner challenge, obtain NSM evidence and prove key custody. + + Renewal replaces the one active session. Old requests cannot enter it. + Nonces are consumed before requesting evidence; failed enrollment needs + a fresh owner challenge, and replay state stays bounded. + """ + with self._lock: + _bytes(hello, NITRO_HELLO_BYTES) + if hello[:6] != _HELLO or hello[6:38] != self._context: + raise BFVProtocolError("BFV enrollment context mismatch") + self._owner.verify(hello[:-64], hello[-64:]) + nonce = hello[38:70] + if nonce in self._seen_nonces or len(self._seen_nonces) >= _MAX_ATTESTATIONS: + raise BFVProtocolError("BFV attestation replayed or budget exhausted") + self._seen_nonces.add(nonce) + document = self._attestor.attest(self._public_spki, nonce, self._context) + if type(document) is not bytes or not 0 < len(document) <= MAX_NITRO_DOCUMENT_BYTES: + raise BFVProtocolError("Nitro evidence exceeds its size limit") + proof = self._enrollment_signer.sign(hello[:-64] + hashlib.sha256(document).digest()) + self._session = _session_id(self._context, nonce, self._public_key) + self._deadline = time.monotonic() + _SESSION_SECONDS + return document, proof + + def search(self, request: bytes) -> tuple[bytes, bytes]: + """Verify owner/session authorization, evaluate once, and sign exact output. + + The serialized gate excludes renewal while a search is running. An + expired computation produces no receipt. Only public BFV input reaches + the arithmetic implementation; the signer never imports a BFV secret. + """ + with self._lock: + if ( + self._session is None + or time.monotonic() >= self._deadline + or type(request) is not bytes + or not NITRO_QUERY_OVERHEAD + < len(request) + <= self.policy.max_query_bytes + NITRO_QUERY_OVERHEAD + or request[:6] != _QUERY + or request[6:38] != self._session + ): + raise BFVProtocolError("BFV attested request rejected") + query = request[38:-64] + digest = hashlib.sha256(query).digest() + self._owner.verify(_QUERY + self._context + self._session + digest, request[-64:]) + response = self._server.search(query) # Exactly one complete BFV evaluation. + if time.monotonic() >= self._deadline: + raise BFVProtocolError("BFV attested session expired during evaluation") + message = ( + _RECEIPT + + self._context + + self._session + + digest + + hashlib.sha256(response).digest() + ) + return response, message + self._signer.sign(message) + + +class BFVAttestedClient: + """Nitro-required BFV session with an owner request key and local secret key. + + Supply exact PCR pins from a trusted build and an owner-maintained index + epoch. The public API cannot accept a synthetic root or a verifier callback. + Restores require fresh attestation; active sessions and pending requests are + never persisted. Trusted local Python code can access internals, as with the + other BFV clients; this is not an isolation boundary within one process. + """ + + def __init__( + self, + client: BFVClient, + authentication_key: bytes, + attestation_policy: NitroAttestationPolicy, + *, + index_epoch: int = 0, + policy: BFVExecutionPolicy | None = None, + private_backend: BFVPrivateBackend = "native", + ) -> None: + self._session = BFVVerifiedClient( + client, + authentication_key, + policy=policy or bfv_review_policy(), + private_backend=private_backend, + ) + self._configure(attestation_policy, index_epoch, secrets.token_bytes(32)) + + def _configure( + self, policy: NitroAttestationPolicy, index_epoch: int, owner_seed: bytes + ) -> None: + """Initialize owner credentials and an unattested state, also after restore.""" + _epoch(index_epoch) + if not isinstance(policy, NitroAttestationPolicy): + raise TypeError("Use an owner-configured NitroAttestationPolicy") + self.attestation_policy = policy + self.index_epoch = index_epoch + self._owner_seed = _key(owner_seed) + if hmac.compare_digest(owner_seed, self._session._authentication_key): + raise ValueError("The owner signing and server transport keys must differ") + key = eddsa.import_private_key(owner_seed) + self._owner_public = key.public_key().export_key(format="raw") + self._owner = eddsa.new(key, "rfc8032", context=_OWNER_CONTEXT) + self._gate = threading.RLock() + self._identity: NitroIdentity | None = None + self._enrolled_session: bytes | None = None + self._receipt_verifier: eddsa.EdDSASigScheme | None = None + self._deadline = 0.0 + self._hello: bytes | None = None + self._hello_started = 0.0 + + @property + def policy(self) -> BFVExecutionPolicy: + return self._session.policy + + @property + def setup_digest(self) -> bytes: + if self._session._setup_digest is None: + raise BFVProtocolError("Prepare a BFV index first") + return self._session._setup_digest + + @property + def vector_ids(self) -> tuple[str, ...]: + return self._session.vector_ids + + @property + def verification_state_bytes(self) -> int: + return self._session.verification_state_bytes + + def _context(self) -> bytes: + return _context(self.setup_digest, self._owner_public, self.index_epoch, self.policy) + + def prepare_index( + self, vectors: list[list[int]], *, vector_ids: Sequence[str] | None = None + ) -> bytes: + """Prepare the existing encrypted setup and private Hamming summary once.""" + with self._gate: + if not hamming_noise_bound(self.policy, len(vectors)).sufficient: + raise BFVProtocolError("BFV circuit exceeds the conservative correctness bound") + return self._session.prepare_index(vectors, vector_ids=vector_ids) + + def registration_packet(self, setup: bytes) -> bytes: + """Authorize public setup import without retaining a second index copy. + + This contains the legacy transport MAC key, which is intentionally not + an owner credential. No BFV secret, owner seed or private summary leaves. + The caller supplies the exact bytes returned by prepare_index. + """ + with self._gate: + if ( + type(setup) is not bytes + or len(setup) > self.policy.max_setup_bytes + or hashlib.sha256(setup).digest() != self.setup_digest + ): + raise BFVProtocolError("Registration is not the owner-prepared setup") + auth = self._session._authentication_key + signature = self._owner.sign( + _REGISTER + self._context() + hashlib.sha256(auth).digest() + ) + return _pack([setup, auth, self._owner_public, self.index_epoch, signature]) + + def begin_attestation(self) -> bytes: + """Start or renew enrollment, discarding old sessions and pending queries.""" + with self._gate: + context = self._context() + self._identity = self._receipt_verifier = self._enrolled_session = None + self._session.cancel_pending_queries() + self._hello_started = time.monotonic() + self._hello = _HELLO + context + secrets.token_bytes(32) + return self._hello + self._owner.sign(self._hello) + + def accept_attestation(self, document: bytes, proof_of_possession: bytes) -> None: + """Verify fresh AWS evidence and key possession against the pending challenge. + + Rejected evidence leaves the challenge pending until its original local + deadline. A valid enrollment consumes it once. No private BFV operation + occurs in this method, and no trust decision comes from the host. + """ + with self._gate: + try: + if self._hello is None or time.monotonic() - self._hello_started > ( + self.attestation_policy.handshake_timeout_seconds + ): + raise BFVProtocolError("No current Nitro challenge") + identity = verify_nitro_attestation( + document, + self.attestation_policy, + nonce=self._hello[38:70], + context=self._context(), + ) + _verifier(identity.public_key, _POP_CONTEXT).verify( + self._hello + hashlib.sha256(document).digest(), _bytes(proof_of_possession, 64) + ) + remaining = min( + self.attestation_policy.max_session_seconds, + identity.certificate_expires_at - time.time(), + ) + if remaining <= 0 or time.monotonic() - self._hello_started > ( + self.attestation_policy.handshake_timeout_seconds + ): + raise BFVProtocolError("Nitro enrollment expired") + self._identity = identity + self._receipt_verifier = _verifier(identity.public_key, _RECEIPT_CONTEXT) + self._enrolled_session = _session_id( + self._context(), self._hello[38:70], identity.public_key + ) + self._deadline = time.monotonic() + remaining + self._hello = None + except (ValueError, TypeError, OverflowError): + raise BFVProtocolError("Nitro enrollment rejected before decryption") from None + + def _require_session(self) -> bytes: + """Enforce both monotonic lease expiry and certificate wall-clock expiry.""" + if ( + self._identity is None + or self._enrolled_session is None + or time.monotonic() >= self._deadline + or time.time() >= self._identity.certificate_expires_at + ): + raise BFVProtocolError("Fresh Nitro attestation is required") + return self._enrolled_session + + def begin_query(self, query: list[int]) -> bytes: + """Encrypt an owner query and sign its digest under the attested session.""" + with self._gate: + session = self._require_session() + inner = self._session.begin_query(query) + self._require_session() + signature = self._owner.sign( + _QUERY + self._context() + session + hashlib.sha256(inner).digest() + ) + return _QUERY + session + inner + signature + + def finish_query(self, response: bytes, receipt: bytes) -> list[int]: + """Check provenance before BFV parsing/decryption and consume a ticket once.""" + with self._gate: + try: + session = self._require_session() + if type(response) is not bytes or len(response) > self.policy.max_response_bytes: + raise BFVProtocolError("BFV response exceeds its size limit") + _bytes(receipt, NITRO_RECEIPT_BYTES) + if ( + receipt[:6] != _RECEIPT + or receipt[6:38] != self._context() + or receipt[38:70] != session + or receipt[102:134] != hashlib.sha256(response).digest() + ): + raise BFVProtocolError("BFV receipt binding mismatch") + with self._session._lock: + if not any( + p.request_digest == receipt[70:102] for p in self._session._pending.values() + ): + raise BFVProtocolError("BFV receipt is not for a pending owner query") + if self._receipt_verifier is None: + raise BFVProtocolError("Missing attested receipt key") + self._receipt_verifier.verify(receipt[:134], receipt[134:]) + self._require_session() + except (ValueError, TypeError, OverflowError): + raise BFVProtocolError("BFV response rejected before decryption") from None + # The outer lock excludes renewal during private work; the inner + # session consumes the ticket before its own private decoding. + return self._session.finish_query(response) + + def cancel_pending_queries(self) -> None: + """Discard pending queries without renewing attestation or query budgets.""" + with self._gate: + self._session.cancel_pending_queries() + + def _storage_binding(self) -> bytes: + return hashlib.sha256( + _STATE_CONTEXT + + self.setup_digest + + _epoch(self.index_epoch) + + _policy_digest(self.policy) + + self.attestation_policy.digest() + ).digest() + + def protect_state(self, wrapping_key: bytes) -> bytes: + """Encrypt owner credentials and BFV state; omit enrollment and pending work. + + The epoch and trust policy are authenticated. The application must retain + its expected epoch durably; encrypted files alone do not prevent rollback. + """ + with self._gate: + if hmac.compare_digest(_key(wrapping_key), self._owner_seed): + raise ValueError("The wrapping and owner signing keys must differ") + return protect_bfv_state( + _pack([self._owner_seed, self._session.protect_state(wrapping_key)]), + wrapping_key, + self._storage_binding(), + ) + + @classmethod + def restore( + cls, + setup: bytes, + protected: bytes, + authentication_key: bytes, + wrapping_key: bytes, + attestation_policy: NitroAttestationPolicy, + *, + index_epoch: int, + policy: BFVExecutionPolicy | None = None, + private_backend: BFVPrivateBackend = "native", + ) -> BFVAttestedClient: + """Restore with the owner-maintained epoch/policy, requiring new attestation.""" + policy = policy or bfv_review_policy() + if type(setup) is not bytes or len(setup) > policy.max_setup_bytes: + raise BFVProtocolError("BFV setup exceeds its size limit") + binding = hashlib.sha256( + _STATE_CONTEXT + + hashlib.sha256(setup).digest() + + _epoch(index_epoch) + + _policy_digest(policy) + + attestation_policy.digest() + ).digest() + seed, inner = _record( + _unpack(unprotect_bfv_state(protected, wrapping_key, binding), limit=1024 * 1024), 2 + ) + _key(seed) + if hmac.compare_digest(_key(wrapping_key), seed): + raise BFVProtocolError("Invalid BFV wrapping key separation") + session = BFVVerifiedClient.restore( + setup, + inner, + authentication_key, + wrapping_key, + policy=policy, + private_backend=private_backend, + ) + if not hamming_noise_bound(policy, len(session.vector_ids)).sufficient: + raise BFVProtocolError("BFV circuit exceeds the conservative correctness bound") + result = cls.__new__(cls) + result._session = session + result._configure(attestation_policy, index_epoch, seed) + return result diff --git a/src/xtrace_sdk/x_vec/crypto/bfv_nitro.py b/src/xtrace_sdk/x_vec/crypto/bfv_nitro.py new file mode 100644 index 0000000..41f51d5 --- /dev/null +++ b/src/xtrace_sdk/x_vec/crypto/bfv_nitro.py @@ -0,0 +1,382 @@ +"""AWS Nitro evidence verification and the optional official NSM C-library adapter. + +OpenSSL validates X.509 paths; cryptography verifies the COSE ES384 signature; +cbor2 supplies bounded CBOR decoding. This module adds the application's strict +wire, measurement, freshness and key-binding policy. It is not an independently +reviewed attestation implementation. No test root or debug bypass is exposed. +""" + +from __future__ import annotations + +import ctypes +import hashlib +import hmac +import io +import math +import threading +import time +from collections import defaultdict +from collections.abc import Mapping +from dataclasses import dataclass +from datetime import UTC, datetime +from pathlib import Path +from types import MappingProxyType +from typing import Any + +from Crypto.Signature import eddsa + +from xtrace_sdk.x_vec.crypto.bfv_security import BFVProtocolError, _bytes, _pack + + +MAX_NITRO_DOCUMENT_BYTES = 16 * 1024 +# DER certificate fingerprint published in the AWS Nitro root-of-trust guide. +# The received CA bundle is UNTRUSTED until its root matches this local anchor. +_AWS_ROOT_SHA256 = bytes.fromhex("641a0321a3e244efe456463195d606317ed7cdcc3c1756e09893f3c68f79bb5b") +_ED25519_SPKI_PREFIX = bytes.fromhex("302a300506032b6570032100") + + +@dataclass(frozen=True) +class NitroAttestationPolicy: + """Owner-provisioned exact PCR pins and bounded evidence/session lifetimes. + + pcrs maps register numbers to 48-byte SHA-384 measurements. PCR0, PCR1 and + PCR2 are mandatory and nonzero; optional additional pins are conjunctive. + Durations are integer seconds. Evidence age is checked at enrollment; + max_session_seconds limits subsequent receipt acceptance without renewal. + Copying the map prevents caller mutation from silently changing trust. + """ + + pcrs: Mapping[int, bytes] + max_age_seconds: int = 120 + max_session_seconds: int = 300 + max_clock_skew_seconds: int = 5 + handshake_timeout_seconds: int = 30 + + def __post_init__(self) -> None: + pins = dict(self.pcrs) + if not {0, 1, 2}.issubset(pins) or len(pins) > 32: + raise ValueError("Pin at least Nitro PCR0, PCR1 and PCR2") + for index, digest in pins.items(): + if type(index) is not int or not 0 <= index < 32: + raise ValueError("Invalid Nitro PCR number") + _bytes(digest, 48) + if digest == bytes(48): + raise ValueError("Zero/debug Nitro measurements are not permitted") + for name, maximum, minimum in ( + ("max_age_seconds", 300, 1), + ("max_session_seconds", 300, 1), + ("max_clock_skew_seconds", 30, 0), + ("handshake_timeout_seconds", 60, 1), + ): + value = getattr(self, name) + if type(value) is not int or not minimum <= value <= maximum: + raise ValueError(f"Invalid Nitro duration: {name}") + object.__setattr__(self, "pcrs", MappingProxyType(pins)) + + def digest(self) -> bytes: + """Bind private exports to the exact locally selected trust policy.""" + return hashlib.sha256( + b"XTRACE-NITRO-POLICY-v1\0" + + _pack( + [ + sorted(self.pcrs.items()), + self.max_age_seconds, + self.max_session_seconds, + self.max_clock_skew_seconds, + self.handshake_timeout_seconds, + ] + ) + ).digest() + + +@dataclass(frozen=True) +class NitroIdentity: + """Verified receipt key and certificate expiry, obtained from signed evidence.""" + + public_key: bytes + certificate_expires_at: float + + +def _reject_tag(*args: Any) -> Any: + """Reject CBOR semantic conversions, including references and bignums.""" + raise BFVProtocolError("CBOR semantic tags are not permitted here") + + +def _cbor(data: bytes) -> Any: + """Decode one bounded, definite CBOR item without tags or duplicate keys.""" + import cbor2 + + if type(data) is not bytes or not 0 < len(data) <= MAX_NITRO_DOCUMENT_BYTES: + raise BFVProtocolError("Nitro CBOR exceeds its size limit") + stream = io.BytesIO(data) + try: + # The public semantic-decoder mapping intercepts every tag, including + # cbor2's built-ins. In particular, shared references cannot form cycles. + decoder = cbor2.CBORDecoder( + stream, + max_depth=6, + allow_indefinite=False, + allow_duplicate_keys=False, + semantic_decoders=defaultdict(lambda: _reject_tag), + tag_hook=_reject_tag, + ) + result = decoder.decode() + if stream.read(1): + raise BFVProtocolError("Trailing Nitro CBOR data") + return result + except (cbor2.CBORDecodeError, ValueError, TypeError, OverflowError) as exc: + raise BFVProtocolError("Invalid Nitro CBOR") from exc + + +def _verified_document(document: bytes, now: float) -> tuple[dict[str, Any], float]: + """Verify AWS's signature and complete certificate path at the client time. + + This private helper does not approve an application or a query. Its caller + must additionally enforce measurements, nonce, context and receipt-key pins. + The time argument exists for deterministic certificate regression fixtures; + the public entrypoint always obtains the local clock itself. + """ + from OpenSSL import crypto + from cryptography import x509 + from cryptography.exceptions import InvalidSignature, UnsupportedAlgorithm + from cryptography.hazmat.primitives import hashes, serialization + from cryptography.hazmat.primitives.asymmetric import ec, utils + import cbor2 + + if ( + type(document) is not bytes + or not 0 < len(document) <= MAX_NITRO_DOCUMENT_BYTES + or not math.isfinite(now) + or now < 0 + ): + raise BFVProtocolError("Invalid Nitro document or verification time") + try: + # AWS permits both an untagged COSE_Sign1 array and its tag-18 form. + cose = _cbor(document[1:] if document[:1] == b"\xd2" else document) + if type(cose) is not list or len(cose) != 4: + raise BFVProtocolError("Expected a Nitro COSE_Sign1 array") + protected, unprotected, payload, signature = cose + headers = _cbor(protected) + if ( + type(headers) is not dict + or len(headers) != 1 + or type(next(iter(headers))) is not int + or headers != {1: -35} + or type(unprotected) is not dict + or unprotected + ): + raise BFVProtocolError("Nitro requires the protected ES384 algorithm") + _bytes(signature, 96) + doc = _cbor(payload) + required = {"module_id", "timestamp", "digest", "pcrs", "certificate", "cabundle"} + optional = {"nonce", "public_key", "user_data"} + if ( + type(doc) is not dict + or not required.issubset(doc) + or set(doc) - required - optional + or type(doc["module_id"]) is not str + or not 1 <= len(doc["module_id"].encode()) <= 256 + or type(doc["timestamp"]) is not int + or not 0 <= doc["timestamp"] < 1 << 64 + or doc["digest"] != "SHA384" + ): + raise BFVProtocolError("Invalid Nitro claims") + pcrs = doc["pcrs"] + if type(pcrs) is not dict or not 1 <= len(pcrs) <= 32: + raise BFVProtocolError("Invalid Nitro measurements") + for index, digest in pcrs.items(): + if type(index) is not int or not 0 <= index < 32: + raise BFVProtocolError("Invalid Nitro PCR number") + _bytes(digest, 48) + for field in optional: + value = doc.get(field) + if value is not None and (type(value) is not bytes or len(value) > 1024): + raise BFVProtocolError("Invalid optional Nitro field") + bundle = doc["cabundle"] + if type(bundle) is not list or not 1 <= len(bundle) <= 8: + raise BFVProtocolError("Invalid Nitro certificate bundle") + cert_bytes = [*bundle, doc["certificate"]] + if any(type(v) is not bytes or not 1 <= len(v) <= 1024 for v in cert_bytes): + raise BFVProtocolError("Invalid Nitro certificate size") + if len(set(cert_bytes)) != len(cert_bytes): + raise BFVProtocolError("Duplicate Nitro certificates") + if not hmac.compare_digest(hashlib.sha256(bundle[0]).digest(), _AWS_ROOT_SHA256): + raise BFVProtocolError("Untrusted Nitro root") + certs = [x509.load_der_x509_certificate(v) for v in cert_bytes] + if any( + cert.public_bytes(serialization.Encoding.DER) != encoded + for cert, encoded in zip(certs, cert_bytes, strict=True) + ): + raise BFVProtocolError("Noncanonical Nitro certificate") + # Nitro's real leaf omits AKI/SKI. X509_STRICT would reject that valid + # AWS profile. Normal OpenSSL path validation still enforces signatures, + # validity, path lengths and critical constraints; explicitly require + # CA authorization and leaf signing usage for this application as well. + for index, cert in enumerate(certs): + basic = cert.extensions.get_extension_for_class(x509.BasicConstraints).value + usage = cert.extensions.get_extension_for_class(x509.KeyUsage).value + is_ca = index < len(certs) - 1 + if ( + basic.ca != is_ca + or (is_ca and not usage.key_cert_sign) + or (not is_ca and (not usage.digital_signature or usage.key_cert_sign)) + ): + raise BFVProtocolError("Invalid Nitro certificate key usage") + store = crypto.X509Store() + store.add_cert(crypto.X509.from_cryptography(certs[0])) + store.set_flags(crypto.X509StoreFlags.CHECK_SS_SIGNATURE) + store.set_time(datetime.fromtimestamp(now, UTC)) + # Delegate path constraints, CA/key usage, signatures and validity to + # OpenSSL. Checking only a sequence of certificate signatures is unsafe. + chain = crypto.X509StoreContext( + store, + crypto.X509.from_cryptography(certs[-1]), + [crypto.X509.from_cryptography(c) for c in certs[1:-1]], + ).get_verified_chain() + verified = [c.to_cryptography() for c in chain] + if len(verified) != len(certs): + raise BFVProtocolError("Unused Nitro certificates") + public_key = certs[-1].public_key() + if not isinstance(public_key, ec.EllipticCurvePublicKey) or not isinstance( + public_key.curve, ec.SECP384R1 + ): + raise BFVProtocolError("Nitro signing key must use P-384") + signed = cbor2.dumps(["Signature1", protected, b"", payload]) + der_signature = utils.encode_dss_signature( + int.from_bytes(signature[:48], "big"), int.from_bytes(signature[48:], "big") + ) + public_key.verify(der_signature, signed, ec.ECDSA(hashes.SHA384())) + expires = min(c.not_valid_after_utc.timestamp() for c in verified) + return doc, expires + except ( + ValueError, + TypeError, + OverflowError, + InvalidSignature, + UnsupportedAlgorithm, + crypto.Error, + crypto.X509StoreContextError, + x509.ExtensionNotFound, + x509.DuplicateExtension, + x509.InvalidVersion, + ) as exc: + raise BFVProtocolError("Nitro attestation rejected") from exc + + +def verify_nitro_attestation( + document: bytes, policy: NitroAttestationPolicy, *, nonce: bytes, context: bytes +) -> NitroIdentity: + """Approve fresh AWS evidence for exact owner pins, nonce and BFV context. + + Inputs are raw COSE bytes, a local policy, and locally retained 32-byte + nonce/context digests. Returns a verified Ed25519 key and certificate expiry. + There is no network fetching, trust-on-first-use or caller-selected root. + """ + now = time.time() + doc, expires = _verified_document(document, now) + if ( + not now - policy.max_age_seconds + <= doc["timestamp"] / 1000 + <= (now + policy.max_clock_skew_seconds) + ): + raise BFVProtocolError("Nitro attestation is not fresh") + for index, expected in policy.pcrs.items(): + actual = doc["pcrs"].get(index) + if type(actual) is not bytes or not hmac.compare_digest(actual, expected): + raise BFVProtocolError("Nitro code measurement mismatch") + if doc.get("nonce") != _bytes(nonce, 32) or doc.get("user_data") != _bytes(context, 32): + raise BFVProtocolError("Nitro attestation context mismatch") + spki = doc.get("public_key") + if type(spki) is not bytes or len(spki) != 44 or not spki.startswith(_ED25519_SPKI_PREFIX): + raise BFVProtocolError("Expected a Nitro-bound Ed25519 receipt key") + key = eddsa.import_public_key(spki[12:]) + if (key.pointQ * 8).is_point_at_infinity(): + raise BFVProtocolError("Invalid Nitro receipt key") + return NitroIdentity(spki[12:], expires) + + +class NitroNSM: + """Use AWS's libnsm.so inside an enclave; fail closed if it is unavailable. + + The path is trusted local configuration, never a network field. The official + NSM C ABI supplies random bytes and evidence; no ioctl protocol is recreated + here. Keep the library and application in the measured enclave image. + """ + + def __init__(self, library: str = "/usr/local/lib/libnsm.so") -> None: + if not Path(library).is_absolute(): + raise ValueError("Use an absolute, trusted NSM library path") + self._lock = threading.Lock() + self._lib = ctypes.CDLL(library) + self._lib.nsm_lib_init.argtypes = [] + self._lib.nsm_lib_init.restype = ctypes.c_int + self._lib.nsm_lib_exit.argtypes = [ctypes.c_int] + self._lib.nsm_lib_exit.restype = None + self._lib.nsm_get_attestation_doc.argtypes = [ + ctypes.c_int, + ctypes.c_void_p, + ctypes.c_uint32, + ctypes.c_void_p, + ctypes.c_uint32, + ctypes.c_void_p, + ctypes.c_uint32, + ctypes.c_void_p, + ctypes.POINTER(ctypes.c_uint32), + ] + self._lib.nsm_get_attestation_doc.restype = ctypes.c_int + self._lib.nsm_get_random.argtypes = [ + ctypes.c_int, + ctypes.c_void_p, + ctypes.POINTER(ctypes.c_size_t), + ] + self._lib.nsm_get_random.restype = ctypes.c_int + self._fd = self._lib.nsm_lib_init() + if self._fd < 0: + raise BFVProtocolError("Nitro NSM device unavailable") + + def random_seed(self) -> bytes: + """Get an independent 32-byte receipt-signing seed from NSM randomness.""" + with self._lock: + buffer = ctypes.create_string_buffer(32) + length = ctypes.c_size_t(32) + if ( + self._fd < 0 + or self._lib.nsm_get_random(self._fd, buffer, ctypes.byref(length)) != 0 + or length.value != 32 + ): + raise BFVProtocolError("Nitro NSM randomness unavailable") + return buffer.raw + + def attest(self, public_key: bytes, nonce: bytes, context: bytes) -> bytes: + """Ask NSM to bind the in-enclave receipt key and the owner's challenge.""" + _bytes(public_key, 44) + _bytes(nonce, 32) + _bytes(context, 32) + with self._lock: + output = ctypes.create_string_buffer(MAX_NITRO_DOCUMENT_BYTES) + length = ctypes.c_uint32(MAX_NITRO_DOCUMENT_BYTES) + status = ( + self._lib.nsm_get_attestation_doc( + self._fd, + context, + len(context), + nonce, + len(nonce), + public_key, + len(public_key), + output, + ctypes.byref(length), + ) + if self._fd >= 0 + else -1 + ) + if status != 0 or not 0 < length.value <= MAX_NITRO_DOCUMENT_BYTES: + raise BFVProtocolError("Nitro NSM attestation unavailable") + return output.raw[: length.value] + + def close(self) -> None: + """Close the device once, excluding concurrent NSM calls.""" + with self._lock: + if self._fd >= 0: + self._lib.nsm_lib_exit(self._fd) + self._fd = -1 diff --git a/tests/x_vec/fixtures/nitro/LICENSE b/tests/x_vec/fixtures/nitro/LICENSE new file mode 100644 index 0000000..1c98f1a --- /dev/null +++ b/tests/x_vec/fixtures/nitro/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025 Joern Barthel + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/tests/x_vec/fixtures/nitro/README.md b/tests/x_vec/fixtures/nitro/README.md new file mode 100644 index 0000000..5cb950c --- /dev/null +++ b/tests/x_vec/fixtures/nitro/README.md @@ -0,0 +1,12 @@ +# Historical AWS-signed attestation fixture + +`aws-2025-01-06.cose` is the 4,781-byte public fixture from the published +`nitro_attest` 0.2.0 crate, `tests/fixtures/attestation.cose`. +Source: https://crates.io/crates/nitro_attest/0.2.0 +Repository: https://github.com/aws-nitro-enclaves/nitro-attest + +It is used only to test the AWS certificate profile and COSE signature at the +historical Unix time 1736179625. Its certificates are expired today, and it lacks +our required owner nonce/context. It is not live attestation of the BFV service. +No private keys are included. The fixture is redistributed under the upstream +MIT license (Joern Barthel, 2025); see the adjacent `LICENSE` file. diff --git a/tests/x_vec/fixtures/nitro/aws-2025-01-06.cose b/tests/x_vec/fixtures/nitro/aws-2025-01-06.cose new file mode 100644 index 0000000000000000000000000000000000000000..e936db6c1a75bbce667d316f5587244c37296bbf GIT binary patch literal 4781 zcmd6q2~bn#7RQsD5DdE@vM8t^o0{ayO$Z4EWK%#;w1_OqlH~>hAwUv>iYP%qtyHBd zf}nsD`T!9@P;sdnwyw3}@=!#;1r!t=k%)rOi>Sqd(_yCXO*3<6a?afI<>a2<|D1DE zTnn&Fll%Z}rxHO_G%rqwga~*mEd(UMK@b)UXmmb}#{di#m4t}6fXbrNxYh`p&7rY* z3^t!92ty=NzK^>TjY$_s1yQKQj*6Lr!C-gT1s}lhVz^?-L5x4xTp92RI==Jv%{yB( zv{?Exg&h)aBfqQW4&JvyX0Ra^KyPQUtFsX?|pl15qY7pv5ieonQBpF{;q{M$RE6Z5~*Eo-}82$gfkc; zH0?4_op0k&|1NMz=*j)vD#^Qa7ab_Y`-9+X-j!ZjYF9Yfdx>7z z^4zD7jsAPlLk3;Hs@R-$dEW|MIsqS*w=MJS?*TKWt~pVtJVoQZZXdqLSJ160%4j6-N*C3n5#ENPw}PI&2N7fI(#d8vqA>Hd&}?9SG!^|BIZR zz}(@SnPcQURx=y*KY$Eq5tKhXS_n%}nk~m;OmG|wUMV4=a07E|`0Lv*Qj?-4l?0p) z63?tkyYVnqWLD|J?QFZ7kt@aw49-2}ym3M6X*bHQm|Hg+H+UX3v}|84b5oq{-xHt3 zpiydn@z0UGtPiJ{xPl<J^kBBbRh|Xcj+-dt9=gZnQiI)(zJ7#H5%9?-t zxPyrYM8&*n5(jSF`Xbh)3%A&5#=I$tNBf(f_F^mMWKvm_fle29qHj9D@2E+&n7+9! z^^ILPm(7V2@q|c90Hll3I1wacGlz+shCYf0q0Zn(ILlL8i=pZmJd88`yX&$U+qI1~Y3H0030PsUH$W>%wq zN6D={^IYaT@t$khCC#3hAJlB5Im8MI2sUlx)5i%gfRC2Da2+M1;Nr>P_M@CNle6l5 zaa*S0T;49{mT=PTSxGzR6&Rm5MgGRO;Xqex^|{;S-Dt?Z?wrX=RWX)DO`iFJ8mtJZ z?z_LWgS-wkU=6?aSVh zmIpPsvyVq+_D>e;Q-^^8OoeH*j~xa^*Ow0_2pboIXj>Xa5YV|W!sXI=2m&*i=;|}7 zMf|nZeTAB?60L6beKx-^(v zr%u|0)r#i!Oq#A^uC**@XZH7TmiwqRmMUas0Wy zQEUl^&YOlu02i?HL-K>tmWNBFF}4&6Dp%5%ji1C73&|21nf+m;49S@r9YsN%6gG#> zVgN3eWW!_7NOYLPBe4(`pJdIV(U@Eg!-mhKlhGST@`4M)03HPF7zXbB7^a5 z&x?9n`i>qZF4U}>J(zK*xs$fU&1v;F44nYz0xFR@^zs~*i7w`{dY4&?tb*r|SRkER zckFT0Qiql=wDK=Q^s}O<*48i;d|FIxKAQi^W%?@?xB%KH!ZZD$JLnfbb($6*y9Y8Jx`)pj2oZ(81D~45W z`w!l)S-5y&oyOTG~?aU7w58xo1E3c#A9i>(RzIz&Vl|Fk#iSS(fvgG-eC0eg^iij#|$C&W_w&IXK-l@eZ|R?jD^h!>XY! z>a&OI-Aba*hqTYO+g9!ypZ99cs?)YjFBdi3^E=%=Nv2%w0DvPNVE2W9A7E8{|yZL?2`ljc_#qp}AdiGx*fZI|pwCJ1hji+8P zM=qL2>t(*_vgp%^3Yx4(yYIxvLtp%~1?JgoxM$6`o%*8qCVz%N z9x=5t9)Db%Y2amBvev73nrRTWJP3L%&$0B0-pSnMc +#include +#include + +static int mode = 0; +static int closes = 0; + +/* Test controls affect only this deliberately synthetic shared library. */ +void test_mode(int value) { mode = value; } +int test_closes(void) { return closes; } + +/* Model a device descriptor and its idempotent caller-side close handling. */ +int nsm_lib_init(void) { return mode == 4 ? -1 : 7; } +void nsm_lib_exit(int fd) { if (fd == 7) ++closes; } + +/* Verify that size_t, capacity and destination pointers match the official ABI. */ +int nsm_get_random(int fd, uint8_t *out, size_t *length) { + if (fd != 7 || *length != 32 || mode == 1) return 1; + for (unsigned i = 0; i < 32; ++i) out[i] = (uint8_t)i; + if (mode == 2) *length = 31; + return 0; +} + +/* Echo inputs to catch swapped nonce/context/key pointers and uint32_t lengths. */ +int nsm_get_attestation_doc(int fd, const uint8_t *user, uint32_t user_len, + const uint8_t *nonce, uint32_t nonce_len, const uint8_t *key, uint32_t key_len, + uint8_t *out, uint32_t *length) { + if (fd != 7 || user_len != 32 || nonce_len != 32 || key_len != 44 || + *length != 16384 || mode == 1) return 1; + if (mode == 3) { *length = 16385; return 0; } + memcpy(out, user, 32); + memcpy(out + 32, nonce, 32); + memcpy(out + 64, key, 44); + *length = 108; + return 0; +} diff --git a/tests/x_vec/native/run_binding_sanitizers.py b/tests/x_vec/native/run_binding_sanitizers.py index 9a30bd4..fe71e13 100644 --- a/tests/x_vec/native/run_binding_sanitizers.py +++ b/tests/x_vec/native/run_binding_sanitizers.py @@ -5,9 +5,16 @@ """ import importlib.util +import os from pathlib import Path import sys +# Installing the Nitro extra also installs CFFI. PyCryptodome then uses a loader +# with RTLD_DEEPBIND, which bypasses ASan's allocator interposition. Its supported +# override keeps sanitizers active; it applies only to this test process and must +# be set before importing the SDK (which imports PyCryptodome). +os.environ["PYCRYPTODOME_DISABLE_DEEPBIND"] = "1" + import xtrace_sdk.x_vec.crypto.bfv_cpu_ext as package assert len(sys.argv) == 3, "Supply both sanitized extension paths; no release fallback" @@ -33,6 +40,7 @@ "tests/x_vec/test_bfv_verified_client.py", "tests/x_vec/test_bfv_private.py", "tests/x_vec/test_bfv_guarded_client.py", + "tests/x_vec/test_bfv_attested_client.py", "tests/x_vec/test_bfv_assurance.py", "-q", ] diff --git a/tests/x_vec/nitro_fixtures.py b/tests/x_vec/nitro_fixtures.py new file mode 100644 index 0000000..f887e46 --- /dev/null +++ b/tests/x_vec/nitro_fixtures.py @@ -0,0 +1,179 @@ +"""Synthetic Nitro-format evidence for OFFLINE TESTS ONLY, never an AWS identity. + +Tests explicitly replace the verifier's private root fingerprint with this +fixture's root. The production SDK has no synthetic-root configuration switch. +All private keys here are newly generated test keys and are never persisted. +""" + +import hashlib +import secrets +import time +from datetime import UTC, datetime, timedelta + +import cbor2 +from Crypto.Util.asn1 import DerObject, DerSequence +from cryptography import x509 +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.hazmat.primitives.asymmetric import ec, utils +from cryptography.x509.oid import NameOID, ObjectIdentifier + + +PCRS = {i: hashlib.sha384(f"XTrace synthetic PCR{i}".encode()).digest() for i in (0, 1, 2)} + + +def malformed_certificate(encoded, fault): + """Mutate real DER to exercise parser errors before certificate verification. + + The modified leaf signature is invalid too. Verification must reject these + bytes through the public protocol error, without leaking a parser exception. + """ + certificate = DerSequence().decode(encoded) + body = DerSequence().decode(certificate[0]) + if fault == "duplicate_extension": + wrapper = DerObject().decode(body[-1]) + extensions = DerSequence().decode(wrapper.payload) + extensions.append(extensions[0]) + wrapper.payload = extensions.encode() + body[-1] = wrapper.encode() + elif fault == "invalid_version": + body[0] = b"\xa0\x03\x02\x01\x03" # Unsupported X.509 version 4. + else: + raise ValueError("Unknown test certificate fault") + certificate[0] = body.encode() + return certificate.encode() + + +class SyntheticNitro: + """Issue cryptographically signed test certificates and COSE evidence. + + ca_authorized, expired, path_length and unknown_critical create certificate + path failures whose signatures are otherwise valid. changes mutates signed + claims, enabling tests of policy checks independently of signature validity. + """ + + def __init__( + self, + *, + ca_authorized=True, + expired=False, + path_length=1, + unknown_critical=False, + changes=None, + ): + self.changes = changes or {} + self.root_key, self.ca_key, self.leaf_key = [ + ec.generate_private_key(ec.SECP384R1()) for _ in range(3) + ] + now = datetime.now(UTC) + + def issue(name, key, issuer, signer, ca, length, expiry): + """Create an explicit X.509 path with AKI, SKI and key-usage constraints.""" + subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, name)]) + builder = ( + x509.CertificateBuilder() + .subject_name(subject) + .issuer_name(issuer.subject if issuer else subject) + .public_key(key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(now - timedelta(days=2)) + .not_valid_after(expiry) + .add_extension(x509.BasicConstraints(ca=ca, path_length=length), critical=True) + .add_extension( + x509.KeyUsage( + digital_signature=not ca, + content_commitment=False, + key_encipherment=False, + data_encipherment=False, + key_agreement=False, + key_cert_sign=ca, + crl_sign=ca, + encipher_only=None, + decipher_only=None, + ), + critical=True, + ) + .add_extension( + x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False + ) + .add_extension( + x509.AuthorityKeyIdentifier.from_issuer_public_key(signer.public_key()), + critical=False, + ) + ) + if unknown_critical and name.endswith("leaf"): + builder = builder.add_extension( + x509.UnrecognizedExtension(ObjectIdentifier("1.2.3.4.99"), b"\x05\x00"), + critical=True, + ) + return builder.sign(signer, hashes.SHA384()) + + self.root = issue( + "TEST ONLY root", + self.root_key, + None, + self.root_key, + True, + path_length, + now + timedelta(days=1), + ) + self.ca = issue( + "TEST ONLY intermediate", + self.ca_key, + self.root, + self.root_key, + ca_authorized, + 0 if ca_authorized else None, + now + timedelta(days=1), + ) + self.leaf = issue( + "TEST ONLY leaf", + self.leaf_key, + self.ca, + self.ca_key, + False, + None, + now + timedelta(hours=-1 if expired else 1), + ) + self.root_digest = hashlib.sha256(self.der(self.root)).digest() + + @staticmethod + def der(cert): + """Return a test certificate's canonical DER bytes.""" + return cert.public_bytes(serialization.Encoding.DER) + + def random_seed(self): + """Generate an ordinary local test signing seed; this is not NSM entropy.""" + return secrets.token_bytes(32) + + def claims(self, public_key, nonce, context): + """Build realistic signed claims, including a complete untrusted CA bundle.""" + return { + "module_id": "TEST-ONLY-NOT-AWS", + "timestamp": int(time.time() * 1000), + "digest": "SHA384", + "pcrs": dict(PCRS), + "certificate": self.der(self.leaf), + "cabundle": [self.der(self.root), self.der(self.ca)], + "public_key": public_key, + "nonce": nonce, + "user_data": context, + **self.changes, + } + + def sign(self, payload, *, protected=None, unprotected=None, tagged=True): + """Sign exact payload bytes using COSE ES384's fixed-width signature format.""" + protected = cbor2.dumps({1: -35}) if protected is None else protected + structure = cbor2.dumps(["Signature1", protected, b"", payload]) + signature = self.leaf_key.sign(structure, ec.ECDSA(hashes.SHA384())) + r, s = utils.decode_dss_signature(signature) + envelope = [ + protected, + {} if unprotected is None else unprotected, + payload, + r.to_bytes(48, "big") + s.to_bytes(48, "big"), + ] + return cbor2.dumps(cbor2.CBORTag(18, envelope) if tagged else envelope) + + def attest(self, public_key, nonce, context): + """Return fake-root evidence which the unmodified AWS verifier must reject.""" + return self.sign(cbor2.dumps(self.claims(public_key, nonce, context))) diff --git a/tests/x_vec/test_bfv_attested_client.py b/tests/x_vec/test_bfv_attested_client.py new file mode 100644 index 0000000..69e47e6 --- /dev/null +++ b/tests/x_vec/test_bfv_attested_client.py @@ -0,0 +1,312 @@ +"""Nitro-gated protocol regressions with real crypto and synthetic test evidence. + +Instrument both private decoders in attack tests: malformed provenance must be +rejected before BFV parsing or secret work. Synthetic roots are test-local only. +""" + +import hashlib +import secrets +from concurrent.futures import ThreadPoolExecutor +from dataclasses import replace + +import pytest + +pytest.importorskip("cbor2") +pytest.importorskip("OpenSSL") +from Crypto.Signature import eddsa +from gmpy2 import mpz + +from tests.x_vec.nitro_fixtures import PCRS, SyntheticNitro +from tests.x_vec.test_bfv_guarded_client import deny_private, rebind_response +from xtrace_sdk.x_vec.crypto import bfv_attested_client as attested +from xtrace_sdk.x_vec.crypto import bfv_nitro as nitro +from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient +from xtrace_sdk.x_vec.crypto.bfv_security import ( + BFVExecutionPolicy, + BFVProtocolError, + _unpack, + _pack, +) +from xtrace_sdk.x_vec.crypto.bfv_verified_client import BFVVerifiedClient, BFVVerifiedServer +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, _coefficient_modulus +from xtrace_sdk.x_vec.utils.xtrace_types import BFVCiphertext + + +@pytest.fixture(scope="module") +def private(): + return BFVClient(3, 16, 97, 120, 15, response_modulus_bits=40, rns_modulus=True) + + +@pytest.fixture(scope="module") +def issuer(): + return SyntheticNitro() + + +@pytest.fixture +def trusted(issuer, monkeypatch): + monkeypatch.setattr(nitro, "_AWS_ROOT_SHA256", issuer.root_digest) + return issuer + + +def make_session(private, issuer, count=3, backend="native", enroll=True): + """Create independent client/server contexts with one authorized owner.""" + if backend == "native": + pytest.importorskip("xtrace_sdk.x_vec.crypto.bfv_cpu_ext._bfv_private") + policy = BFVExecutionPolicy(params=private.params, embed_len=3, response_modulus_bits=40) + auth = secrets.token_bytes(32) + client = attested.BFVAttestedClient( + private, + auth, + nitro.NitroAttestationPolicy(PCRS), + index_epoch=7, + policy=policy, + private_backend=backend, + ) + vectors = ([[0, 1, 0], [1, 0, 1], [0, 0, 0]] * ((count + 2) // 3))[:count] + setup = client.prepare_index(vectors) + server = attested.BFVAttestedServer.from_registration( + client.registration_packet(setup), issuer, policy=policy, backend="optimized" + ) + if enroll: + client.accept_attestation(*server.attest(client.begin_attestation())) + return client, server, setup, auth, vectors + + +@pytest.mark.parametrize("backend", ["python", "native"]) +@pytest.mark.parametrize("count", [1, 5, 39]) +def test_one_public_evaluation_identical_to_baseline(private, trusted, monkeypatch, backend, count): + client, server, setup, auth, vectors = make_session(private, trusted, count, backend) + query = client.begin_query([0, 1, 0]) + baseline = BFVVerifiedServer(setup, auth, policy=client.policy, backend="optimized") + expected = baseline.search(query[38:-64]) + calls = [] + original = server._server.search + + def record(inner): + calls.append(inner) + return original(inner) + + monkeypatch.setattr(server._server, "search", record) + with monkeypatch.context() as scope: + deny_private(scope) + response, receipt = server.search(query) + assert calls == [query[38:-64]] + assert response == expected + assert server._server._client.keys is None + assert len(receipt) == attested.NITRO_RECEIPT_BYTES == 198 + assert client.finish_query(response, receipt) == [ + sum(a != b for a, b in zip(v, [0, 1, 0], strict=True)) for v in vectors + ] + with pytest.raises(BFVProtocolError): + client.finish_query(response, receipt) + with pytest.raises(BFVProtocolError): + server.search(query) + + +def test_default_client_rejects_fake_aws_identity(private, issuer): + client, server, *_ = make_session(private, issuer, enroll=False) + with pytest.raises(BFVProtocolError, match="enrollment rejected"): + client.accept_attestation(*server.attest(client.begin_attestation())) + with pytest.raises(BFVProtocolError, match="attestation is required"): + client.begin_query([0, 1, 0]) + + +@pytest.mark.parametrize( + "change", ["signature", "missing", "context", "session", "query", "response", "extra"] +) +def test_receipt_failures_never_parse_or_decrypt(private, trusted, monkeypatch, change): + client, server, *_ = make_session(private, trusted) + response, receipt = server.search(client.begin_query([0, 1, 0])) + positions = {"signature": 197, "context": 6, "session": 38, "query": 70, "response": 102} + bad = b"" if change == "missing" else receipt + b"x" if change == "extra" else receipt + if change in positions: + p = positions[change] + bad = receipt[:p] + bytes([receipt[p] ^ 1]) + receipt[p + 1 :] + with monkeypatch.context() as scope: + deny_private(scope) + import xtrace_sdk.x_vec.crypto.bfv_security as framing + + scope.setattr( + framing.msgpack, "unpackb", lambda *a, **kw: pytest.fail("Unapproved BFV parsing") + ) + with pytest.raises(BFVProtocolError, match="before decryption"): + client.finish_query(response, bad) + assert client.finish_query(response, receipt) == [0, 3, 1] + + +def test_previous_rounding_oracle_blocked_before_private_work(private, trusted, monkeypatch): + client, server, _, auth, _ = make_session(private, trusted) + n, t = private.params.poly_modulus_degree, private.params.plain_modulus + q = _coefficient_modulus(private.response_modulus_bits) + deny_private(monkeypatch) + for index in range(n): + for offset in (0, 1): + request = client.begin_query([0, 1, 0]) + _, receipt = server.search(request) + c0 = (q // (2 * t) + offset,) + (mpz(0),) * (n - 1) + c1 = [mpz(0)] * n + c1[0 if index == 0 else n - index] = mpz(1) + ct = BFVCiphertext((c0, tuple(c1)), q, private._pk()["key_id"]) + forged = rebind_response( + request[38:-64], + [BFV.ciphertext_to_ints(ct, private._pk())], + private, + auth, + client.policy, + ) + with pytest.raises(BFVProtocolError, match="before decryption"): + client.finish_query(forged, receipt) + client.cancel_pending_queries() + + +def test_parent_cannot_authorize_queries_with_transport_mac_key(private, trusted, monkeypatch): + client, server, _, auth, _ = make_session(private, trusted) + query = client.begin_query([0, 1, 0]) + fake_owner = eddsa.new( + eddsa.import_private_key(auth), "rfc8032", context=attested._OWNER_CONTEXT + ) + forged = query[:-64] + fake_owner.sign( + attested._QUERY + client._context() + query[6:38] + hashlib.sha256(query[38:-64]).digest() + ) + monkeypatch.setattr( + server._server, "search", lambda *a: pytest.fail("Unauthorized BFV evaluation") + ) + with pytest.raises(ValueError): + server.search(forged) + + +def test_even_valid_receipt_must_match_a_pending_owner_query(private, trusted, monkeypatch): + client, server, *_ = make_session(private, trusted) + response, receipt = server.search(client.begin_query([0, 1, 0])) + # Model a separately signed transcript for an unrelated request. It must + # not authorize private work even though its signature is cryptographically valid. + message = receipt[:70] + bytes(32) + receipt[102:134] + forged = message + server._signer.sign(message) + deny_private(monkeypatch) + with pytest.raises(BFVProtocolError, match="before decryption"): + client.finish_query(response, forged) + + +def test_registration_signature_precedes_bfv_import(private, trusted, monkeypatch): + client, _, setup, _, _ = make_session(private, trusted) + registration = _unpack( + client.registration_packet(setup), limit=client.policy.max_setup_bytes + 1024 + ) + registration[3] += 1 + monkeypatch.setattr( + BFV, "deserialize_public_key", lambda *a, **kw: pytest.fail("Unauthorized BFV key import") + ) + with pytest.raises(ValueError): + attested.BFVAttestedServer.from_registration( + _pack(registration), trusted, policy=client.policy + ) + + +@pytest.mark.parametrize("change", ["proof", "nonce", "pins", "epoch", "execution_policy", "owner"]) +def test_enrollment_binding_and_possession(private, trusted, change): + client, server, *_ = make_session(private, trusted, enroll=False) + hello = client.begin_attestation() + document, proof = server.attest(hello) + if change == "proof": + proof = bytes(64) + elif change == "nonce": + client.begin_attestation() + elif change == "pins": + client.attestation_policy = nitro.NitroAttestationPolicy({**PCRS, 2: bytes([1]) * 48}) + elif change == "epoch": + client.index_epoch += 1 + elif change == "execution_policy": + client._session.policy = replace(client.policy, max_queries=128) + else: + client._owner_public = bytes(32) + with pytest.raises(BFVProtocolError): + client.accept_attestation(document, proof) + with pytest.raises(BFVProtocolError): + client.begin_query([0, 1, 0]) + + +def test_expiry_replay_renewal_and_restart(private, trusted, monkeypatch): + client, server, setup, *_ = make_session(private, trusted, enroll=False) + hello = client.begin_attestation() + document, proof = server.attest(hello) + with pytest.raises(BFVProtocolError): + server.attest(hello) + client.accept_attestation(document, proof) + with pytest.raises(BFVProtocolError): + client.accept_attestation(document, proof) + query = client.begin_query([0, 1, 0]) + response, receipt = server.search(query) + client._deadline = 0 + with monkeypatch.context() as scope: + deny_private(scope) + with pytest.raises(BFVProtocolError, match="before decryption"): + client.finish_query(response, receipt) + client.accept_attestation(*server.attest(client.begin_attestation())) + with pytest.raises(BFVProtocolError): + client.finish_query(response, receipt) + with pytest.raises(BFVProtocolError): + server.search(query) + server = attested.BFVAttestedServer.from_registration( + client.registration_packet(setup), trusted, policy=client.policy, backend="optimized" + ) + with pytest.raises(BFVProtocolError): + server.search(client.begin_query([0, 1, 0])) + client.accept_attestation(*server.attest(client.begin_attestation())) + assert client.finish_query(*server.search(client.begin_query([0, 1, 0]))) == [0, 3, 1] + + +def test_handshake_and_search_deadlines_are_checked(private, trusted, monkeypatch): + client, server, *_ = make_session(private, trusted, enroll=False) + document, proof = server.attest(client.begin_attestation()) + client._hello_started -= 100 + with pytest.raises(BFVProtocolError): + client.accept_attestation(document, proof) + client.accept_attestation(*server.attest(client.begin_attestation())) + query = client.begin_query([0, 1, 0]) + original = server._server.search + + def expire_during_search(inner): + result = original(inner) + server._deadline = 0 + return result + + monkeypatch.setattr(server._server, "search", expire_during_search) + with pytest.raises(BFVProtocolError, match="expired during"): + server.search(query) + + +def test_concurrent_finish_decrypts_once(private, trusted): + client, server, *_ = make_session(private, trusted) + response, receipt = server.search(client.begin_query([0, 1, 0])) + + def finish(): + try: + return client.finish_query(response, receipt) + except BFVProtocolError: + return None + + with ThreadPoolExecutor(max_workers=4) as pool: + assert list(pool.map(lambda _: finish(), range(4))).count([0, 3, 1]) == 1 + + +def test_restore_requires_fresh_attestation_and_exact_epoch_policy(private, trusted): + client, server, setup, auth, _ = make_session(private, trusted) + wrap = secrets.token_bytes(32) + saved = client.protect_state(wrap) + with pytest.raises(BFVProtocolError): + BFVVerifiedClient.restore(setup, saved, auth, wrap, policy=client.policy) + for epoch, pins in ( + (8, client.attestation_policy), + (7, nitro.NitroAttestationPolicy({**PCRS, 0: bytes([9]) * 48})), + ): + with pytest.raises(BFVProtocolError): + attested.BFVAttestedClient.restore( + setup, saved, auth, wrap, pins, index_epoch=epoch, policy=client.policy + ) + restored = attested.BFVAttestedClient.restore( + setup, saved, auth, wrap, client.attestation_policy, index_epoch=7, policy=client.policy + ) + with pytest.raises(BFVProtocolError, match="attestation is required"): + restored.begin_query([0, 1, 0]) + restored.accept_attestation(*server.attest(restored.begin_attestation())) + assert restored.finish_query(*server.search(restored.begin_query([0, 1, 0]))) == [0, 3, 1] diff --git a/tests/x_vec/test_bfv_nitro.py b/tests/x_vec/test_bfv_nitro.py new file mode 100644 index 0000000..49712a1 --- /dev/null +++ b/tests/x_vec/test_bfv_nitro.py @@ -0,0 +1,275 @@ +"""Exercise real certificate/signature validation with isolated synthetic roots. + +These tests check public verification, not hardware isolation. Positive fixtures +patch the private trust anchor locally; a separate regression verifies that the +unmodified AWS-root path rejects exactly the same synthetic evidence. +""" + +import hashlib +import time +import shutil +import subprocess +from dataclasses import replace +from pathlib import Path + +import pytest + +pytest.importorskip("cbor2") +pytest.importorskip("OpenSSL") +import cbor2 +from Crypto.Signature import eddsa + +from tests.x_vec.nitro_fixtures import PCRS, SyntheticNitro, malformed_certificate +from xtrace_sdk.x_vec.crypto import bfv_nitro as nitro +from xtrace_sdk.x_vec.crypto.bfv_security import BFVProtocolError + + +@pytest.fixture(scope="module") +def issuer(): + return SyntheticNitro() + + +@pytest.fixture +def trusted(issuer, monkeypatch): + monkeypatch.setattr(nitro, "_AWS_ROOT_SHA256", issuer.root_digest) + return issuer + + +def evidence(issuer): + """Use a real Ed25519 SPKI and fixed public test bindings.""" + key = eddsa.import_private_key(bytes(range(32))).public_key() + nonce, context = bytes([3]) * 32, bytes([4]) * 32 + return issuer.attest(key.export_key(format="DER"), nonce, context), key, nonce, context + + +@pytest.mark.parametrize("tagged", [False, True]) +def test_valid_chain_cose_and_application_bindings(trusted, tagged): + doc, key, nonce, context = evidence(trusted) + if not tagged: + doc = doc[1:] + result = nitro.verify_nitro_attestation( + doc, nitro.NitroAttestationPolicy(PCRS), nonce=nonce, context=context + ) + assert result.public_key == key.export_key(format="raw") + assert result.certificate_expires_at > time.time() + + +def test_default_aws_root_rejects_synthetic_evidence(issuer): + doc, _, nonce, context = evidence(issuer) + assert issuer.root_digest != nitro._AWS_ROOT_SHA256 + with pytest.raises(BFVProtocolError): + nitro.verify_nitro_attestation( + doc, nitro.NitroAttestationPolicy(PCRS), nonce=nonce, context=context + ) + + +@pytest.mark.parametrize( + "kwargs", + [ + {"ca_authorized": False}, + {"expired": True}, + {"path_length": 0}, + {"unknown_critical": True}, + ], +) +def test_certificate_constraints_are_enforced(kwargs, monkeypatch): + issuer = SyntheticNitro(**kwargs) + monkeypatch.setattr(nitro, "_AWS_ROOT_SHA256", issuer.root_digest) + doc, _, nonce, context = evidence(issuer) + with pytest.raises(BFVProtocolError): + nitro.verify_nitro_attestation( + doc, nitro.NitroAttestationPolicy(PCRS), nonce=nonce, context=context + ) + + +@pytest.mark.parametrize("fault", ["duplicate_extension", "invalid_version"]) +def test_malformed_certificate_uses_public_protocol_error(trusted, fault): + _, key, nonce, context = evidence(trusted) + claims = trusted.claims(key.export_key(format="DER"), nonce, context) + claims["certificate"] = malformed_certificate(claims["certificate"], fault) + document = trusted.sign(cbor2.dumps(claims)) + with pytest.raises(BFVProtocolError): + nitro.verify_nitro_attestation( + document, nitro.NitroAttestationPolicy(PCRS), nonce=nonce, context=context + ) + + +@pytest.mark.parametrize( + "change", + [ + "nonce", + "context", + "key", + "small_order_key", + "pcr", + "missing_pcr", + "debug", + "old", + "future", + "digest", + "timestamp_type", + "bool_pcr", + "certificate", + "bundle", + ], +) +def test_signed_but_unacceptable_claims_fail(trusted, change): + doc, key, nonce, context = evidence(trusted) + claims = trusted.claims(key.export_key(format="DER"), nonce, context) + if change in ("nonce", "context"): + claims["nonce" if change == "nonce" else "user_data"] = bytes(32) + elif change == "key": + claims["public_key"] = bytes(44) + elif change == "small_order_key": + claims["public_key"] = nitro._ED25519_SPKI_PREFIX + b"\x01" + bytes(31) + elif change in ("pcr", "debug"): + claims["pcrs"][0] = bytes(48) if change == "debug" else bytes([1]) * 48 + elif change == "missing_pcr": + del claims["pcrs"][2] + elif change in ("old", "future"): + claims["timestamp"] += -400_000 if change == "old" else 60_000 + elif change == "digest": + claims["digest"] = "SHA256" + elif change == "timestamp_type": + claims["timestamp"] = True + elif change == "bool_pcr": + value = claims["pcrs"].pop(1) + claims["pcrs"][True] = value + elif change == "certificate": + claims["certificate"] = claims["cabundle"][1] + else: + claims["cabundle"] = [] + altered = trusted.sign(cbor2.dumps(claims)) + with pytest.raises(BFVProtocolError): + nitro.verify_nitro_attestation( + altered, nitro.NitroAttestationPolicy(PCRS), nonce=nonce, context=context + ) + + +@pytest.mark.parametrize( + "change", ["signature", "algorithm", "unprotected", "trailing", "oversize"] +) +def test_cose_and_size_rejections(trusted, change): + doc, key, nonce, context = evidence(trusted) + if change == "signature": + doc = doc[:-1] + bytes([doc[-1] ^ 1]) + elif change in ("algorithm", "unprotected"): + doc = trusted.sign( + cbor2.dumps(trusted.claims(key.export_key(format="DER"), nonce, context)), + protected=cbor2.dumps({1: -7}) if change == "algorithm" else None, + unprotected={1: -35} if change == "unprotected" else None, + ) + elif change == "trailing": + doc += b"\x00" + else: + doc = bytes(nitro.MAX_NITRO_DOCUMENT_BYTES + 1) + with pytest.raises(BFVProtocolError): + nitro.verify_nitro_attestation( + doc, nitro.NitroAttestationPolicy(PCRS), nonce=nonce, context=context + ) + + +@pytest.mark.parametrize( + "raw", + [ + b"\xa2\x01\x01\x01\x02", + b"\x9f\x01\xff", + b"\x01\x02", + b"\x81" * 50 + b"\x00", + b"\xd8\x1c\x81\xd8\x1d\x00", + b"\xc2\x41\x01", + b"\xd9\xd9\xf7\x01", + b"\xd8\x63\x01", + b"\x5b\xff\xff\xff\xff\xff\xff\xff\xff", + b"\xa1\xf5\x01\x00", + ], +) +def test_bounded_cbor_rejects_ambiguous_or_dangerous_encodings(raw): + with pytest.raises(BFVProtocolError): + nitro._cbor(raw) + + +def test_policy_copies_pins_and_rejects_empty_debug_or_mutable_trust(): + pins = dict(PCRS) + policy = nitro.NitroAttestationPolicy(pins) + digest = policy.digest() + pins[0] = bytes(48) + assert policy.digest() == digest + with pytest.raises(TypeError): + policy.pcrs[0] = bytes(48) + for invalid in ({}, {0: PCRS[0]}, {**PCRS, 0: bytes(48)}, {**PCRS, 3: b"x"}): + with pytest.raises(ValueError): + nitro.NitroAttestationPolicy(invalid) + for kwargs in ( + {"max_age_seconds": 301}, + {"max_session_seconds": 0}, + {"max_clock_skew_seconds": True}, + {"handshake_timeout_seconds": 61}, + ): + with pytest.raises(ValueError): + replace(policy, **kwargs) + + +def test_missing_nsm_never_falls_back_to_local_claims(): + with pytest.raises(OSError): + nitro.NitroNSM("/nonexistent/xtrace-libnsm.so") + with pytest.raises(ValueError): + nitro.NitroNSM("relative-libnsm.so") + + +def test_historical_real_aws_certificate_profile_and_signature(): + """Catch incompatibilities hidden by synthetic certificates with ideal extensions.""" + raw = (Path(__file__).parent / "fixtures/nitro/aws-2025-01-06.cose").read_bytes() + doc, expiry = nitro._verified_document(raw, 1736179625) + assert doc["timestamp"] == 1736179625472 + assert expiry > 1736179625 + # Historical evidence is not a fresh attestation of this application. + with pytest.raises(BFVProtocolError): + nitro._verified_document(raw, time.time()) + with pytest.raises(BFVProtocolError): + nitro._verified_document(raw[:-1] + bytes([raw[-1] ^ 1]), 1736179625) + + +def test_nsm_c_abi_lengths_order_errors_and_close(tmp_path): + """Compile a test-only ABI shim; no hardware evidence is fabricated by the SDK.""" + compiler = shutil.which("cc") + if compiler is None: + pytest.skip("C compiler required for the NSM ABI regression") + source = Path(__file__).parent / "native/nitro_nsm_stub.c" + library = tmp_path / "test-only-nsm.so" + subprocess.run( + [ + compiler, + "-std=c11", + "-shared", + "-fPIC", + "-Wall", + "-Wextra", + "-Werror", + str(source), + "-o", + str(library), + ], + check=True, + ) + provider = nitro.NitroNSM(str(library)) + assert provider.random_seed() == bytes(range(32)) + assert provider.attest(b"k" * 44, b"n" * 32, b"c" * 32) == b"c" * 32 + b"n" * 32 + b"k" * 44 + for mode in (1, 2): + provider._lib.test_mode(mode) + with pytest.raises(BFVProtocolError): + provider.random_seed() + for mode in (1, 3): + provider._lib.test_mode(mode) + with pytest.raises(BFVProtocolError): + provider.attest(b"k" * 44, b"n" * 32, b"c" * 32) + provider.close() + provider.close() + assert provider._lib.test_closes() == 1 + with pytest.raises(BFVProtocolError): + provider.random_seed() + with pytest.raises(BFVProtocolError): + provider.attest(b"k" * 44, b"n" * 32, b"c" * 32) + provider._lib.test_mode(4) + with pytest.raises(BFVProtocolError): + nitro.NitroNSM(str(library)) diff --git a/tests/x_vec/test_bfv_nitro_aws.py b/tests/x_vec/test_bfv_nitro_aws.py new file mode 100644 index 0000000..92f4d9e --- /dev/null +++ b/tests/x_vec/test_bfv_nitro_aws.py @@ -0,0 +1,41 @@ +"""Opt-in integration against a fresh real enclave; never satisfied by a mock. + +Set XTRACE_BFV_NITRO_PINS to OWNER-TRUSTED nitro-cli measurement JSON and expose +the empty enclave's relay on localhost:9000 (or the explicit HOST/PORT variables). +Only synthetic data is used. This test registers one immutable index. +""" + +import os +import secrets +from pathlib import Path + +import pytest + +pytestmark = pytest.mark.skipif( + not os.environ.get("XTRACE_BFV_NITRO_PINS"), + reason="A fresh real Nitro enclave and trusted EIF pins are required", +) + + +def test_real_nitro_end_to_end(): + from experiments.bfv_nitro.client import load_measurements + from experiments.bfv_nitro.transport import NitroRemote + from xtrace_sdk.x_vec.crypto.bfv_assurance import bfv_review_policy + from xtrace_sdk.x_vec.crypto.bfv_attested_client import BFVAttestedClient + from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient + + policy = bfv_review_policy() + pins = load_measurements(Path(os.environ["XTRACE_BFV_NITRO_PINS"])) + client = BFVAttestedClient( + BFVClient(**policy.config()), secrets.token_bytes(32), pins, index_epoch=1, policy=policy + ) + remote = NitroRemote( + os.environ.get("XTRACE_BFV_NITRO_HOST", "127.0.0.1"), + int(os.environ.get("XTRACE_BFV_NITRO_PORT", "9000")), + policy=policy, + ) + vectors = [[0] * 512, [1] * 512, [i % 2 for i in range(512)]] + setup = client.prepare_index(vectors) + remote.register(client.registration_packet(setup)) + client.accept_attestation(*remote.attest(client.begin_attestation())) + assert client.finish_query(*remote.search(client.begin_query([0] * 512))) == [0, 512, 256] diff --git a/tests/x_vec/test_bfv_nitro_transport.py b/tests/x_vec/test_bfv_nitro_transport.py new file mode 100644 index 0000000..25f99f8 --- /dev/null +++ b/tests/x_vec/test_bfv_nitro_transport.py @@ -0,0 +1,130 @@ +"""Exercise the actual framed service with hostile headers and an untrusted relay. + +Local socket pairs substitute only for networking/hardware, while owner/receipt +signatures and certificate validation execute normally against a test-only CA. +""" + +import json +import socket +import threading +import time + +import pytest + +pytest.importorskip("cbor2") +pytest.importorskip("OpenSSL") + +from experiments.bfv_nitro.service import EnclaveApplication +from experiments.bfv_nitro import transport as wire +from tests.x_vec.test_bfv_attested_client import make_session, private, issuer, trusted +from xtrace_sdk.x_vec.crypto.bfv_security import BFVProtocolError, _unpack + + +def call(app, kind, payload=b"", header=None): + """Exchange a single frame with the measured service's real handler.""" + client, server = socket.socketpair() + + def serve(): + with server: + app.handle(server, timeout=1.0) + + thread = threading.Thread(target=serve) + thread.start() + with client: + client.settimeout(2) + if header is not None: + client.sendall(header) + else: + wire.send_frame(client, kind, payload, app.policy.max_setup_bytes + 1024) + deadline = time.monotonic() + 2 + returned, length = wire.receive_header(client, deadline) + assert length <= app.policy.max_response_bytes + 256 + body = wire.receive_exact(client, length, deadline) + thread.join(2) + assert not thread.is_alive() + return returned, body + + +def test_complete_framed_roundtrip(private, trusted): + client, _, setup, *_ = make_session(private, trusted, enroll=False) + app = EnclaveApplication(trusted, policy=client.policy, backend="optimized") + assert call(app, wire.REGISTER, client.registration_packet(setup)) == ( + wire.REGISTER, + b"registered", + ) + kind, body = call(app, wire.ATTEST, client.begin_attestation()) + assert kind == wire.ATTEST + client.accept_attestation(*_unpack(body, limit=32768)) + kind, body = call(app, wire.SEARCH, client.begin_query([0, 1, 0])) + assert kind == wire.SEARCH + assert client.finish_query(*_unpack(body, limit=client.policy.max_response_bytes + 256)) == [ + 0, + 3, + 1, + ] + # Reject replacement before reading the declared (large) setup body. + header = wire._HEADER.pack(wire._MAGIC, wire.REGISTER, app.policy.max_setup_bytes) + assert call(app, wire.REGISTER, header=header)[0] == wire.ERROR + + +@pytest.mark.parametrize( + "magic,kind,length", + [(b"BAD!", 1, 0), (b"XBN1", 99, 0), (b"XBN1", 1, 2**32 - 1), (b"XBN1", 3, 1_000_000)], +) +def test_invalid_headers_fail_before_body_allocation(private, trusted, magic, kind, length): + client, *_ = make_session(private, trusted, enroll=False) + app = EnclaveApplication(trusted, policy=client.policy) + assert call(app, kind, header=wire._HEADER.pack(magic, kind, length))[0] == wire.ERROR + assert app.server is None + + +def test_truncation_and_absolute_deadline(): + left, right = socket.socketpair() + with left, right: + right.sendall(b"x") + right.shutdown(socket.SHUT_WR) + with pytest.raises(BFVProtocolError): + wire.receive_exact(left, 2, time.monotonic() + 1) + with pytest.raises(TimeoutError): + wire.receive_exact(left, 1, time.monotonic() - 1) + + +def test_remote_rejects_wrong_kind_or_oversize_response_before_reading(): + """An attacker cannot make a small RPC buffer a setup-sized response.""" + for kind, length in ((wire.REGISTER, 1), (wire.ATTEST, 2**32 - 1), (wire.ERROR, 65)): + with socket.create_server(("127.0.0.1", 0)) as listener: + + def malicious(kind=kind, length=length): + stream, _ = listener.accept() + with stream: + deadline = time.monotonic() + 2 + _, size = wire.receive_header(stream, deadline) + wire.receive_exact(stream, size, deadline) + stream.sendall(wire._HEADER.pack(wire._MAGIC, kind, length)) + + worker = threading.Thread(target=malicious) + worker.start() + remote = wire.NitroRemote(port=listener.getsockname()[1], timeout=2) + with pytest.raises(BFVProtocolError): + remote.attest(bytes(134)) + worker.join(2) + assert not worker.is_alive() + + +@pytest.mark.parametrize("algorithm", ["Sha384 { ... }", "Sha384"]) +def test_owner_loads_official_cli_measurement_format(tmp_path, algorithm): + """Accept the literal HashAlgorithm emitted by the locally built Nitro CLI.""" + from experiments.bfv_nitro.client import load_measurements + from tests.x_vec.nitro_fixtures import PCRS + + path = tmp_path / "measurements.json" + measurements = { + "HashAlgorithm": algorithm, + **{f"PCR{i}": digest.hex() for i, digest in PCRS.items()}, + } + path.write_text(json.dumps({"Measurements": measurements})) + assert load_measurements(path).pcrs == PCRS + measurements["HashAlgorithm"] = "Sha384-unrecognized" + path.write_text(json.dumps({"Measurements": measurements})) + with pytest.raises(ValueError, match="SHA-384"): + load_measurements(path) diff --git a/uv.lock b/uv.lock index a7fcce6..bf372fd 100644 --- a/uv.lock +++ b/uv.lock @@ -250,6 +250,54 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3c/2a/1428f6594799780fe6ee845d8e6aeffafe026cd16a70c878684e2dcbbfc8/botocore-1.42.68-py3-none-any.whl", hash = "sha256:9df7da26374601f890e2f115bfa573d65bf15b25fe136bb3aac809f6145f52ab", size = 14668816, upload-time = "2026-03-13T19:31:58.572Z" }, ] +[[package]] +name = "cbor2" +version = "6.1.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c6/14/b02446bacfe44351b1689c04937ade007588f44570431880a6937e525e6c/cbor2-6.1.4.tar.gz", hash = "sha256:01ecc79a28f33d17331943ce508fc1e21f4b06553c73f874f4c77120d72b2ef9", size = 90840, upload-time = "2026-08-01T20:41:39.797Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a7/84/1e363301c06f509963d134f5479e82b3ade87fb1495ddacf9bf7ff24ac42/cbor2-6.1.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8156fdeb73c3ff6c8cf67ad414fb5c887cd708ff0af6d61f62629f41cb4c17b2", size = 414947, upload-time = "2026-08-01T20:40:37.405Z" }, + { url = "https://files.pythonhosted.org/packages/8d/96/d8e1ed3e79ea20a3423a96b5c89ce794fa02cb428e4429e601f8ebcbac7c/cbor2-6.1.4-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:e1fe2d62c50df290576280b18247ec63486f78be73e285bae269c2456c6ddff0", size = 457343, upload-time = "2026-08-01T20:40:38.868Z" }, + { url = "https://files.pythonhosted.org/packages/d5/0c/5796c2ed2dcd0696fc4abedf0ea0dfd5361b3f022a311481f977fa51b2b8/cbor2-6.1.4-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:c204a75f91f8cd9ed0881f6b88ec395c59aeac9fcf4d08155e7f899db2a1c46e", size = 464314, upload-time = "2026-08-01T20:40:40.63Z" }, + { url = "https://files.pythonhosted.org/packages/b1/88/de524c6c2c91b740e5df6e6955a113fb616e979b26fd2e6a0693082d36e0/cbor2-6.1.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:28fa5db05a7eae8fd80709959988d8a7f12838c6d4e5c58ec951414058641195", size = 523053, upload-time = "2026-08-01T20:40:42.602Z" }, + { url = "https://files.pythonhosted.org/packages/84/07/cb5fd92834633508d680a5b5695aeaf99d33ca0bdc5b844550d538f335b0/cbor2-6.1.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:316e217a496640418d3137483279d0e70053b000cdd4b52a4dbf20ea478bc40a", size = 532177, upload-time = "2026-08-01T20:40:44.058Z" }, + { url = "https://files.pythonhosted.org/packages/c9/19/be98721365edfe6fc23e6bcd1385afa0e960b247c5f0b50bb67f5d05e2d9/cbor2-6.1.4-cp311-cp311-win32.whl", hash = "sha256:4903f24e0f9087275a0b6606c8b0aa586277001d51e4844fcdbc5b7211330aa8", size = 281660, upload-time = "2026-08-01T20:40:45.761Z" }, + { url = "https://files.pythonhosted.org/packages/16/23/d54f679d4b155918f5a0879dab78203ce4fd514d311b7cfeba27dafe480b/cbor2-6.1.4-cp311-cp311-win_amd64.whl", hash = "sha256:5b99305d4013867e059f147752b95f728680682ab03d75a3f4dcfbb270d8dfe9", size = 303207, upload-time = "2026-08-01T20:40:47.293Z" }, + { url = "https://files.pythonhosted.org/packages/53/3c/b3839d6213c88b249ba860525df05ff18b27bdc28ebc09cb1547790f001a/cbor2-6.1.4-cp311-cp311-win_arm64.whl", hash = "sha256:bd20ecc5c8ece24db952e48a91c8c47319eaa6358af707c85ac2bb388a79abc8", size = 296123, upload-time = "2026-08-01T20:40:48.808Z" }, + { url = "https://files.pythonhosted.org/packages/2e/76/fb64293c19cafb860060310c57b768fd9cfb7cf592449660b756538cc116/cbor2-6.1.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1fc15061553e4494dc10883237501e3402c645fe509248dd698e1faf2460d68b", size = 404608, upload-time = "2026-08-01T20:40:50.219Z" }, + { url = "https://files.pythonhosted.org/packages/96/ac/f58b3bafce7c86ada2ad8eaf189453136d2cf5bae526ea0540e1b9bc9d06/cbor2-6.1.4-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:d9ada5a6ccfbb8ea7a3aa2aeb028421b52d8e0cd9323f0a2aeaa9c09d25fbce2", size = 449851, upload-time = "2026-08-01T20:40:51.725Z" }, + { url = "https://files.pythonhosted.org/packages/f0/a5/10c6c126d59b07f2bd005094dd12a20afa46146f7e2673ed6f61a57641a7/cbor2-6.1.4-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:310f3dfb296ba48fe9b63c5cf26e691e3548a1eae6901d2f0c18e941d151f220", size = 461193, upload-time = "2026-08-01T20:40:53.446Z" }, + { url = "https://files.pythonhosted.org/packages/15/e4/4445e6237088d1cca3b8536daeb90d6b4e23776de5609c9fa46773874757/cbor2-6.1.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5e6c76004d674ad1c620660cb0bc5a8a0b72a5d8c7b70926d8e09e6d7e87332f", size = 516937, upload-time = "2026-08-01T20:40:54.952Z" }, + { url = "https://files.pythonhosted.org/packages/8c/87/9c0959510f7a402e5995c81ccfd82cb9f314140dc0cce88c12836e5b93f1/cbor2-6.1.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:32a4663425fbca4a4a7aa918eb5789d844c406439e58424cf34511f79f559242", size = 529229, upload-time = "2026-08-01T20:40:56.365Z" }, + { url = "https://files.pythonhosted.org/packages/91/8e/6811e4ee84203ac657f6f461a37c7c9ba0287bde80eb83c7971e9b3fe156/cbor2-6.1.4-cp312-cp312-win32.whl", hash = "sha256:2310f07db3f9ba26f2a623774ff9f3dc7185af54f732ea119785a6b1bf7e1e7e", size = 278810, upload-time = "2026-08-01T20:40:57.76Z" }, + { url = "https://files.pythonhosted.org/packages/da/27/87440788fc0d9513534c3c699238e2a9ca6010f8cb72e9c203b7af20a9f6/cbor2-6.1.4-cp312-cp312-win_amd64.whl", hash = "sha256:cc8cd300e236e9797b2e1ce306109dc481fcccf78bfa2682bf36d99e6eab1ec6", size = 299971, upload-time = "2026-08-01T20:40:59.256Z" }, + { url = "https://files.pythonhosted.org/packages/23/f9/77981e6e63092de19d7306a09a12b0eb3fd2907dc22c10dd5d389eb27faf/cbor2-6.1.4-cp312-cp312-win_arm64.whl", hash = "sha256:553a46bda7d09552631a714e22b91e6ff2c867ecd91511596ce290d8879b8d5b", size = 290662, upload-time = "2026-08-01T20:41:00.89Z" }, + { url = "https://files.pythonhosted.org/packages/0d/17/0b20c88e76942ede86c98cdce138681690f95908c540c264fff847729cd4/cbor2-6.1.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c48a7c938fc5fa5300ff82b5df09068dcb4838685ae8556b5ee8279d74f97ab4", size = 403677, upload-time = "2026-08-01T20:41:02.561Z" }, + { url = "https://files.pythonhosted.org/packages/35/3d/93eed770864540c5c9ea0841008208e9db686b7335f42520705b7d6dc6b2/cbor2-6.1.4-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:4bd29f21529e279d50fc14f1a811f7b05b4d8e66a7969163cce98983b6817245", size = 449762, upload-time = "2026-08-01T20:41:04.094Z" }, + { url = "https://files.pythonhosted.org/packages/e3/21/69e4d37f00319b3d37322355aedc83154b4d8b75dc9e9789c06e1fbd8a92/cbor2-6.1.4-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:36ae16d64b1f7b620c1af748e7b6947e20069ef80eee56871c5fbb84cc635905", size = 460420, upload-time = "2026-08-01T20:41:05.891Z" }, + { url = "https://files.pythonhosted.org/packages/be/26/2cfdd5ee826205a88a826bb38b7a572c676ec3efa29574be5cdbd04b4859/cbor2-6.1.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:69978901302ecbc8cda57b520487c5c5240ed217de783eb7728fceb258311d76", size = 516490, upload-time = "2026-08-01T20:41:07.52Z" }, + { url = "https://files.pythonhosted.org/packages/82/86/d687cd1c2c9f9a986e8552ad1fdbd22411cc86389b5705dba6ec6f7e3226/cbor2-6.1.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ad4efa23fee6447e56a269191044e06eb39e809458bcd674e164fe9445feafd0", size = 528810, upload-time = "2026-08-01T20:41:09.144Z" }, + { url = "https://files.pythonhosted.org/packages/40/08/88cecf20b8825bdd991c47b317415c08ef9e7d5f05a1def9acd346edabde/cbor2-6.1.4-cp313-cp313-win32.whl", hash = "sha256:d2560c2ba6a95904ba2a0ca257af878c4344409d9b46d8e646d8ebb617b1e0dd", size = 278058, upload-time = "2026-08-01T20:41:10.48Z" }, + { url = "https://files.pythonhosted.org/packages/0e/67/ba140234a6415c16dcfbe0585ce12f905157b70e9cb1bb63a2b6d5721e70/cbor2-6.1.4-cp313-cp313-win_amd64.whl", hash = "sha256:c08b9c7d2ea013e24a0cb819b872b0119dde404f64a1182c0b24095b7bba781f", size = 299315, upload-time = "2026-08-01T20:41:12.067Z" }, + { url = "https://files.pythonhosted.org/packages/5f/7f/35d53ff4252a5a85656480d3a81d5a5af823979ccd0c5cac95196a7548a6/cbor2-6.1.4-cp313-cp313-win_arm64.whl", hash = "sha256:598710183daae69cbdeb177a870ec64aa601de8138a61491fd256826d15a860f", size = 289976, upload-time = "2026-08-01T20:41:13.63Z" }, + { url = "https://files.pythonhosted.org/packages/05/5d/c5374c76471ab41dff4420a276569a56352e83166374fba6f40fd0bde7ad/cbor2-6.1.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:24da0a481294ac416e1e369e2d204b2b1d993cbd082d0d99fa3d6f5f27ae5e69", size = 407497, upload-time = "2026-08-01T20:41:15.189Z" }, + { url = "https://files.pythonhosted.org/packages/46/f9/b9f12a5e24d5ae355e4c0f6d37330a2bbedad3331247a223a51c4cd39d5e/cbor2-6.1.4-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:0859a0837e6e2d4fe5f5b849f6475797e4db545da98c19db4b1d3487bd47aa22", size = 452191, upload-time = "2026-08-01T20:41:16.705Z" }, + { url = "https://files.pythonhosted.org/packages/67/22/8224b01f95a6fe07b1a64082aea34d9f49068392b3de93f5f3a10c73c62e/cbor2-6.1.4-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:c0f5f2d6d3b58e44146860c049f3c082207a4005588b8926d51bf937ab66773c", size = 462383, upload-time = "2026-08-01T20:41:18.17Z" }, + { url = "https://files.pythonhosted.org/packages/92/52/437e4aa4f5df1fb41020d64b3d99a8239f0f99a3a75eb6ffa5cb66004b7f/cbor2-6.1.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:239db0f92d537fd29eaec4e40195fc3b2b48bc34a5887059658162489a9eb6ae", size = 518700, upload-time = "2026-08-01T20:41:19.592Z" }, + { url = "https://files.pythonhosted.org/packages/7d/45/2f5ea5bfe0fd800b3739c7df8679bdffa9f7def6b2f2fee064ada1c63e85/cbor2-6.1.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:3f4a434c36bb0d33aeb48ddae8e8b673ca7e1f14545ee7cf4a4c7c39380ea9a2", size = 531243, upload-time = "2026-08-01T20:41:21.21Z" }, + { url = "https://files.pythonhosted.org/packages/bd/c6/0beac64cb74cd3217f295f9bb0d64675e1809c683a31ea2a49ac9d4d1504/cbor2-6.1.4-cp314-cp314-win32.whl", hash = "sha256:6abcf072b8c0fdc8ad7902ee26a906cafbf3427d026b662ff21166a253f85e18", size = 285248, upload-time = "2026-08-01T20:41:22.658Z" }, + { url = "https://files.pythonhosted.org/packages/bb/7d/4afa096ddc94049f5a514690891b02a18319e146ceb14465ce30c8340a8b/cbor2-6.1.4-cp314-cp314-win_amd64.whl", hash = "sha256:855764e02dc60ab9413acd044e997c3170000fdea6155d6c43a923a1d966dbe6", size = 313044, upload-time = "2026-08-01T20:41:24.066Z" }, + { url = "https://files.pythonhosted.org/packages/e5/b5/e614cee861772f6b5c4d926b066d2e7dbc11e220b50ba716ba91e430fb0f/cbor2-6.1.4-cp314-cp314-win_arm64.whl", hash = "sha256:c6b28b928c5f2dbf47dffa12dce9c8e36fe6ac1c1358bc326499c0736263b66f", size = 304088, upload-time = "2026-08-01T20:41:25.431Z" }, + { url = "https://files.pythonhosted.org/packages/9e/41/3b28184154f6cbf7e47c1b7fb4a7a291c54f27a6f3a0a2f64b078c6a13e1/cbor2-6.1.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7336ff4cb7d161ec43b65eef43bf3e9bcab44bd152efb54dd637b7afe711254f", size = 401042, upload-time = "2026-08-01T20:41:26.819Z" }, + { url = "https://files.pythonhosted.org/packages/d5/1a/a8624023b84b41c43a150a89517c104aed0e467bd258866f13be4c3ac0c6/cbor2-6.1.4-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:8f1019494b0ec81a3df3ebb01b6acb446d5b946fe35845b1726379abd66a71da", size = 445301, upload-time = "2026-08-01T20:41:28.35Z" }, + { url = "https://files.pythonhosted.org/packages/60/39/07dd0ea957c1f48673d3947f97ee36826efd4a824053dd0ec4df2f0c89d6/cbor2-6.1.4-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:179a794bf4be1d46ff190695929f65f0b42019c156919846ae539d2a7ec42e54", size = 459816, upload-time = "2026-08-01T20:41:29.839Z" }, + { url = "https://files.pythonhosted.org/packages/23/8e/2015175132a27c1daed434f671ac6d9c1311461995df47f201307700e0da/cbor2-6.1.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:9b904b8d0f4ddac9259197d21d121fae4cb8b555700d65bc12c5d46a2e6c2025", size = 511565, upload-time = "2026-08-01T20:41:31.939Z" }, + { url = "https://files.pythonhosted.org/packages/82/66/420991095d9473614b205d4c4e40b5d3b9f1ee4410eb3c48c1e902947837/cbor2-6.1.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:71fcf4f237d68bf4445bf45070f36f82b333f2e6a62612aa2c256683b51378a9", size = 527709, upload-time = "2026-08-01T20:41:33.413Z" }, + { url = "https://files.pythonhosted.org/packages/cc/7c/73057e7a38488a816a0d40ff9e7cd9f418800894582e2e48fb2f47ce66a2/cbor2-6.1.4-cp314-cp314t-win32.whl", hash = "sha256:7deccc50fd0b55c4c7dd265b144c5358a645121e457c0ae3722b5ad59832b257", size = 281462, upload-time = "2026-08-01T20:41:35.127Z" }, + { url = "https://files.pythonhosted.org/packages/99/5d/d5db22837cb566de733b9d1c418cdf1912ccb1efc7b179e295430b1d81a2/cbor2-6.1.4-cp314-cp314t-win_amd64.whl", hash = "sha256:f3fc7d15cba4174373df2496070faa4a927fe3ed772130d281808120aec7b61c", size = 309165, upload-time = "2026-08-01T20:41:36.716Z" }, + { url = "https://files.pythonhosted.org/packages/29/5f/ff2c6da83553a692219a0a62a21b57a27ded4405200e50db758a17fbaf15/cbor2-6.1.4-cp314-cp314t-win_arm64.whl", hash = "sha256:164ca22b509408435b2d8236c80c964e4fc77c085ab034569cd04c40d5cc8883", size = 298386, upload-time = "2026-08-01T20:41:38.392Z" }, +] + [[package]] name = "certifi" version = "2026.2.25" @@ -259,6 +307,104 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9a/3c/c17fb3ca2d9c3acff52e30b309f538586f9f5b9c9cf454f3845fc9af4881/certifi-2026.2.25-py3-none-any.whl", hash = "sha256:027692e4402ad994f1c42e52a4997a9763c646b73e4096e4d5d6db8af1d6f0fa", size = 153684, upload-time = "2026-02-25T02:54:15.766Z" }, ] +[[package]] +name = "cffi" +version = "2.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pycparser", marker = "implementation_name != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9e/ef/008a1939e372c06329a3fce4279c02f328488f3526744906eeec3da7ad5f/cffi-2.1.1.tar.gz", hash = "sha256:dd31f52ea1086513bb9df30f8fcee9b8918323ae067a3d5b78bc826a000712be", size = 530807, upload-time = "2026-08-03T21:21:18.939Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/70/d2/16d99a0c4948febc0ebd133a13b2f688ff7f8cb04da971e1128872ce0c03/cffi-2.1.1-cp311-cp311-macosx_10_15_x86_64.whl", hash = "sha256:c8d2c9fd1f2d16f780d15127abb050d13d1a76c03a4bd87d7e4980e45e511e12", size = 183838, upload-time = "2026-08-03T21:19:29.637Z" }, + { url = "https://files.pythonhosted.org/packages/cd/95/31b535a9f0220ae9f357de4a08d57ce89cb417653c2fd9f075f50822a388/cffi-2.1.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:398aff33cee2767e3e781d2554c54bd0dff386bb437581e0d8011fde1a942ec1", size = 184168, upload-time = "2026-08-03T21:19:30.764Z" }, + { url = "https://files.pythonhosted.org/packages/ad/5a/4707a0dc1f203f5dde5a907b0d4e3c25d71120241048bd5bc6f1bb9d4e71/cffi-2.1.1-cp311-cp311-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:154852545011f779917b11c78db2358d095da62a9a172b78ad0a583ee5adc0d0", size = 211805, upload-time = "2026-08-03T21:19:31.867Z" }, + { url = "https://files.pythonhosted.org/packages/ad/66/c19feabb28485b6e0bbaaafa90837a1ef5d302e90f2178bd33f17a49879b/cffi-2.1.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3311ed60d36f83378794e1009ac6258bafbf81f7888b4caa7b35a521e3f95813", size = 218716, upload-time = "2026-08-03T21:19:32.896Z" }, + { url = "https://files.pythonhosted.org/packages/a7/92/500760486c8baab49a7a8a58ba7fc3355ec3974b454b8a09e528efde9e1d/cffi-2.1.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:6e192623c49c94421616a5778fba35cf0d5a8d000650c1967ef4448ee5cdd990", size = 205569, upload-time = "2026-08-03T21:19:34.142Z" }, + { url = "https://files.pythonhosted.org/packages/a5/a7/a67c733254d6e7373f7822f8082d8d6beade791e0cf12a7611f376fa61c7/cffi-2.1.1-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a6e721d4b0e45d5b65e87534470e67b18dcd092c83f68fba09f152b9cbc061af", size = 204907, upload-time = "2026-08-03T21:19:35.174Z" }, + { url = "https://files.pythonhosted.org/packages/f7/a4/4399daaf8f7dfee9d7c3327fdb0426ee041cc63edc358b93911ceb2bfc7a/cffi-2.1.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:34e261f78cb6ceaaa36f42f2613f4380d94d9c759a9c73c769ee6e0247364632", size = 217807, upload-time = "2026-08-03T21:19:36.286Z" }, + { url = "https://files.pythonhosted.org/packages/28/f7/dabe6da2466ecbd82dc62e7342dc6b1065dad990c06f00f0ede9ebf2a0ed/cffi-2.1.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:7225e4514edb64eb6740324353e0da0711954fd8d7da4576755b1c6e09b697cd", size = 221252, upload-time = "2026-08-03T21:19:37.416Z" }, + { url = "https://files.pythonhosted.org/packages/ce/87/616202d8e51342c07d2534c510111c4cc37201775ce8f60802c9335d1edd/cffi-2.1.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:df913725b79db7bcf03448f36b7bf8815363417d5b58deecf9305e3e30f0f21a", size = 214214, upload-time = "2026-08-03T21:19:38.507Z" }, + { url = "https://files.pythonhosted.org/packages/b4/c6/ab025d75d2c26c19b087c0124e75ee31cb65032f4fe345d356d8c507ab97/cffi-2.1.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f5cfbc5fe74540d335175b656c725d74d90e3730c626d92575eea35029d9afaa", size = 219408, upload-time = "2026-08-03T21:19:39.809Z" }, + { url = "https://files.pythonhosted.org/packages/db/e2/7e8109f65445bdc673a7b54f02c677de462db75674220fd1335efc8eb598/cffi-2.1.1-cp311-cp311-win32.whl", hash = "sha256:f8ec5e643a9a937f64e1999eb9f75d072263751912dc5cd06d3c85f8f44be7c3", size = 174470, upload-time = "2026-08-03T21:19:41.246Z" }, + { url = "https://files.pythonhosted.org/packages/73/c0/77ba02423c2f7d7091143c45cd49e0e6575c4c1967394bb542bd923a9b74/cffi-2.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:42f6930c31dc7f50732c9ae793c2786c7b6b044195967bbdde40bb9be81c4cc0", size = 185096, upload-time = "2026-08-03T21:19:42.615Z" }, + { url = "https://files.pythonhosted.org/packages/7c/47/9f1f85f9672ceda4984dc6c4f8824e8558992a2972c3d3c81fb8eb28d4ba/cffi-2.1.1-cp311-cp311-win_arm64.whl", hash = "sha256:c7659f22557c5a0bc4855cd635f55edec690cc008a40768527762cb9fb263455", size = 179941, upload-time = "2026-08-03T21:19:43.747Z" }, + { url = "https://files.pythonhosted.org/packages/10/69/43965eccfdead3b9220015fd1320e117be8c6ed01a62ffab76eeb752f5d5/cffi-2.1.1-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:c8c69575568085ba0b1b10c0249d779a214aea6f6522e949a0fc9fb0fcb449d0", size = 184821, upload-time = "2026-08-03T21:19:44.887Z" }, + { url = "https://files.pythonhosted.org/packages/54/7d/16e5a096677b5e313ca80cd5e5170efa3ea44624a82bb111925522da64b1/cffi-2.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f81b3b8f3d4e343550fa4baa0e479bba9f2d29ce9c2e9b51d1ce1718d7442fcf", size = 184719, upload-time = "2026-08-03T21:19:46.129Z" }, + { url = "https://files.pythonhosted.org/packages/56/e6/8941622732edec876dd17d0453dce07317ae96db34f2ec1436c9d3785986/cffi-2.1.1-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:811bd1e21d32de12efca32393a0ab3f5133b54fce9bd44b8bd77ab07da14bf6a", size = 214799, upload-time = "2026-08-03T21:19:47.218Z" }, + { url = "https://files.pythonhosted.org/packages/44/de/f98430906df1545ffde0d543dd124a7a439bc2cd32b36b9c53f805df7333/cffi-2.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:68e62fe11f30d5ca8289242866f0a5291402d8529ca2178ab8afc5c9694ae890", size = 222389, upload-time = "2026-08-03T21:19:48.331Z" }, + { url = "https://files.pythonhosted.org/packages/6a/5b/717f1526b9957b34456313c31645c5b82b8fb5c3fe9e4752999be7128bfc/cffi-2.1.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:4a7c934f7360e8cd64fe9efadcbd10c7c6364f531e432b9a4bf5ccbc9e0e8b50", size = 210249, upload-time = "2026-08-03T21:19:49.543Z" }, + { url = "https://files.pythonhosted.org/packages/64/b3/f8aa4f3e34986c7e4ec45072d1b1b9dd295b6b18007b45518d79726dd725/cffi-2.1.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:3143d81e29e1e20a9ce10901ec369012947876596f75a222235965f2b7ae832e", size = 208775, upload-time = "2026-08-03T21:19:50.918Z" }, + { url = "https://files.pythonhosted.org/packages/b1/db/dceb9dd5b231e1da801793f8acc9f3c52a7e1afe40bb1aae37e02b0faad5/cffi-2.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c1453022f490d2459a11819d83ad1d586e9ff65a12ac3e705ffebd46d3685dcf", size = 221822, upload-time = "2026-08-03T21:19:52.054Z" }, + { url = "https://files.pythonhosted.org/packages/a0/d2/6cd24ae3be000a634109c247d1475d62e5616d0dc78c82770942ec384248/cffi-2.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:208f941bb9d18e768138677f0a6d2ce01f590df56043dda1df1535ac57c88517", size = 225232, upload-time = "2026-08-03T21:19:53.109Z" }, + { url = "https://files.pythonhosted.org/packages/cb/52/3fa190537004dd7f0ab860a6dc7c0175b8667f68d1e618a46f5498d30250/cffi-2.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:210019b6c7cf07f081b4c54635c8cf744377001350e29cc0f81c4377b4797735", size = 223597, upload-time = "2026-08-03T21:19:54.515Z" }, + { url = "https://files.pythonhosted.org/packages/80/fb/0bb75b7039588c074b37ae99f40d9bfddf990ecb2fbc346ebccd2e56b9be/cffi-2.1.1-cp312-cp312-win32.whl", hash = "sha256:046bfc24911b37851ee1b51aab8bffe713d89c68c6a057b09484ce9fd5f69b4e", size = 175292, upload-time = "2026-08-03T21:19:55.566Z" }, + { url = "https://files.pythonhosted.org/packages/d9/79/615cc094e2fb508cade7de88d3b4f6c4ec2bab695c97bce9153dc65aadf5/cffi-2.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:f53e442b08449d42821fa4a4fba000095af9f62742a500f978a9f557ec44339a", size = 185919, upload-time = "2026-08-03T21:19:56.89Z" }, + { url = "https://files.pythonhosted.org/packages/70/c6/d0ea84713fe46b243a436a18fcd47d639732747e21635c8a27191b06dc30/cffi-2.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:7bde5e4cc5c10140859842b9d383af292b22639a4dffb725314baf45968cef80", size = 180093, upload-time = "2026-08-03T21:19:58.155Z" }, + { url = "https://files.pythonhosted.org/packages/9d/f4/035513d4117049066b4779dc3b7c0c0fdad175fa13731c9f4003f1cd1478/cffi-2.1.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:b5bdfd1c873d4e093aabc0ca84c4ca6dbc4f752afb5c86f146d9742580c9da2e", size = 194248, upload-time = "2026-08-03T21:19:59.399Z" }, + { url = "https://files.pythonhosted.org/packages/76/af/2aeb4dbb5fc41a04161ae9ff1518de7cec08e164f44a8ce6a4cf7fd2cd1d/cffi-2.1.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:31348097ff5bbe827ccc41795d4dd099d9f0625e7def00ee653c137a490c2a6c", size = 196908, upload-time = "2026-08-03T21:20:00.746Z" }, + { url = "https://files.pythonhosted.org/packages/a7/46/2e5fdde8555706dd98139a910ca11be02809f3f605ce956f655d0214e100/cffi-2.1.1-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:9d2055050ea716bd38b7f7f1579c275386646b4894c155a3e2f3cd62ed41b7c6", size = 184805, upload-time = "2026-08-03T21:20:02.02Z" }, + { url = "https://files.pythonhosted.org/packages/55/41/4c7042f317b9217502988f0873af87e16ad606dc20f84e546e3e6ce9764c/cffi-2.1.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:19ee6127ee34de7d83ce3d371ebc5ed91addbdcc39f9ab15ce4eb35a4e534971", size = 184764, upload-time = "2026-08-03T21:20:03.141Z" }, + { url = "https://files.pythonhosted.org/packages/43/1f/1c3d90d91811c8f86ced9ed637956c54bfe5b79ca98fe976d7f8c8979f6b/cffi-2.1.1-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:6a8dddef476fab96d066d578fc88526767b836ab5ab21754e1d5bf3879c31c7c", size = 214722, upload-time = "2026-08-03T21:20:04.377Z" }, + { url = "https://files.pythonhosted.org/packages/37/6f/3b5ce4c3b2192d250f04908f2bfd91ef34552ec8f7716a5d4abdb8d67bb2/cffi-2.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f16c709686a78c727bbbf059f92b0bf41c6fc60deec706d2dc19f529175a6125", size = 222369, upload-time = "2026-08-03T21:20:05.544Z" }, + { url = "https://files.pythonhosted.org/packages/02/10/4b3c75dde3d9663c9e02ba05c2668b954f671d4bbe346413ca8c696b295a/cffi-2.1.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:fcd22650c908d7b7da162bbfaab594a1227a15d1643a98c68b122ac642fa2264", size = 210175, upload-time = "2026-08-03T21:20:06.75Z" }, + { url = "https://files.pythonhosted.org/packages/df/62/14f74b9543e605d17701dc797b815958b8bb70b7624ce1b832ddad48ed6c/cffi-2.1.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:aa9511c62d14da7aacc9b4bf51f3f697a621e83b2d6919008243c3aad168eea3", size = 208670, upload-time = "2026-08-03T21:20:08.04Z" }, + { url = "https://files.pythonhosted.org/packages/95/95/86342356ff5953b3fb06f7ef7c5bee212d45e770abc7218d451b9148313c/cffi-2.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a931079504ecc49efed7744c476a5c343a92fabf66dec2db95edb1b2fdc770e2", size = 221824, upload-time = "2026-08-03T21:20:09.274Z" }, + { url = "https://files.pythonhosted.org/packages/eb/ff/7b3429ff53aafe931ed8a5fc69f481bbef7ba6de87ddcbb63d08f483f613/cffi-2.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a2d7755bef5a12ed488f4ef1f1b69ee9191d7396083b755a5d2295f6edb4768b", size = 225148, upload-time = "2026-08-03T21:20:10.7Z" }, + { url = "https://files.pythonhosted.org/packages/34/34/a95870b9221e09cf4f2ce3178b1a210abdfe63a1bd357da940418d7b8d15/cffi-2.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e0bcb7e0f677f543555d2adff3bf19c05f66cdb4796e5ff602442ab2fe3c4ef7", size = 223564, upload-time = "2026-08-03T21:20:12.165Z" }, + { url = "https://files.pythonhosted.org/packages/70/ea/839b50531021a647fb5e929f72cf97bc1ff702b5472166164b5b6e76b851/cffi-2.1.1-cp313-cp313-win32.whl", hash = "sha256:334644fbac4eff73d985a17a91226df55d0f394160c4cfb880e084c8f7161cac", size = 175263, upload-time = "2026-08-03T21:20:13.559Z" }, + { url = "https://files.pythonhosted.org/packages/60/a6/8b149b2c3f2e11aaa1618ef64500b45f50f22c57a977a4dff1aff1f91042/cffi-2.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:1aa5645c30469b09530c4ebca77ebf8f17618293c58f8549cb1a543a50236e7d", size = 185688, upload-time = "2026-08-03T21:20:14.69Z" }, + { url = "https://files.pythonhosted.org/packages/01/9a/11f687cb39d6a3504060d5242f04f48c735afb4d3d533958a20594890cb2/cffi-2.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:63bbfd5ded17c4840ac07cd8f1c21ba9d9708141f840b324f422f41b207e3973", size = 180078, upload-time = "2026-08-03T21:20:15.917Z" }, + { url = "https://files.pythonhosted.org/packages/d3/7b/d6bbf82b8b96e7391438898c42f5bd96dd02030fd5b64937d248220003e2/cffi-2.1.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7dbb61fe3a7699468030f71bbe5f8a0e326a151daa91beb11a6fc1f980c55e1c", size = 194064, upload-time = "2026-08-03T21:20:17.148Z" }, + { url = "https://files.pythonhosted.org/packages/94/e6/bcc91b283be94735e268487a054004f0aa19947b6348fa367db53230abc8/cffi-2.1.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:f24fb43132a4c6b4cb4eb029492919b2db645be6808d738f244fd146c03c32cb", size = 196720, upload-time = "2026-08-03T21:20:18.268Z" }, + { url = "https://files.pythonhosted.org/packages/d9/99/c4b0c17cacdc9c3b8f280026286a9826d6a208c0f047591a3c3ce99b91fd/cffi-2.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d28630f5854ab07ab1fd4aba756de52326c82e6be15d414b12793f1975048b54", size = 184964, upload-time = "2026-08-03T21:20:19.708Z" }, + { url = "https://files.pythonhosted.org/packages/b3/a9/9db617d05d7367c1ad0ab00b3aa6e6f9281edd689b4ee9ea0e5a84e89c97/cffi-2.1.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:661c298b4821edebead0c91edd2b00374d67ad7c5a1f7a91d4442633b79d6a72", size = 184962, upload-time = "2026-08-03T21:20:20.833Z" }, + { url = "https://files.pythonhosted.org/packages/67/b8/b42132ca113dc567d37684437b46ca1dafc885902b02a110a02d5b511857/cffi-2.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:58acb8ab8e295e6c5ea12f888cbb13cf21511ef2a3303a23f4325c29d17fe5c1", size = 222328, upload-time = "2026-08-03T21:20:22.118Z" }, + { url = "https://files.pythonhosted.org/packages/80/10/c5c0cbf0a657aecf59ef511409734230bf556f05a0d6c9eed7aa5c0a0166/cffi-2.1.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:456a61fa52d579ebf9df2e9552ead5129855dbaff6c1e5a9b1bc408809bdc062", size = 209985, upload-time = "2026-08-03T21:20:23.401Z" }, + { url = "https://files.pythonhosted.org/packages/d5/6c/bfa0b87b03b9238148beca990292843c9396ba069b54496596594173de7b/cffi-2.1.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a4f00aa42f75d6e4595e8866e748cc1705adc0cddfeb2ca86d0d03993d63ba03", size = 208530, upload-time = "2026-08-03T21:20:24.628Z" }, + { url = "https://files.pythonhosted.org/packages/e9/02/4e7d553a7ac4b4238b38b3c1b80d486e9d4436f8d2acbf87a0997fe3f402/cffi-2.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b0431303acaea1089ad4b3e9ce4e6518193def1118d4073ca848635ee4ea2e96", size = 221525, upload-time = "2026-08-03T21:20:25.758Z" }, + { url = "https://files.pythonhosted.org/packages/82/1d/a4aaf9babd75acb4d5f223bff71533bee748dd770a382619a798960ee9ba/cffi-2.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:64faea20f4e2613363a1a9b9c7dd73058f3ecd00133a511e72ad7c511658f527", size = 225053, upload-time = "2026-08-03T21:20:26.985Z" }, + { url = "https://files.pythonhosted.org/packages/81/10/5dc0e7bdd18e22107054288283380fc97a06ae3f1656a106908d666a3c88/cffi-2.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5c58fe613dc5e5336357eff555824a314d8e43282600435c8d1cb6a7a2fedd13", size = 223213, upload-time = "2026-08-03T21:20:28.277Z" }, + { url = "https://files.pythonhosted.org/packages/0b/e9/d0061c364cde06ee43168a0d076ac1da512cbc380d44767b844ba34fe2b6/cffi-2.1.1-cp314-cp314-win32.whl", hash = "sha256:1a18a57b58cfb21fc28d72e876acf10eaed67a1ed96226f92af4df681d571c4c", size = 177682, upload-time = "2026-08-03T21:20:44.288Z" }, + { url = "https://files.pythonhosted.org/packages/a7/06/1c3e01e3ba14c39f6d10bfbac52753b7e22259e38088e5cfe1d704918690/cffi-2.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:3222ba5d678f80a030e6afbcc33dc1ae5cb45facabb61cee2c7016b8432fde48", size = 187949, upload-time = "2026-08-03T21:20:45.623Z" }, + { url = "https://files.pythonhosted.org/packages/87/5b/da4e39efe18eeb89cf580ea9cfc66b6a7c3eadb808fc0cc1d3a295cb5a5d/cffi-2.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:ab36d55f9ed2d067327667c2fea18dda018eb628dd6347aa01dda6cf1f5d3836", size = 182947, upload-time = "2026-08-03T21:20:46.955Z" }, + { url = "https://files.pythonhosted.org/packages/23/59/40338bf421c5accea1d45158170c87006ef1cd371b05c077e76476949728/cffi-2.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7750c6449dff7864bb9bb27ddfb0267756189201a3afc911d82b3caacd70dfc3", size = 188504, upload-time = "2026-08-03T21:20:29.495Z" }, + { url = "https://files.pythonhosted.org/packages/7d/47/5ecf1023850036e674c77ec4de86182d309ae344e39e7cba984b7df5d647/cffi-2.1.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0beceaabe56af686895136a2de78db54ecd8e4046b236b8fd6d6cb61389e9bf2", size = 188259, upload-time = "2026-08-03T21:20:31.291Z" }, + { url = "https://files.pythonhosted.org/packages/2a/9c/92934c3bea9f785b23eba304538c0b4d37a2a96d2431eb3a1bc87a11aa19/cffi-2.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:49cbc70e6542d4ccccb936558d1064a8012541e78f821f955cff24e357776c94", size = 223864, upload-time = "2026-08-03T21:20:32.571Z" }, + { url = "https://files.pythonhosted.org/packages/4d/45/ba4c93527bc38616a8bd36488acb69a2212d60486794f0c1f318949bbb76/cffi-2.1.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:e2d65b31f36619cda3999b78b2aa9632e76b78448e7a56fc4240824200e7c4fc", size = 211538, upload-time = "2026-08-03T21:20:33.808Z" }, + { url = "https://files.pythonhosted.org/packages/80/e9/b6ef565e452acb932fb0cb5443f44a78efbd1233e566f02b5a83855e9115/cffi-2.1.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:28907ab9bfb6aa13184cfc17c6b8e1023c5ab6fd7076d8c20a35e59fe04f8f29", size = 210688, upload-time = "2026-08-03T21:20:34.974Z" }, + { url = "https://files.pythonhosted.org/packages/9a/95/eff5f0cee78d2eabc7eebffec40d3fc1876b5f3c95582e018bb4b99601f2/cffi-2.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:51b31d1c98274844cfd7838ce00bfc27c7423a4dc00fc0772fc3331c2cc90676", size = 223803, upload-time = "2026-08-03T21:20:36.564Z" }, + { url = "https://files.pythonhosted.org/packages/fa/01/579d39fb8bef00a335a23d83757b44feb24cd6345a2c451b64cb67b9c362/cffi-2.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5e7cecbaadb83884793e05828cee59b210b24583b9c7425d0ba6a754fe22eb4e", size = 226763, upload-time = "2026-08-03T21:20:37.816Z" }, + { url = "https://files.pythonhosted.org/packages/8d/b0/0b44f47c60b01b57b6e2bbd92343f13a85a1d93bc46ccf6e47e244acd99c/cffi-2.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:25792eac27877609e7bb06d42ff88278a6624fff2ba9bbb523c09616b117e80f", size = 225688, upload-time = "2026-08-03T21:20:38.959Z" }, + { url = "https://files.pythonhosted.org/packages/eb/d2/3b7176cb570a1d3e27faf67b72f591af508036e0d8b2be2ef9af9e8c84bb/cffi-2.1.1-cp314-cp314t-win32.whl", hash = "sha256:8ef53b2de9bcb9197d31854256575d59dbac0cba72ac627bb291ef5eceb74be4", size = 182868, upload-time = "2026-08-03T21:20:40.388Z" }, + { url = "https://files.pythonhosted.org/packages/56/78/31f00c1bcd97c9bbf55f1bfdf5bc809a5de8887473e90bb9960dca825e80/cffi-2.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:616f097f2fe415bc92a247f02e11f634e1f9e9a83d327e3c915c15089c87869e", size = 194104, upload-time = "2026-08-03T21:20:41.725Z" }, + { url = "https://files.pythonhosted.org/packages/7b/1b/58496f2ed0a35de575250c02a43ab3cc2c04d494a88fed31c1cabc0fd176/cffi-2.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:ad2c86c495b899d862ea0f4b42891b8713a3bd45dd4105c7fd51c2a72f39f3a5", size = 186402, upload-time = "2026-08-03T21:20:43.042Z" }, + { url = "https://files.pythonhosted.org/packages/c1/8f/9ebe220eab48a093d1a5a5e339ab0dc7316eef3bb04d63c42f0251b61f50/cffi-2.1.1-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:dddad92b554513a31f272570678ba307fb9f618f05e3d4a5eacafff9eae03e1d", size = 194043, upload-time = "2026-08-03T21:20:48.179Z" }, + { url = "https://files.pythonhosted.org/packages/ff/69/844bad3ece306c4782c2ecb93597035b6690d48704b803914c199da1e8b3/cffi-2.1.1-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:da0e573f9f97159390c89d9f1a9e41908b66d408cc5b58d08cf3847d844c531b", size = 196737, upload-time = "2026-08-03T21:20:49.457Z" }, + { url = "https://files.pythonhosted.org/packages/1b/8a/af668013284634733f02d683458a0728739c7d6ddb5e14cb0c20832266fe/cffi-2.1.1-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:fb92203a88b3d3053034db775110081c49d28be6551923805e039924093761e4", size = 184933, upload-time = "2026-08-03T21:20:50.639Z" }, + { url = "https://files.pythonhosted.org/packages/0c/75/2f5207ff6d1a613133b23a5203cc0c2a628313b5eb3974d7956ae3c57950/cffi-2.1.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:2ae64be792b8966f2c69538199728b290e34726562896df1e5dc8ffd8d8188e8", size = 185002, upload-time = "2026-08-03T21:20:52.173Z" }, + { url = "https://files.pythonhosted.org/packages/e2/31/9e1313b0a6e30e91b3b3d3fff51ae99c857c07738e3afcce1f7334e1b7ab/cffi-2.1.1-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:507a24c282e0f42f8ed737cf048572cbf580468da5555764a8331735e9c736b6", size = 222271, upload-time = "2026-08-03T21:20:53.462Z" }, + { url = "https://files.pythonhosted.org/packages/50/e3/f6234a833e6e08c7007003074723c406559eecf9b48dfc97471e5a8eb7a0/cffi-2.1.1-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:246fa40ce8645a614ff682e0b70f37134e460eaf93a775e0cbe3cca585a67a80", size = 209919, upload-time = "2026-08-03T21:20:54.783Z" }, + { url = "https://files.pythonhosted.org/packages/0d/fc/5f74e293fced6edb51af3a46c4ccf6c23c9943774ecb375ddbd522c76add/cffi-2.1.1-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:471cee653ae88de62096552e6d24ccb4a5adb8c8c9f10b5054d0122c15bf2779", size = 208529, upload-time = "2026-08-03T21:20:56.066Z" }, + { url = "https://files.pythonhosted.org/packages/44/16/29e6d01b388bef055ecd6ca8244b3f4d336bd09e92d5d892187b9601084e/cffi-2.1.1-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aeae0e330c9f6acd681f647d46cefd30c29f93e3392882e792e82080c9691399", size = 221630, upload-time = "2026-08-03T21:20:57.336Z" }, + { url = "https://files.pythonhosted.org/packages/a4/18/fa7f1f6857d5eb88a4ca99ffcbfb7c387a287ccc154c64a73e86314745d7/cffi-2.1.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:42a494cee34437f05546455144f2b5d9ac09b1face62bcfce597d2e521066688", size = 225134, upload-time = "2026-08-03T21:20:58.675Z" }, + { url = "https://files.pythonhosted.org/packages/e0/9f/e8e3dfa04a1b4c241f8c91faacad872b4d4efd051d49764ad4e2fd4b9fea/cffi-2.1.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:cc572dace3f60ef98d7b12ff411d20f5362feb31a0439eab0085bbfd349982d7", size = 223197, upload-time = "2026-08-03T21:20:59.968Z" }, + { url = "https://files.pythonhosted.org/packages/f8/7e/8debeb04f1ab9fe2a6963964cd6f1aaf7192627b83926586a6a4e089c9fa/cffi-2.1.1-cp315-cp315-win32.whl", hash = "sha256:4f42141fc14250de6dde5ee7ea4432be017252d91f19c5ad043c084cea629cac", size = 177683, upload-time = "2026-08-03T21:21:14.901Z" }, + { url = "https://files.pythonhosted.org/packages/e0/31/5158704cc474ab65c1647932e88be78dc0873f47130e253be38bcaf13d01/cffi-2.1.1-cp315-cp315-win_amd64.whl", hash = "sha256:e6e8cff14d6fb0be70a09c0bdc58096f501952d04624ebf867e0e56da2df8960", size = 187897, upload-time = "2026-08-03T21:21:16.108Z" }, + { url = "https://files.pythonhosted.org/packages/cc/4b/b3a2da8570c704ffc0f9762cdc3ec0f02c8573798e0b5cf7f11c82bbb70f/cffi-2.1.1-cp315-cp315-win_arm64.whl", hash = "sha256:27350daa11d4f10c540e6e89dada4c54feb7256ad03e9a4dc075ebad7ba360d1", size = 182935, upload-time = "2026-08-03T21:21:17.271Z" }, + { url = "https://files.pythonhosted.org/packages/d0/ef/5443574510a1207e6f6bc38ba6e1f1de36cb48fef07b2728bb896a21f430/cffi-2.1.1-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:c26608d2222fb1e94487e4a387d85f13eb55d5ed725cb25a0c589ac4ee60e7bc", size = 188464, upload-time = "2026-08-03T21:21:01.163Z" }, + { url = "https://files.pythonhosted.org/packages/7e/ae/a56fa8c4686ad50e148fcbc8d3ae0d03915ff5c30d795058988c24118cef/cffi-2.1.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:4be96343e422f2dfcd12ab5c9f5aebe03f82f737c6bffeca6830b3875cb44aab", size = 188262, upload-time = "2026-08-03T21:21:02.382Z" }, + { url = "https://files.pythonhosted.org/packages/53/b2/6187f46f2912276a3ae284076109cc5c8680482f11f766ccf26db4a86427/cffi-2.1.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:937c0052c05a31ca1daf18de3158eed4dbfcb9cc107adbea227728d647be701e", size = 223779, upload-time = "2026-08-03T21:21:03.553Z" }, + { url = "https://files.pythonhosted.org/packages/8a/f6/c3ad28bd19f77047a03084424fbd4cbe997303267c14423737324be0385d/cffi-2.1.1-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:df423d40ee8654634421812bc3b196da3f9bd7d32929da813f8394c4348a5358", size = 211520, upload-time = "2026-08-03T21:21:04.863Z" }, + { url = "https://files.pythonhosted.org/packages/a0/cd/ccac9013a5bd9fd764de118674ab9c805b5ca10c19270d90ee273f8b2240/cffi-2.1.1-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a730a083190634c65cca36ba5f489531576ebd79bcd5c8e172130f6453127231", size = 210673, upload-time = "2026-08-03T21:21:06.223Z" }, + { url = "https://files.pythonhosted.org/packages/52/86/2976131c639aead931c5bee5aba67e4b09fbeb8018b6f282f70803f923a7/cffi-2.1.1-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:363e05fa78e15116c3c32c210ee36884fd6b9afa6d440e47112c3bd511d64cb6", size = 223835, upload-time = "2026-08-03T21:21:07.539Z" }, + { url = "https://files.pythonhosted.org/packages/ac/0c/33a7aeab2f9c76918c52e084beb39c570db3588133412929e8ec06fab90b/cffi-2.1.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:770de9db11e84213beec501cfcaa013b019820ca881e03344dea5844f7876d94", size = 226705, upload-time = "2026-08-03T21:21:08.774Z" }, + { url = "https://files.pythonhosted.org/packages/e3/26/2cde30fdde421130bfc18f70395731a6e6b2053c6a1978a5258ff04e72fa/cffi-2.1.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:7da0c5eff80f0197f3b3d1232ec5a682a9325f4ae9016a78f5f5ca35f9ced1f5", size = 225539, upload-time = "2026-08-03T21:21:09.911Z" }, + { url = "https://files.pythonhosted.org/packages/6d/cd/a361394c94b2129d604bb846f624a8e88255a3ee33129c434a00d715e64f/cffi-2.1.1-cp315-cp315t-win32.whl", hash = "sha256:06c72bb76605a4b0cd0aad6930b69d4baf7dd5d806cfc409b824191099700e66", size = 182707, upload-time = "2026-08-03T21:21:11.226Z" }, + { url = "https://files.pythonhosted.org/packages/9b/b5/ba2b299993c26577d529b6ae29841f9e15b9fcf004d65f423f4fcf94ade9/cffi-2.1.1-cp315-cp315t-win_amd64.whl", hash = "sha256:d9c275eaacd24aa73f94ffd6de08fc3f932424d8b6c376f4bed7cde376fe7bc3", size = 193772, upload-time = "2026-08-03T21:21:12.39Z" }, + { url = "https://files.pythonhosted.org/packages/aa/29/35e016098c814cd93de9cd320c66b5bfba14dc6ecedd3cb518fa7c408c69/cffi-2.1.1-cp315-cp315t-win_arm64.whl", hash = "sha256:d18e5ac0f2f03f4f518d3e23db0f0cad7faa1da8620e9c09461d443bbf6e6692", size = 186360, upload-time = "2026-08-03T21:21:13.636Z" }, +] + [[package]] name = "cfgv" version = "3.5.0" @@ -362,6 +508,62 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, ] +[[package]] +name = "cryptography" +version = "50.0.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cffi", marker = "platform_python_implementation != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/bb/ad/5d6702db60b1e40b41ef513b6967ff5848f307d50f8449baf1634f5908f1/cryptography-50.0.1.tar.gz", hash = "sha256:5dd9bda1c12b4162f6ff568eeb5e0ff956c28d14406e875cfe8a63a2d414ff20", size = 880381, upload-time = "2026-08-25T19:45:45.499Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ba/19/797e2aaac9df6a66f1550f49979dc1b1e39ecd2077501c30efa81e8d5d67/cryptography-50.0.1-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:b8f852c65863251b9e3a1b8c150ce21e59b522dbb6a7d4bc80e680d38388e986", size = 4010153, upload-time = "2026-08-25T19:44:03.155Z" }, + { url = "https://files.pythonhosted.org/packages/90/34/9ce9a62ed9dc82ca9fd6a34445b6904af56e5f38b3eae2ed32e49c36053d/cryptography-50.0.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:53e279950892dc102c6b4e52af03ae5ea92fac572a1ddab78ca73a997f62b69f", size = 4723133, upload-time = "2026-08-25T19:44:05.461Z" }, + { url = "https://files.pythonhosted.org/packages/57/26/e6d4fc8512a51a5f9ee7bfdbfb853bce1197087df40c9ad993ad370b846f/cryptography-50.0.1-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ff838d62ec1bfce4f9ba7fa16f4a7b554cd8d0c299e6be37502161a660c84eef", size = 4712478, upload-time = "2026-08-25T19:44:07.375Z" }, + { url = "https://files.pythonhosted.org/packages/e6/de/d3cdc2815697aae84126cbd6a030ca7b6b452e28a88b501b836bd3aa7a86/cryptography-50.0.1-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e74591e283fe6eb956416c929eb58262a719fe0311fd9054c62c3350ed8760d8", size = 4730726, upload-time = "2026-08-25T19:44:09.294Z" }, + { url = "https://files.pythonhosted.org/packages/55/32/38c0d344b98c06d34b5df8946565a9c0d6dbf32c8e0730a7f05f0a3c6cab/cryptography-50.0.1-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:5fe002589592ed749ce77fe0695fcbd3500dd61d7d6db5858a7544c612fa8e45", size = 5353524, upload-time = "2026-08-25T19:44:11.96Z" }, + { url = "https://files.pythonhosted.org/packages/e1/1b/82f0f0d8858d4432be1af790477edf62aef90324041aa07c57e57bef1af7/cryptography-50.0.1-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:51593d180cf6d179bde5c5d065bed81386b1f381656ae7d042b7ffc87a9895ad", size = 4746720, upload-time = "2026-08-25T19:44:14.051Z" }, + { url = "https://files.pythonhosted.org/packages/29/ba/042ca458b8c64348c768284b5d23e69b92ed53d057ab779fee628564676d/cryptography-50.0.1-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:359e62deae718bce96170e223fdcb6357e4fbd3bb7a3a75f4430763532560e49", size = 4361866, upload-time = "2026-08-25T19:44:16.167Z" }, + { url = "https://files.pythonhosted.org/packages/39/3b/e96c1ef71edef71057c7e3c3d982ce8fda554e0c52d0cc19c18845cde3eb/cryptography-50.0.1-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:e2ca8fd1b6b4b82a1c4cb02841d0837e3c12336c2e24b520ab8ab3b969733d8f", size = 4730028, upload-time = "2026-08-25T19:44:18.085Z" }, + { url = "https://files.pythonhosted.org/packages/e3/38/45abd72ef63f2e7d0754a6cacf97bd8b69512ace7f6130d24c39ece65da2/cryptography-50.0.1-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:76de83fbd91ac49c0feaaa983d0748fd7a53176afac5fb3bf7478d244f0eb527", size = 5308405, upload-time = "2026-08-25T19:44:20.197Z" }, + { url = "https://files.pythonhosted.org/packages/85/66/6ccca4722987ddedaa7fc9c3f4708af7431f5535666c174350830888c6b7/cryptography-50.0.1-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:51afcfceb15597cf2635068e4ac9a56b2abde622edde17f37d85fd7b5306497a", size = 4746230, upload-time = "2026-08-25T19:44:22.376Z" }, + { url = "https://files.pythonhosted.org/packages/13/0e/b1f92e013228111413f2e6743948b80bc24dfd3c1b87ba98ceea16f5df89/cryptography-50.0.1-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:be224a65493ec5b74a158ff22a5522ce4a5ca1e543c647a3a4730d4a09e5f959", size = 4862596, upload-time = "2026-08-25T19:44:24.472Z" }, + { url = "https://files.pythonhosted.org/packages/7e/22/c3654cccc856e9d682817b04ac3ee79731cb09ca6f95996a95c904de2883/cryptography-50.0.1-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9ebcdd5519be9b652a46f507817a74591774fc3d6923ac364e4dfa64e36b291b", size = 5014082, upload-time = "2026-08-25T19:44:26.709Z" }, + { url = "https://files.pythonhosted.org/packages/42/8b/cb12b1b60c91b074ca6bf0fdd59aa8f10d8bc5f73af8faece86ef0421b37/cryptography-50.0.1-cp311-abi3-win_amd64.whl", hash = "sha256:aed8db4f6d71c51efb89530e12d9464e7bf2923d46c3205dc794a2a93f8c0648", size = 3842826, upload-time = "2026-08-25T19:44:28.784Z" }, + { url = "https://files.pythonhosted.org/packages/5b/f0/424cb557d99aa86ac55da5e2add02e2882e44047b6264f93ade1b975a993/cryptography-50.0.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:30a125032e5642a21ff816e021152bd4e7e94f03eff3f4b7fca41cd22bc3110f", size = 3973525, upload-time = "2026-08-25T19:44:30.7Z" }, + { url = "https://files.pythonhosted.org/packages/4d/72/3a2711d967977ab5fc80b782837c7e8d1ac7445e764c20c381a265c57ef3/cryptography-50.0.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a0b1a59e3a089064a0ec309e9428c8e3ae4e161419d20ac33600767e83fc658a", size = 4708817, upload-time = "2026-08-25T19:44:32.773Z" }, + { url = "https://files.pythonhosted.org/packages/b4/f2/bb1f56e10815b789df0b409a69fa4992ff3d3fef9c72747f4a6b26fed38e/cryptography-50.0.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8921d58f426793c5f1b47f0b59575780de9a095214958d0eb37d909593db8367", size = 4697300, upload-time = "2026-08-25T19:44:35.144Z" }, + { url = "https://files.pythonhosted.org/packages/08/bd/ed5396be499ffcf8807a585bfe38b71a1fbdd1c342b4f9b6d0ef5162a946/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a8f40ea47330e71b594a7e246898f93177c259490c63183dbaf9e571d71ed9a5", size = 4716039, upload-time = "2026-08-25T19:44:37.192Z" }, + { url = "https://files.pythonhosted.org/packages/f6/6e/1cf405c5c8e8df7545378048e954792f00b7f2367af8863ce8b8f3e10607/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:a255449073358275b64b67d3f595f268bbef70e72b6edb65e0c70c735bf739c9", size = 5332388, upload-time = "2026-08-25T19:44:39.16Z" }, + { url = "https://files.pythonhosted.org/packages/47/92/b4317e8c32c4f47b062f5398bd79106b220a124546f42be83bf32b761e2a/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:8df2de9102026855887e4587084f6eabd80ed0f345b8ad8a7ac27ab9bf4723e0", size = 4730293, upload-time = "2026-08-25T19:44:41.298Z" }, + { url = "https://files.pythonhosted.org/packages/39/0d/a1e7633e2c744d0f2983320a27e924ef2264c79c56e1a58d5fb0a1cfd413/cryptography-50.0.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:ac02b07824d4d1001bd4367599f839c19cb171924c796e52c23508ac14c2c0cc", size = 4346031, upload-time = "2026-08-25T19:44:43.245Z" }, + { url = "https://files.pythonhosted.org/packages/88/dd/b215616f9bab3fc18510c78a4e5c9f362d77838503c363dc747c7d4f5c6f/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:cbf74a81765ee67413503ca6e26dcc4f6f5a519822436cc0a1b97aab6c1b8a17", size = 4715344, upload-time = "2026-08-25T19:44:45.291Z" }, + { url = "https://files.pythonhosted.org/packages/b1/1b/ec3ebd31741d0e963612c4fe43caa39341b9b1e031e469820e42e4c83918/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:16c5ecd954b3330ebfb6605eca4fd952da8bef376551d5cc264534e3770a9ee6", size = 5287201, upload-time = "2026-08-25T19:44:47.297Z" }, + { url = "https://files.pythonhosted.org/packages/1a/01/0127d11a762b31a9ee0221894f540318761783f3fdc4bc5d057698caebd5/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:79bf008d1f9af6071c797ad133e39915dfee7614f18f18f4db9072eb715064a3", size = 4730023, upload-time = "2026-08-25T19:44:49.435Z" }, + { url = "https://files.pythonhosted.org/packages/9e/b9/e7425ebfb599241a0c1d7000f1b466c3062da66c19d9525031315dff7213/cryptography-50.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:330fbb252391c596f1ae42c5754449dc924e6ad012dca8efe0d703f9f2d12ec6", size = 4847362, upload-time = "2026-08-25T19:44:51.94Z" }, + { url = "https://files.pythonhosted.org/packages/2d/fd/60d0ddf4defa12e482c9d5e0f554384d6e8ab25341fd15f060028fd92e6a/cryptography-50.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:42be3bb70596b3abe4ac097b75be223e8b3ab614a0e5de068e3dcc54d71d6149", size = 4999247, upload-time = "2026-08-25T19:44:53.876Z" }, + { url = "https://files.pythonhosted.org/packages/4d/56/bc4f2b209e766c93372cfcd59b781a0b2b59700f62a969580415b699c2b2/cryptography-50.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:f74455bb086a85d5e81246412602aaa97ed095e504cd40dd261ef50be42205bf", size = 3825806, upload-time = "2026-08-25T19:44:56.209Z" }, + { url = "https://files.pythonhosted.org/packages/84/a9/ee16a903f13755e914d1eecc482fe64d1f10761c3960e5d8fa6837377aff/cryptography-50.0.1-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ca83d00d9e69cd5eb63f2e69c3a5a59e0cecae5ae14c6ae0b35830fe3b37bad0", size = 4035307, upload-time = "2026-08-25T19:44:58.305Z" }, + { url = "https://files.pythonhosted.org/packages/5e/a5/9ec7e81e8526c0d7a387d73386b2daed3f39e10d81a85930bd1b6bfba65c/cryptography-50.0.1-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:05ba322c4da95b262a212c345af888ef2c37c88c0509756ea00a0e6d68850f23", size = 4751900, upload-time = "2026-08-25T19:45:00.401Z" }, + { url = "https://files.pythonhosted.org/packages/7e/3c/0e77bd5ffcf078e9dd27d3074aad6c030d9b10d0bf69329d573c927a188c/cryptography-50.0.1-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e22dfed744bd4002e909464cb23d2f0b05c6f3113a79ef2e9864a53db737c733", size = 4738357, upload-time = "2026-08-25T19:45:02.786Z" }, + { url = "https://files.pythonhosted.org/packages/27/3a/3c5f80daa4dcd47323c7af8a2fcb90de27a33564d4fcac69846c0972691a/cryptography-50.0.1-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:4c4188f7c0cf655be5c06342b817ed0f9595b69ffa2b12026e5353eed29dea88", size = 4758474, upload-time = "2026-08-25T19:45:04.889Z" }, + { url = "https://files.pythonhosted.org/packages/6e/2b/214cf0cf93db9628c3c20c896b229f327f6fb1b20e4b3743d8ad3f00af8b/cryptography-50.0.1-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:2ebbfb0f1fed745e91796e3e1080a1440423fdae8ece1b995a1d80883a409054", size = 5375862, upload-time = "2026-08-25T19:45:07.163Z" }, + { url = "https://files.pythonhosted.org/packages/d6/51/3f9701867a46b6c1740c9b52fc4d3bed6cbdcfedcc9b6e64305c07f39cff/cryptography-50.0.1-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:407fe2b6db00939c05c0e945e9914238f2f0a430974839429dafc82b1ee6bee5", size = 4772942, upload-time = "2026-08-25T19:45:09.396Z" }, + { url = "https://files.pythonhosted.org/packages/0d/5c/13ea642e08e2544d0f5396122055f4820cfacb3203562197b5967125ea97/cryptography-50.0.1-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:2b34d76a652ea2b6faf777c35df230c5637842cd904e04f16230c3f9f03e4361", size = 4383347, upload-time = "2026-08-25T19:45:11.659Z" }, + { url = "https://files.pythonhosted.org/packages/84/d5/7d1fe1cb93f91c428093ff234e128c89ba8ea61a6f26aab406081f9b996e/cryptography-50.0.1-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:01f41478cf33fc605a6a089cd56d28b45c6c0b45a1928b61797f2621a04bac71", size = 4758050, upload-time = "2026-08-25T19:45:13.745Z" }, + { url = "https://files.pythonhosted.org/packages/dd/04/557fc5ead96a829e0bc812a3b9dc4a52a2f27e4f7f5950da7ff27653a805/cryptography-50.0.1-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:fc3ed7ebd2a8c96f5b166de0ab9b624996bef3b07bbeb19364dfb78222c22c80", size = 5332955, upload-time = "2026-08-25T19:45:16.193Z" }, + { url = "https://files.pythonhosted.org/packages/8c/eb/5d7124083e8d8cda8f5b348f544b71ad6f707ad63193758ef4d8e569da02/cryptography-50.0.1-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:9dde0a357190eb3b1da1bb9ab750e9c85cba82ca5977aa0836cbb94e92611239", size = 4772694, upload-time = "2026-08-25T19:45:18.315Z" }, + { url = "https://files.pythonhosted.org/packages/63/8e/f1f955e0921dd2b6d22eae7e8d24a4c4b638d10735ffbf6a71f99eb0fcb8/cryptography-50.0.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd3718b960d0b5dd213cdf03f3bcb7000e69dda0de8b956061947ff6bcff5558", size = 4888413, upload-time = "2026-08-25T19:45:20.4Z" }, + { url = "https://files.pythonhosted.org/packages/1f/ab/89e2b798d2c3925f82e2bb72d5979f3d2f6da2dd22ef4a8cd8b70d920039/cryptography-50.0.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:2a93d05e34d5f67fba6f891fe85d929999baa7195e853923ea6d7576c9e68c5e", size = 5044355, upload-time = "2026-08-25T19:45:22.353Z" }, + { url = "https://files.pythonhosted.org/packages/99/89/87ef49ffe383ef4e147d27b7bf2088fb0b54ea409dd87b5a89442e5828a5/cryptography-50.0.1-cp39-abi3-win_amd64.whl", hash = "sha256:55d16b1ef3ee0958d893a977b19777887e546c9954ea81b200c3301a864013f2", size = 3875429, upload-time = "2026-08-25T19:45:24.418Z" }, + { url = "https://files.pythonhosted.org/packages/c7/27/8d207af749c453ee17ea087340b3f2b4adef75aadd1d277b1b129bdda84e/cryptography-50.0.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:9cb3cb952cf5a8abd50c782a98a89d71699715e802fe349704b47f2425b42a94", size = 3974350, upload-time = "2026-08-25T19:45:26.551Z" }, + { url = "https://files.pythonhosted.org/packages/14/9a/6d3a4d7852e22d657438b7bf51f66102c7d71c0e1fafeec652281d0403e5/cryptography-50.0.1-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:5fe939deeb161024a6be98229c953b6591fef1f41214497a78fe793a244c017f", size = 4698675, upload-time = "2026-08-25T19:45:28.658Z" }, + { url = "https://files.pythonhosted.org/packages/73/35/5c3717edf9e68a0550ce04e28eab493fe545eccd81742af03f6a75fe260b/cryptography-50.0.1-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:fb4b9672d389c738b175c4166e78310f8a70358886aacd9173ee03a85ffdc671", size = 4707410, upload-time = "2026-08-25T19:45:30.816Z" }, + { url = "https://files.pythonhosted.org/packages/1d/e0/e786934472e3ac4ecdecc7b129a0ca1a2a40dffdafcf2c3ea9d4397f8def/cryptography-50.0.1-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:d63ae8f6481fec907ac0f588eee8a90aefde112c633131fe540e5711ddbb5a4e", size = 4698378, upload-time = "2026-08-25T19:45:33.043Z" }, + { url = "https://files.pythonhosted.org/packages/51/cf/5b3f53a0b74d122f023476ede40ba5d3e70d5cf475f73b899740d26a4fb2/cryptography-50.0.1-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:804728ce710890870f3aaa344b2e161172d258d768ac139d02cfd9092d0d94e6", size = 4706889, upload-time = "2026-08-25T19:45:35.086Z" }, + { url = "https://files.pythonhosted.org/packages/71/44/711e61f7d014be825ef79b285b047292d1bf893732ac1bc030a351fb517f/cryptography-50.0.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:693c99b49bd37d0d096e4334c10232c77248c415b98d35236094cdf96d57258b", size = 3824006, upload-time = "2026-08-25T19:45:37.281Z" }, +] + [[package]] name = "cuda-bindings" version = "12.9.4" @@ -1692,6 +1894,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" }, ] +[[package]] +name = "pycparser" +version = "3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" }, +] + [[package]] name = "pycryptodome" version = "3.23.0" @@ -1843,6 +2054,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" }, ] +[[package]] +name = "pyopenssl" +version = "26.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cryptography" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3f/e8/7325d258199b159eb2c03fe32107533e2832e70e63f4fb88a6aa00023201/pyopenssl-26.4.0.tar.gz", hash = "sha256:28dfcce0162b9211413e26dfbfdf1d24317fbeba18fc93c12400a1856b2a0bc7", size = 182046, upload-time = "2026-08-01T19:50:50.512Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/ad/2cf6d3fa2fae5c79e1ed9960c0d42badd0f94d81dd12b50604cdc839e648/pyopenssl-26.4.0-py3-none-any.whl", hash = "sha256:f0eb0cb2d581d3ad2b9c489468485e7f2ab6727d08401bcf9d824c3caddf3c1c", size = 56026, upload-time = "2026-08-01T19:50:48.94Z" }, +] + [[package]] name = "pytest" version = "9.0.2" @@ -2375,23 +2599,23 @@ resolution-markers = [ "python_full_version < '3.12'", ] dependencies = [ - { name = "alabaster", marker = "python_full_version < '3.12'" }, - { name = "babel", marker = "python_full_version < '3.12'" }, - { name = "colorama", marker = "python_full_version < '3.12' and sys_platform == 'win32'" }, - { name = "docutils", marker = "python_full_version < '3.12'" }, - { name = "imagesize", marker = "python_full_version < '3.12'" }, - { name = "jinja2", marker = "python_full_version < '3.12'" }, - { name = "packaging", marker = "python_full_version < '3.12'" }, - { name = "pygments", marker = "python_full_version < '3.12'" }, - { name = "requests", marker = "python_full_version < '3.12'" }, - { name = "roman-numerals", marker = "python_full_version < '3.12'" }, - { name = "snowballstemmer", marker = "python_full_version < '3.12'" }, - { name = "sphinxcontrib-applehelp", marker = "python_full_version < '3.12'" }, - { name = "sphinxcontrib-devhelp", marker = "python_full_version < '3.12'" }, - { name = "sphinxcontrib-htmlhelp", marker = "python_full_version < '3.12'" }, - { name = "sphinxcontrib-jsmath", marker = "python_full_version < '3.12'" }, - { name = "sphinxcontrib-qthelp", marker = "python_full_version < '3.12'" }, - { name = "sphinxcontrib-serializinghtml", marker = "python_full_version < '3.12'" }, + { name = "alabaster" }, + { name = "babel" }, + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "docutils" }, + { name = "imagesize" }, + { name = "jinja2" }, + { name = "packaging" }, + { name = "pygments" }, + { name = "requests" }, + { name = "roman-numerals" }, + { name = "snowballstemmer" }, + { name = "sphinxcontrib-applehelp" }, + { name = "sphinxcontrib-devhelp" }, + { name = "sphinxcontrib-htmlhelp" }, + { name = "sphinxcontrib-jsmath" }, + { name = "sphinxcontrib-qthelp" }, + { name = "sphinxcontrib-serializinghtml" }, ] sdist = { url = "https://files.pythonhosted.org/packages/42/50/a8c6ccc36d5eacdfd7913ddccd15a9cee03ecafc5ee2bc40e1f168d85022/sphinx-9.0.4.tar.gz", hash = "sha256:594ef59d042972abbc581d8baa577404abe4e6c3b04ef61bd7fc2acbd51f3fa3", size = 8710502, upload-time = "2025-12-04T07:45:27.343Z" } wheels = [ @@ -2406,23 +2630,23 @@ resolution-markers = [ "python_full_version >= '3.12'", ] dependencies = [ - { name = "alabaster", marker = "python_full_version >= '3.12'" }, - { name = "babel", marker = "python_full_version >= '3.12'" }, - { name = "colorama", marker = "python_full_version >= '3.12' and sys_platform == 'win32'" }, - { name = "docutils", marker = "python_full_version >= '3.12'" }, - { name = "imagesize", marker = "python_full_version >= '3.12'" }, - { name = "jinja2", marker = "python_full_version >= '3.12'" }, - { name = "packaging", marker = "python_full_version >= '3.12'" }, - { name = "pygments", marker = "python_full_version >= '3.12'" }, - { name = "requests", marker = "python_full_version >= '3.12'" }, - { name = "roman-numerals", marker = "python_full_version >= '3.12'" }, - { name = "snowballstemmer", marker = "python_full_version >= '3.12'" }, - { name = "sphinxcontrib-applehelp", marker = "python_full_version >= '3.12'" }, - { name = "sphinxcontrib-devhelp", marker = "python_full_version >= '3.12'" }, - { name = "sphinxcontrib-htmlhelp", marker = "python_full_version >= '3.12'" }, - { name = "sphinxcontrib-jsmath", marker = "python_full_version >= '3.12'" }, - { name = "sphinxcontrib-qthelp", marker = "python_full_version >= '3.12'" }, - { name = "sphinxcontrib-serializinghtml", marker = "python_full_version >= '3.12'" }, + { name = "alabaster" }, + { name = "babel" }, + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "docutils" }, + { name = "imagesize" }, + { name = "jinja2" }, + { name = "packaging" }, + { name = "pygments" }, + { name = "requests" }, + { name = "roman-numerals" }, + { name = "snowballstemmer" }, + { name = "sphinxcontrib-applehelp" }, + { name = "sphinxcontrib-devhelp" }, + { name = "sphinxcontrib-htmlhelp" }, + { name = "sphinxcontrib-jsmath" }, + { name = "sphinxcontrib-qthelp" }, + { name = "sphinxcontrib-serializinghtml" }, ] sdist = { url = "https://files.pythonhosted.org/packages/cd/bd/f08eb0f4eed5c83f1ba2a3bd18f7745a2b1525fad70660a1c00224ec468a/sphinx-9.1.0.tar.gz", hash = "sha256:7741722357dd75f8190766926071fed3bdc211c74dd2d7d4df5404da95930ddb", size = 8718324, upload-time = "2025-12-31T15:09:27.646Z" } wheels = [ @@ -2779,6 +3003,11 @@ dependencies = [ ] [package.optional-dependencies] +bfv-nitro = [ + { name = "cbor2" }, + { name = "cryptography" }, + { name = "pyopenssl" }, +] cli = [ { name = "prompt-toolkit" }, { name = "python-dotenv" }, @@ -2816,13 +3045,16 @@ dev = [ requires-dist = [ { name = "aiohttp", specifier = ">=3.12.14" }, { name = "boto3", specifier = ">=1.39.4" }, + { name = "cbor2", marker = "extra == 'bfv-nitro'", specifier = "==6.1.4" }, { name = "certifi", specifier = ">=2024.2.2" }, + { name = "cryptography", marker = "extra == 'bfv-nitro'", specifier = "==50.0.1" }, { name = "gmpy2", specifier = ">=2.2.1" }, { name = "msgpack" }, { name = "numpy", specifier = ">=2.3.1" }, { name = "openai", specifier = ">=1.96.0" }, { name = "prompt-toolkit", marker = "extra == 'cli'", specifier = ">=3.0.0" }, { name = "pycryptodome", specifier = ">=3.23.0" }, + { name = "pyopenssl", marker = "extra == 'bfv-nitro'", specifier = "==26.4.0" }, { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.4.1" }, { name = "python-dotenv", specifier = ">=1.1.1" }, { name = "python-dotenv", marker = "extra == 'cli'", specifier = ">=1.1.1" }, @@ -2837,7 +3069,7 @@ requires-dist = [ { name = "typer", marker = "extra == 'cli'", specifier = ">=0.12.0" }, { name = "typing-extensions", specifier = ">=4.14.1" }, ] -provides-extras = ["cli", "dev", "embedding"] +provides-extras = ["bfv-nitro", "cli", "dev", "embedding"] [package.metadata.requires-dev] dev = [ From 241413d95d0ce289d1c8d13c41ae85197167bcd2 Mon Sep 17 00:00:00 2001 From: Yavor Litchev Date: Thu, 17 Sep 2026 15:18:13 -0400 Subject: [PATCH 12/12] Benchmark packed BFV against Paillier on 8192 vectors --- benchmarks/bfv_client_matrix.py | 25 ++- benchmarks/results/.gitignore | 3 + benchmarks/results/native_bfv_8192.json | 210 ++++++++++++++++++ .../results/native_bfv_review_8192.json | 145 ++++++++++++ docs/research/native-bfv.md | 95 ++++++++ 5 files changed, 474 insertions(+), 4 deletions(-) create mode 100644 benchmarks/results/native_bfv_8192.json create mode 100644 benchmarks/results/native_bfv_review_8192.json diff --git a/benchmarks/bfv_client_matrix.py b/benchmarks/bfv_client_matrix.py index a4eec1d..3d33c1a 100644 --- a/benchmarks/bfv_client_matrix.py +++ b/benchmarks/bfv_client_matrix.py @@ -30,7 +30,7 @@ sys.path.insert(0, str(REPO_ROOT / "src")) from xtrace_sdk.x_vec.crypto.bfv_client import BFVClient # noqa: E402 -from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV # noqa: E402 +from xtrace_sdk.x_vec.crypto.encryption.bfv import BFV, MAX_PUBLIC_KEY_CHARS # noqa: E402 from xtrace_sdk.x_vec.crypto.encryption.paillier import Paillier # noqa: E402 from xtrace_sdk.x_vec.crypto.paillier_client import PaillierClient # noqa: E402 from xtrace_sdk.x_vec.crypto.paillier_lookup_client import PaillierLookupClient # noqa: E402 @@ -126,7 +126,9 @@ def benchmark( def load_server() -> BFVClient: result = BFVClient(skip_key_gen=True, server_backend=args.server_backend) result.load_config(config) - result.load_stringified_keys(public_json) + result.load_stringified_keys( + public_json, max_public_key_chars=args.max_public_key_chars + ) assert result.keys is None return result @@ -236,6 +238,12 @@ def main() -> None: parser.add_argument("--coeff-modulus-bits", type=int, default=180) parser.add_argument("--decomposition-bits", type=int, default=30) parser.add_argument("--response-modulus-bits", type=int, default=50) + parser.add_argument( + "--max-public-key-chars", + type=int, + default=MAX_PUBLIC_KEY_CHARS, + help="Local JSON import bound; use 268435456 for the N=16384 review profile", + ) parser.add_argument("--no-compact", action="store_true") parser.add_argument( "--server-backend", @@ -264,8 +272,13 @@ def main() -> None: if args.server_backend == "residue" and not args.rns_modulus: parser.error("residue requires --rns-modulus") variants = [v.strip() for v in args.variants.split(",")] - if args.num_vectors < 1 or args.embed_len < 1 or args.repeats < 1: - parser.error("num-vectors, embed-len and repeats must be positive") + if ( + args.num_vectors < 1 + or args.embed_len < 1 + or args.repeats < 1 + or args.max_public_key_chars < 1 + ): + parser.error("num-vectors, embed-len, repeats and max-public-key-chars must be positive") if any(v not in VARIANTS for v in variants): parser.error("unknown variant") vectors, query, expected = make_data(args.num_vectors, args.embed_len, args.seed) @@ -301,6 +314,10 @@ def main() -> None: Path(__file__).resolve(), *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto").glob("*client.py"), *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/encryption").glob("*.py"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.h"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.cpp"), + *(REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext").glob("*.so"), + REPO_ROOT / "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile", REPO_ROOT / "src/xtrace_sdk/x_vec/utils/xtrace_types.py", ] }, diff --git a/benchmarks/results/.gitignore b/benchmarks/results/.gitignore index ecf5b70..0c7792e 100644 --- a/benchmarks/results/.gitignore +++ b/benchmarks/results/.gitignore @@ -1,3 +1,6 @@ # The Nitro protocol evidence is intended to accompany its implementation. !native_bfv_nitro_local_1024.json !native_bfv_nitro_audit_1024.json +# Paired 8,192-vector I/O measurements, with both BFV parameter profiles. +!native_bfv_8192.json +!native_bfv_review_8192.json diff --git a/benchmarks/results/native_bfv_8192.json b/benchmarks/results/native_bfv_8192.json new file mode 100644 index 0000000..9500a67 --- /dev/null +++ b/benchmarks/results/native_bfv_8192.json @@ -0,0 +1,210 @@ +{ + "environment": { + "utc": "2026-09-17T19:10:37.064659+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "be71b0dd4fa3693f6a6628e61f31e4b54ece9760", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_client_matrix.py", + "--num-vectors", + "8192", + "--embed-len", + "512", + "--poly-modulus-degree", + "8192", + "--server-backend", + "residue", + "--rns-modulus", + "--variants", + "bfv-packed,paillier-cpu", + "--repeats", + "1", + "--json-out", + "benchmarks/results/native_bfv_8192.json" + ], + "measurement_notes": "CPU synthetic experiment; no equivalent-security claim. Keys are public JSON including BFV evaluation keys. Index/query/response use common MessagePack framing with LE integer bytes and implicit IDs. No network/HTTP/TLS, secret-key, plaintext-content or AES-content bytes. Timings are separate from serialization; no concurrent test/benchmark processes intended.", + "seed": 1337, + "runs": [ + { + "variant": "bfv-packed", + "server_backend": "residue", + "vectors": 8192, + "embed_len": 512, + "config": { + "embed_len": 512, + "poly_modulus_degree": 8192, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "compact": true, + "timings": { + "key_generation_s": 4.473671418149024, + "public_key_export_s": 0.7987699322402477, + "public_key_import_s": 1.4380085780285299, + "index_encrypt_s": 54.573941993992776, + "query_encrypt_s": 0.10487454803660512, + "index_serialize_s": 0.38077715784311295, + "query_serialize_s": 0.0003552851267158985, + "index_deserialize_s": 0.2512285150587559, + "query_deserialize_s": 0.0002061929553747177, + "server_compute_s": 19.836358953267336, + "response_serialize_s": 8.567096665501595e-05, + "response_deserialize_s": 7.523223757743835e-05, + "client_decode_s": 0.04191199177876115 + }, + "sizes": { + "encrypted_index_bytes": 188787740, + "public_keys_bytes": 85603562, + "query_bytes": 368754, + "response_bytes": 102496, + "query_plus_response_bytes": 471250 + }, + "index_ciphertexts": 512, + "response_ciphertexts": 1, + "minimum_response_noise_budget_bits": 26, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 8191, + "distance": 0 + }, + { + "index": 2654, + "distance": 217 + } + ], + "correct": true, + "repeat": 0 + }, + { + "variant": "paillier-cpu", + "server_backend": null, + "vectors": 8192, + "embed_len": 512, + "config": { + "embed_len": 512, + "key_len": 1024 + }, + "compact": false, + "timings": { + "key_generation_s": 0.22122686309739947, + "public_key_export_s": 4.7971028834581375e-05, + "public_key_import_s": 2.322997897863388e-05, + "index_encrypt_s": 86.23786324588582, + "query_encrypt_s": 0.010293608997017145, + "index_serialize_s": 0.012443804182112217, + "query_serialize_s": 3.6510173231363297e-05, + "index_deserialize_s": 0.004360671620815992, + "query_deserialize_s": 2.3310072720050812e-05, + "server_compute_s": 0.043896660674363375, + "response_serialize_s": 0.005283337086439133, + "response_deserialize_s": 0.00477636931464076, + "client_decode_s": 57.76321804802865 + }, + "sizes": { + "encrypted_index_bytes": 4226979, + "public_keys_bytes": 2502, + "query_bytes": 544, + "response_bytes": 4226979, + "query_plus_response_bytes": 4227523 + }, + "index_ciphertexts": 8192, + "response_ciphertexts": 8192, + "minimum_response_noise_budget_bits": null, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 8191, + "distance": 0 + }, + { + "index": 2654, + "distance": 217 + } + ], + "correct": true, + "repeat": 0 + } + ], + "source_sha256": { + "benchmarks/bfv_client_matrix.py": "2f93923f21f3b9165ddc510ce029d61e8df873c7b519c1d61a99a193697a2f38", + "src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py": "d3bed7a596fd19f834ea308fc332f0a28109a1136c7dd63026819b4cc106c7de", + "src/xtrace_sdk/x_vec/crypto/paillier_client.py": "40582d7acb86ee7dd2060e2d795c76829f4879db3b1eca9518be7a9599de4788", + "src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py": "2e684a8a7f2364a1134aa0271ee34fc6e2562367f26ae453f61a03fa80e31db3", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "6a384b16c347d84f90e40ccc9903b1ae916ae277e6171bd668181366d56e88a0", + "src/xtrace_sdk/x_vec/crypto/goldwasser_micali_client.py": "0982ad7532d07c8e26a701359ce8f4f7d4c1a3beb714050e741f3b534d59935f", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_client.py": "4faa0dd71d03b3e9bc9e49c4e60e8be7b173ff514b52cf1d94a3b1a2426bf699", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier_lookup.py": "eed0be7e0e2c1f6e8e8c0138262ec071fef73648e334a948ef1f82df6e9b9712", + "src/xtrace_sdk/x_vec/crypto/encryption/__init__.py": "261a50ed1758cee38c880cc625187f8acb051c6d5c96ba7a71ee956bdbea2cc5", + "src/xtrace_sdk/x_vec/crypto/encryption/goldwasser_micali.py": "4b32296c6210663c098718fa49bd810c427b1f8102d026d12a7d734f58e8e3d1", + "src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py": "915da9e500d0a142651243bb13de7631dc96103fea2ec9e5b9288bee498010e7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier.py": "525a1944caef38d5d216ae84fcac982627d2a2fddea50b747e47ef05e2e91c9f", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py": "b1291e6d017be84638b054fb84e651734a44b2d54d601eb50dae2e9d2016bdfa", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/encryption/aes.py": "2c833351db3b249c381e315307d858791ee4e5b633d3465127cad763b4d095d5", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h": "66aceec7cc8452dabdde0061d347195e49a9500fba7bcbbe805c967b7171d767", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp": "7b48d92e526147d97d9401e868c8d3e9dbf0b8ba567a67e05047e01f2a371fc0", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.cpython-312-x86_64-linux-gnu.so": "c55be18d74e56a1d6bb82c9fa31f55cb9ecf568aecf191a26278a660502b6c72", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "ee3f7371f92ba9fa3ebb8037906fc23c5f7832f96908d5879f1a4326847ffb53", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962" + }, + "median_timings": { + "bfv-packed": { + "key_generation_s": 4.473671418149024, + "public_key_export_s": 0.7987699322402477, + "public_key_import_s": 1.4380085780285299, + "index_encrypt_s": 54.573941993992776, + "query_encrypt_s": 0.10487454803660512, + "index_serialize_s": 0.38077715784311295, + "query_serialize_s": 0.0003552851267158985, + "index_deserialize_s": 0.2512285150587559, + "query_deserialize_s": 0.0002061929553747177, + "server_compute_s": 19.836358953267336, + "response_serialize_s": 8.567096665501595e-05, + "response_deserialize_s": 7.523223757743835e-05, + "client_decode_s": 0.04191199177876115 + }, + "paillier-cpu": { + "key_generation_s": 0.22122686309739947, + "public_key_export_s": 4.7971028834581375e-05, + "public_key_import_s": 2.322997897863388e-05, + "index_encrypt_s": 86.23786324588582, + "query_encrypt_s": 0.010293608997017145, + "index_serialize_s": 0.012443804182112217, + "query_serialize_s": 3.6510173231363297e-05, + "index_deserialize_s": 0.004360671620815992, + "query_deserialize_s": 2.3310072720050812e-05, + "server_compute_s": 0.043896660674363375, + "response_serialize_s": 0.005283337086439133, + "response_deserialize_s": 0.00477636931464076, + "client_decode_s": 57.76321804802865 + } + } +} diff --git a/benchmarks/results/native_bfv_review_8192.json b/benchmarks/results/native_bfv_review_8192.json new file mode 100644 index 0000000..1368861 --- /dev/null +++ b/benchmarks/results/native_bfv_review_8192.json @@ -0,0 +1,145 @@ +{ + "environment": { + "utc": "2026-09-17T19:15:01.868692+00:00", + "python": "3.12.3", + "platform": "Linux-7.0.0-30-generic-x86_64-with-glibc2.39", + "gmpy2": "2.3.0", + "gmp": "GMP 6.3.0", + "revision": "be71b0dd4fa3693f6a6628e61f31e4b54ece9760", + "working_tree_dirty": true + }, + "command": [ + "benchmarks/bfv_client_matrix.py", + "--num-vectors", + "8192", + "--embed-len", + "512", + "--poly-modulus-degree", + "16384", + "--server-backend", + "residue", + "--rns-modulus", + "--max-public-key-chars", + "268435456", + "--variants", + "bfv-packed", + "--repeats", + "1", + "--json-out", + "benchmarks/results/native_bfv_review_8192.json" + ], + "measurement_notes": "CPU synthetic experiment; no equivalent-security claim. Keys are public JSON including BFV evaluation keys. Index/query/response use common MessagePack framing with LE integer bytes and implicit IDs. No network/HTTP/TLS, secret-key, plaintext-content or AES-content bytes. Timings are separate from serialization; no concurrent test/benchmark processes intended.", + "seed": 1337, + "runs": [ + { + "variant": "bfv-packed", + "server_backend": "residue", + "vectors": 8192, + "embed_len": 512, + "config": { + "embed_len": 512, + "poly_modulus_degree": 16384, + "plain_modulus": 65537, + "coeff_modulus_bits": 180, + "decomposition_bits": 30, + "error_eta": 21, + "rns_modulus": true, + "response_modulus_bits": 50, + "device": "cpu" + }, + "compact": true, + "timings": { + "key_generation_s": 9.108531635720283, + "public_key_export_s": 1.587051787879318, + "public_key_import_s": 2.978017693851143, + "index_encrypt_s": 56.27537882234901, + "query_encrypt_s": 0.2093489640392363, + "index_serialize_s": 0.4128873459994793, + "query_serialize_s": 0.0010858788155019283, + "index_deserialize_s": 0.2603874821215868, + "query_deserialize_s": 0.00039217714220285416, + "server_compute_s": 20.878492071758956, + "response_serialize_s": 0.00013659195974469185, + "response_deserialize_s": 0.00012690294533967972, + "client_decode_s": 0.0861029070802033 + }, + "sizes": { + "encrypted_index_bytes": 188765728, + "public_keys_bytes": 171204881, + "query_bytes": 737394, + "response_bytes": 204900, + "query_plus_response_bytes": 942294 + }, + "index_ciphertexts": 256, + "response_ciphertexts": 1, + "minimum_response_noise_budget_bits": 25, + "top3": [ + { + "index": 0, + "distance": 0 + }, + { + "index": 8191, + "distance": 0 + }, + { + "index": 2654, + "distance": 217 + } + ], + "correct": true, + "repeat": 0 + } + ], + "source_sha256": { + "benchmarks/bfv_client_matrix.py": "2f93923f21f3b9165ddc510ce029d61e8df873c7b519c1d61a99a193697a2f38", + "src/xtrace_sdk/x_vec/crypto/bfv_attested_client.py": "d3bed7a596fd19f834ea308fc332f0a28109a1136c7dd63026819b4cc106c7de", + "src/xtrace_sdk/x_vec/crypto/paillier_client.py": "40582d7acb86ee7dd2060e2d795c76829f4879db3b1eca9518be7a9599de4788", + "src/xtrace_sdk/x_vec/crypto/bfv_guarded_client.py": "2e684a8a7f2364a1134aa0271ee34fc6e2562367f26ae453f61a03fa80e31db3", + "src/xtrace_sdk/x_vec/crypto/bfv_client.py": "becd81b9c41c027fe26de3e675adfba32fae7c0e6b0e1bede73f96a14effc775", + "src/xtrace_sdk/x_vec/crypto/bfv_verified_client.py": "6a384b16c347d84f90e40ccc9903b1ae916ae277e6171bd668181366d56e88a0", + "src/xtrace_sdk/x_vec/crypto/goldwasser_micali_client.py": "0982ad7532d07c8e26a701359ce8f4f7d4c1a3beb714050e741f3b534d59935f", + "src/xtrace_sdk/x_vec/crypto/paillier_lookup_client.py": "4faa0dd71d03b3e9bc9e49c4e60e8be7b173ff514b52cf1d94a3b1a2426bf699", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv.py": "51eee672c2d0514146f97c9ddf33c7f4cfb99114a2b7d86654ec7de7422ebca7", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier_lookup.py": "eed0be7e0e2c1f6e8e8c0138262ec071fef73648e334a948ef1f82df6e9b9712", + "src/xtrace_sdk/x_vec/crypto/encryption/__init__.py": "261a50ed1758cee38c880cc625187f8acb051c6d5c96ba7a71ee956bdbea2cc5", + "src/xtrace_sdk/x_vec/crypto/encryption/goldwasser_micali.py": "4b32296c6210663c098718fa49bd810c427b1f8102d026d12a7d734f58e8e3d1", + "src/xtrace_sdk/x_vec/crypto/encryption/homomorphic_base.py": "915da9e500d0a142651243bb13de7631dc96103fea2ec9e5b9288bee498010e7", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_native.py": "13b0028e889351ade4d9dff2f5605e969c015cebe890aa80fd5489593d4ef0cf", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_rns.py": "74663e4b22d2fc0128ad2572cb10f9d3727b404fc0240661b08d308ba57cad40", + "src/xtrace_sdk/x_vec/crypto/encryption/paillier.py": "525a1944caef38d5d216ae84fcac982627d2a2fddea50b747e47ef05e2e91c9f", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_private.py": "b1291e6d017be84638b054fb84e651734a44b2d54d601eb50dae2e9d2016bdfa", + "src/xtrace_sdk/x_vec/crypto/encryption/bfv_evaluator.py": "895fc201b641a1ea5242b0c54a001d9402afcfb4d21668ca33a8b9992b4132dc", + "src/xtrace_sdk/x_vec/crypto/encryption/aes.py": "2c833351db3b249c381e315307d858791ee4e5b633d3465127cad763b4d095d5", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/profile.h": "3a98bb8c322dec75d1c3ed2531767ac1f5b43a1d7e79a9927d47052d43541252", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_residue.h": "e77f270c620d432370e8204c24e1dc6f3c0d382d6bd777ad63852dd8a89f575a", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bfv_server.h": "70f96bc7f1fc25827979cea7b350edaf9cffccf2fa5af0a9275b9d514fb400cf", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_scale.h": "3f191dc1532afe2e8774279d251d3374b34dc8b40b4e8a97d3dc341959ab89d1", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/packed_wire.h": "0fe06f2c6cb333ef90cf0b66847815dbf521e81be881ffd55aff21009cde42fd", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_decoder.h": "66aceec7cc8452dabdde0061d347195e49a9500fba7bcbbe805c967b7171d767", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/rns_ntt.h": "5f46b7490d556a2f314dbb5cda6f022cacac6f952f3eda48640e068f352e3796", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/private_bindings.cpp": "7b48d92e526147d97d9401e868c8d3e9dbf0b8ba567a67e05047e01f2a371fc0", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/bindings.cpp": "da003cf5e6553f85fa402c943a762504d661fb16657d25b16116f3f667509cae", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_rns.cpython-312-x86_64-linux-gnu.so": "539e19f0591c9dd375a7cf519569c4777067fb6593f5157d87b7edb37a42eb04", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/_bfv_private.cpython-312-x86_64-linux-gnu.so": "c55be18d74e56a1d6bb82c9fa31f55cb9ecf568aecf191a26278a660502b6c72", + "src/xtrace_sdk/x_vec/crypto/bfv_cpu_ext/Makefile": "ee3f7371f92ba9fa3ebb8037906fc23c5f7832f96908d5879f1a4326847ffb53", + "src/xtrace_sdk/x_vec/utils/xtrace_types.py": "3d4daa19d5d9d536573b2e2c65aa13d9559a4bc5e6ae01e2b5ffdeeac8bae962" + }, + "median_timings": { + "bfv-packed": { + "key_generation_s": 9.108531635720283, + "public_key_export_s": 1.587051787879318, + "public_key_import_s": 2.978017693851143, + "index_encrypt_s": 56.27537882234901, + "query_encrypt_s": 0.2093489640392363, + "index_serialize_s": 0.4128873459994793, + "query_serialize_s": 0.0010858788155019283, + "index_deserialize_s": 0.2603874821215868, + "query_deserialize_s": 0.00039217714220285416, + "server_compute_s": 20.878492071758956, + "response_serialize_s": 0.00013659195974469185, + "response_deserialize_s": 0.00012690294533967972, + "client_decode_s": 0.0861029070802033 + } + } +} diff --git a/docs/research/native-bfv.md b/docs/research/native-bfv.md index bd22479..02bb0c4 100644 --- a/docs/research/native-bfv.md +++ b/docs/research/native-bfv.md @@ -423,3 +423,98 @@ tests and the optional SEAL oracle. Ruff and mypy passed for the SDK and BFV benchmark files. The live service and CUDA tests remain excluded from this CPU change. An offline Sphinx build retains the preexisting README cross-reference warning in the earlier SEAL research report. + +### 8,192-vector comparison (2026-09-17) + +There is no 1,024-vector limit in the homemade BFV client. That was the earlier +development workload; packed evaluation processes additional index tiles and +returns `ceil(vector_count/N)` response ciphertexts. With 8,192 vectors, both +parameter profiles below still return **one** response ciphertext, while +Paillier returns 8,192. The larger batch amortizes BFV's fixed query cost. + +These new runs use the same 8,192 synthetic 512-bit vectors and query, seed 1337, +the current C++ `residue` backend, and standard CPU Paillier with 1,024-bit primes. +One Paillier baseline is shared between the two BFV comparisons. The runs were +sequential, with no concurrent tests or other benchmarks. Each variant checked +**every distance** against plaintext Hamming distance and selected the same top +three positions: `[0, 8191, 2654]` with distances `[0, 0, 217]`. + +Both BFV profiles use t=65,537, a 180-bit product modulus, 30-bit gadget digits, +eta=21 and 50-bit response compaction. N=8,192 retains the earlier engineering +ring size; N=16,384 matches the arithmetic parameters in `bfv_review_policy()`. +Neither comparison establishes equivalent cryptographic security to Paillier +or SEAL. + +| Exact serialized bytes | Paillier CPU | Homemade BFV, N=8,192 | Homemade BFV, N=16,384 | +| --- | ---: | ---: | ---: | +| Query upload per search | 544 | 368,754 | 737,394 | +| Response download per search | 4,226,979 | 102,496 | 204,900 | +| **Query + response per search** | 4,227,523 | 471,250 | 942,294 | +| Encrypted index | 4,226,979 | 188,787,740 | 188,765,728 | +| Public/evaluation keys, JSON | 2,502 | 85,603,562 | 171,204,881 | + +| BFV profile versus the same Paillier baseline | Response reduction factor | Query + response reduction factor | Fewer query + response bytes | +| --- | ---: | ---: | ---: | +| N=8,192 | 41.24× | 8.97× | 88.9% | +| N=16,384 review parameters | 20.63× | 4.49× | 77.7% | + +A reduction factor is `Paillier bytes / BFV bytes`. Index and public/evaluation +keys are initial upload/storage or key-rotation costs, excluded from recurring +query-plus-response totals. They are substantially larger for BFV. The index +contains 512 ciphertexts at N=8,192 and 256 +at N=16,384. Increasing the ring halves the index ciphertext count while +doubling each ciphertext's size; it also increases the query and response size. + +All three variants use the same MessagePack framing with implicit candidate IDs +and little-endian integer bytes. Counts exclude HTTP/TLS, explicit IDs, content +retrieval and private keys. This measures raw BFV, including when using the +review parameters: it does not include the authenticated/Nitro protocol, +receipts, attestation or network transport. The local Nitro protocol is +measured separately in the [Nitro report](native-bfv-nitro.md). These sizes do +not inherit the original SEAL experiment's different framing or security +configuration. + +Single-run CPU timings on an AMD Ryzen 7 5800X are included for context: + +| Seconds, excluding serialization | Paillier CPU | Homemade BFV, N=8,192 | Homemade BFV, N=16,384 | +| --- | ---: | ---: | ---: | +| Encrypt index | 86.237863 | 54.573942 | 56.275379 | +| Public server computation, first search | 0.043897 | 19.836359 | 20.878492 | +| Client decrypt/decode | 57.763218 | 0.041912 | 0.086103 | + +These are one fresh setup and one first search per variant, not warm medians or +a statistically established latency comparison. BFV's first search includes +lazy public-key cache preparation. Client decode uses the raw client's GMP +path, without the guarded/attested client's native private decoder or receipt +checks. No GPU, EC2 or live service was used. Minimum compacted diagnostic noise +budgets were 26 bits (N=8,192) and +25 bits (N=16,384); these diagnostics are +correctness observations, not security-level estimates. + +Raw records, including exact commands, configuration, phase timings, correctness +and source/native-binary hashes: + +- `benchmarks/results/native_bfv_8192.json`: N=8,192 BFV and the shared Paillier baseline. +- `benchmarks/results/native_bfv_review_8192.json`: N=16,384 BFV on the same input. + +Reproduce sequentially from the repository root: + +```bash +.venv/bin/python benchmarks/bfv_client_matrix.py \ + --num-vectors 8192 --embed-len 512 --poly-modulus-degree 8192 \ + --server-backend residue --rns-modulus \ + --variants bfv-packed,paillier-cpu --repeats 1 \ + --json-out benchmarks/results/native_bfv_8192.json + +.venv/bin/python benchmarks/bfv_client_matrix.py \ + --num-vectors 8192 --embed-len 512 --poly-modulus-degree 16384 \ + --server-backend residue --rns-modulus \ + --max-public-key-chars 268435456 --variants bfv-packed --repeats 1 \ + --json-out benchmarks/results/native_bfv_review_8192.json +``` + +The benchmark's explicit public-key import allowance accommodates the larger +locally generated review-profile key bundle. SDK and protocol defaults are +unchanged. The records identify base commit `be71b0d` +and the measured working-tree source hashes; the benchmark-only changes add +that import option and native source/binary provenance.