Release v0.3.0: native TCP safety

.github/workflows/ci.yml

@@ -15,9 +15,10 @@ concurrency:
 jobs:
   test:
     timeout-minutes: 25
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
     strategy:
       matrix:
+        os: [ubuntu-latest, macos-latest]
         python-version: ["3.10", "3.12"]
     steps:
       - name: Checkout Python driver
@@ -27,7 +28,7 @@ jobs:
         uses: actions/checkout@v4
         with:
           repository: puffball1567/koutendb
-          ref: v0.12.0
+          ref: e36b424bcfd9cd0dfa24ae121f4b4dd028b0eaac
           path: koutendb-core
 
       - name: Set up Python
@@ -36,21 +37,37 @@ jobs:
           python-version: ${{ matrix.python-version }}
 
       - name: Install Nim
+        if: runner.os == 'Linux'
         uses: jiro4989/setup-nim-action@v2
         with:
           nim-version: "2.2.10"
 
-      - name: Install dependencies
+      - name: Install dependencies on Linux
+        if: runner.os == 'Linux'
         run: sudo apt-get update && sudo apt-get install -y libsodium-dev
 
+      - name: Install dependencies on macOS
+        if: runner.os == 'macOS'
+        run: |
+          brew install nim libsodium openssl@3
+          sodium="$(brew --prefix libsodium)"
+          ssl="$(brew --prefix openssl@3)"
+          echo "$ssl/bin" >> "$GITHUB_PATH"
+          echo "LIBRARY_PATH=$sodium/lib" >> "$GITHUB_ENV"
+          echo "CPATH=$sodium/include" >> "$GITHUB_ENV"
+          echo "DYLD_LIBRARY_PATH=$sodium/lib:$ssl/lib" >> "$GITHUB_ENV"
+
       - name: Build koutend
         run: |
           cd koutendb-core
           nimble install -y
-          nim c -d:release --nimcache:/tmp/nimcache_koutend -o:src/koutend src/koutend.nim
+          nim c -d:release -d:ssl --nimcache:/tmp/nimcache_koutend -o:src/koutend src/koutend.nim
 
       - name: Install package
-        run: python -m pip install -e .
+        run: python -m pip install -e '.[secure]'
 
       - name: Run tests
         run: KOUTENDB_CORE_DIR="${{ github.workspace }}/koutendb-core" python -m unittest discover -s tests
+
+      - name: Shared native TCP conformance
+        run: python koutendb-core/scripts/native_driver_conformance.py --server koutendb-core/src/koutend -- python "$PWD/tests/tcp_adapter.py"

MANIFEST.in

@@ -0,0 +1,3 @@
+include docs/native-tcp.md
+include docs/native-tcp-validation.md
+include tests/tcp_adapter.py

README.md

@@ -6,9 +6,12 @@ This driver talks to `koutend` over KoutenDB's high-level wire protocol. It does
 not reimplement KoutenDB's ring-key, period, head-angle, or placement rules.
 Applications pass a human-readable ring name, and KoutenDB returns a typed ID.
 
+Version 0.3.0 adds stricter framing, version negotiation, typed errors and safe
+retry behavior. See [the TCP safety and migration guide](docs/native-tcp.md).
+
 ## Status
 
-- package: PyPI [`koutendb`](https://pypi.org/project/koutendb/) v0.2.1
+- package: PyPI [`koutendb`](https://pypi.org/project/koutendb/) v0.3.0
 - current mode: native TCP wire driver
 - Python: 3.10+
 - runtime dependencies: none
@@ -24,9 +27,9 @@ Implemented:
 - codec metadata negotiation with `CODECMETA ON`
 - `batch_get`
 - direct owner redirects from extended `FWD ... owner` responses
-- routed multi-node `batch_get` fallback with stable input ordering
+- ordered `batch_get` using epoch-aware GETID requests
 - typed `KoutenId`
-- one reconnect retry
+- at most one reconnect retry for reads; no automatic write replay
 - context manager support
 - username/password, shared-secret transport, and TLS authentication
 

docs/native-tcp-validation.md

@@ -0,0 +1,22 @@
+# Native TCP Validation
+
+Date: 2026-09-19
+
+Local Linux verification against the shared KoutenDB conformance harness at
+core commit `e36b424bcfd9cd0dfa24ae121f4b4dd028b0eaac`:
+
+- All 27 scripted protocol/failure cases passed.
+- All six real-server modes passed: plain, password, token, secret, TLS, TLS+secret.
+- Verified Unicode, empty/binary data and 1 MiB payload round trips.
+- Verified invalid credentials, certificate rejection, hostname mismatch,
+  bounded redirects, partial frames, timeout, disconnection and unsafe-write replay prevention.
+
+- Unit, two-node integration and crypto tests: 17 passed.
+- Empty payloads and duplicate IDs retain their values and ordering in batch_get.
+- Wheel and source distribution build: passed; transport/error modules included.
+
+The GitHub workflow runs the shared conformance matrix on Linux and macOS.
+See the release commit's workflow checks for CI results. The local results above
+are correctness/integration checks, not load or long-duration operational tests.
+
+See [native TCP usage and reproduction commands](native-tcp.md).

docs/native-tcp.md

@@ -0,0 +1,114 @@
+# Native TCP Safety Update
+
+Python already uses native TCP and still needs no libkoutendb or FFI. This
+release hardens that transport rather than adding another one.
+
+```python
+from koutendb import KoutenClient, IndeterminateWriteException
+
+with KoutenClient.connect(
+    ["127.0.0.1:17301"], timeout=3,
+    read_timeout=5, write_timeout=5,
+) as db:
+    document_id = db.put_json("articles", {"title": "Hello"})
+    print(db.get_json(document_id))
+```
+
+For TLS set `tls=True`, `tls_ca_file="ca.pem"` and
+`tls_server_name="db.example.com"`. Omit the CA file to use system roots.
+Credentials are `username`, `password`, `auth_token`, `secret_key` and `galaxy`.
+Shared-secret mode requires `pip install 'koutendb[secure]'`; ordinary TLS uses
+Python's standard library.
+
+New exceptions inherit KoutenError, so existing broad error handlers still work:
+ConnectionException, ConnectionTimeoutException, AuthenticationException,
+ProtocolException, VersionMismatchException, ServerException and
+IndeterminateWriteException.
+
+Construction remains lazy. The first operation negotiates the protocol before
+use. A client serializes operations with a lock; close is now terminal.
+For local development install this checkout with `pip install -e '.[secure]'`.
+DNS resolution is OS-controlled and may outlast the connection timeout.
+
+## Behavior Changes
+
+Unsafe automatic write replay and all-peer miss probing have been removed.
+Reads follow explicit server redirects only. Keep the peer ordering consistent
+with the server cluster.
+
+`batch_get` now uses ordered GETID requests. Wire-v1 BGET omits the epoch and
+conflates an empty value with a miss. This prioritizes correct identity and
+empty-value handling, but means one request per ID rather than one BGET frame.
+Do not expect the previous batch throughput. The return shape and input ordering
+are unchanged.
+
+```sh
+KOUTENDB_CORE_DIR=../koutendb python3 -m unittest discover -s tests
+bash ../koutendb/scripts/native_driver_conformance.sh python3 "$PWD/tests/tcp_adapter.py"
+```
+
+## Server Setup
+
+Run a TLS-enabled `koutend` build. For a local-only first test:
+
+```sh
+koutend --id=0 --peers=127.0.0.1:17301 --data=./kouten-data
+```
+
+Keep plaintext connections on localhost or an isolated, trusted private network.
+A Docker network is not a substitute for access control. Use verified TLS when
+traffic crosses a trust boundary. For password authentication, start the server
+with `--user=app --password=...`; prefer the server's configuration/secret
+management facilities for production rather than putting secrets in shell history.
+
+Native TCP implements wire version 1: WIREVER, CODECMETA, PUTR, GETID, QRYID,
+HEALTH, authentication and bounded FWD handling. It is not a replacement for
+every embedded/admin API. It uses server-provided IDs and does not calculate
+ring placement or orbit ownership. Peer ordering must match the server cluster
+configuration, because explicit redirect owners are node indexes.
+
+## Safety Contract
+
+- Every new connection authenticates, checks WIREVER and enables codec metadata
+  before sending application requests. Unsupported versions fail closed.
+- Headers are bounded to 8 KiB; payload frames default to at most 64 MiB.
+  The configurable payload cap cannot exceed that hard limit.
+- Partial reads/writes are handled. A read deadline covers the complete response,
+  not a fresh timeout for every fragment.
+- A read may reconnect and retry once. An unknown write outcome is never retried.
+- After a broken or malformed response the connection is discarded.
+- Redirects default to eight hops (configurable up to 32), and an out-of-range
+  owner is rejected. Missing values do not trigger a scan of every server.
+- CA and hostname verification are enabled by default. TLS 1.2 is the minimum.
+  Insecure verification bypass is explicitly development-only.
+- Password/token and shared-secret challenge authentication are supported.
+  Library transport errors do not include raw server error text or credentials.
+
+A successful send is not proof that a write committed. If the connection breaks
+or the reply is malformed after a PUT may have been sent, handle an
+**indeterminate write** separately from a definite server rejection. Do not
+blindly repeat the insert or assume a fallback database is now authoritative.
+Reconcile at the application level until a server-side idempotency contract is
+available.
+
+The pre-v1 protocol is version-checked, not promised compatible with future
+versions. Authentication errors, protocol errors, connection failures, timeouts,
+server rejections and indeterminate writes are distinguishable.
+
+## Verification
+
+The adapter in this repository runs against KoutenDB's language-independent
+`scripts/native_driver_conformance.py` suite, pinned in CI to core commit
+`e36b424bcfd9cd0dfa24ae121f4b4dd028b0eaac`.
+
+The shared matrix covers 27 scripted cases: fragmented/empty/Unicode/binary
+responses, missing values, projections, invalid lengths/codecs/headers, redacted
+server errors, version mismatches, connection loss, partial-response retry,
+timeouts, backpressure, redirects and poisoned-connection disposal.
+Six real-server configurations cover plaintext, password, token, shared-secret,
+TLS and TLS plus shared-secret; these include 1 MiB round trips, invalid
+credentials, untrusted certificates and hostname mismatch.
+
+These are bounded correctness/integration checks, not endurance or throughput
+benchmarks. Linux results are checked locally; Linux/macOS CI must pass before
+release. Existing embedded regressions remain separate from native TCP checks.

koutendb/__init__.py

@@ -1,3 +1,10 @@
 from .client import EncodedPayload, PayloadCodec, KoutenClient, KoutenError, KoutenId
+from .errors import (ConnectionException, ConnectionTimeoutException,
+                     AuthenticationException, ProtocolException,
+                     VersionMismatchException, ServerException,
+                     IndeterminateWriteException)
 
-__all__ = ["EncodedPayload", "PayloadCodec", "KoutenClient", "KoutenError", "KoutenId"]
+__all__ = ["EncodedPayload", "PayloadCodec", "KoutenClient", "KoutenError", "KoutenId",
+           "ConnectionException", "ConnectionTimeoutException", "AuthenticationException",
+           "ProtocolException", "VersionMismatchException", "ServerException",
+           "IndeterminateWriteException"]