Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .changeset/bounded-history-pages.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
"@triplex-build/triplex": minor
---

**Bounded history pages, and an index that makes optional projections fast on SQL.**

`Triples.historyPage(entityId, { attribute?, limit?, cursor?, recordedPosition? })` returns one page
of an entity's history, or of one attribute's history, oldest first by commit position and then fact
ID, including retracted assertions. The first page captures a commit position (the head, or
`recordedPosition`) and every continuation reads at that position, so writes between pages neither
repeat nor skip facts: a later assertion is not shown, and a fact retracted after the snapshot is shown
unretracted, as it was then. Pages default to 100 facts (maximum 1,000) and are read by an index seek,
so a page costs the same for an entity with 10 facts as for one with 10,000. The cursor is opaque and
bound to its database scope, entity, attribute and position; reusing it elsewhere fails with
`PaginationCursorError`. Sealed values stay sealed. The host protocol has a matching `HistoryPage`
operation. `history(entityId)` is unchanged.

SQL migration 2 (`triplex_entity_attribute_history_index`) and Cloudflare migration 8 add
`idx_entity_attribute_history ON triples(entity_id, attribute, recorded_position)`. Besides serving
per-attribute history pages, it lets a Datalog `optionalProjection` look up each row's value directly;
before, SQLite planned those per-row lookups on an attribute-wide index. On the explorer's
1,200-worker demo database, a 50-row page sorted by an optional attribute drops from 6.2–8.7 s to
38–56 ms.

`runMigrations` and the migrating layers upgrade a 0.5.0 database in place. The non-migrating and
read-only layers refuse it with `SchemaVersionError` (`pending: true`) until migrations run. In-memory
and FoundationDB KV stores backfill the new history keys once, in bounded batches, on the first history
page.
5 changes: 4 additions & 1 deletion docs/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,10 @@ Corrections therefore append evidence:
or one attributed raw `Triples.transact` call.

`history(entityId)` includes retracted assertions. Current `entity` and `match` reads show only
facts visible at their requested bitemporal basis.
facts visible at their requested bitemporal basis. For an entity with a long history, read it in
bounded pages with `historyPage(entityId, { attribute?, limit?, cursor? })`: pages are oldest first,
come from an index seek, and stay pinned to the first page's commit position, so writes between pages
neither repeat nor skip facts.

### End a fact at a time

Expand Down
6 changes: 6 additions & 0 deletions packages/cloudflare/src/migrations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import {
import {
NUMERIC_VALUE_INDEX_DDL,
ENTITY_RECORDED_POSITION_INDEX_DDL,
ENTITY_ATTRIBUTE_HISTORY_INDEX_DDL,
ENTITY_BLOBS_TABLE_DDL,
ENTITY_SNAPSHOTS_TABLE_DDL,
SNAPSHOT_INDEX_DDLS,
Expand Down Expand Up @@ -68,4 +69,9 @@ export const migrations: readonly Migration[] = [
name: "triplex_journal_store",
up: [...JOURNAL_DDLS, "DROP TABLE IF EXISTS triplex_command_receipts"],
},
{
version: 8,
name: "triplex_entity_attribute_history_index",
up: [ENTITY_ATTRIBUTE_HISTORY_INDEX_DDL],
},
];
3 changes: 3 additions & 0 deletions packages/cloudflare/src/schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import {
NUMERIC_VALUE_INDEX_DDL,
SEALED_BLIND_INDEX_DDL,
ENTITY_RECORDED_POSITION_INDEX_DDL,
ENTITY_ATTRIBUTE_HISTORY_INDEX_DDL,
} from "@triplex-build/triplex-sql";

/**
Expand Down Expand Up @@ -79,6 +80,7 @@ export const INDEX_DDLS = [
NUMERIC_VALUE_INDEX_DDL,
SEALED_BLIND_INDEX_DDL,
ENTITY_RECORDED_POSITION_INDEX_DDL,
ENTITY_ATTRIBUTE_HISTORY_INDEX_DDL,
] as const;

const TRIPLES_V3_COLUMNS =
Expand Down Expand Up @@ -120,6 +122,7 @@ export const INDEX_NAMES = [
"idx_temporal",
"idx_recorded_position",
"idx_entity_recorded_position",
"idx_entity_attribute_history",
"idx_entity_attr",
"idx_tx_id",
] as const;
10 changes: 10 additions & 0 deletions packages/cloudflare/src/storage/CloudflareAdapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import {
makeSqlOperationalEntryStorage,
makeSqlLivePreconditionLookup,
makeSqlLiveFactsLookup,
makeSqlHistoryPage,
makeSqlTransactionJournal,
SqliteDialect,
} from "@triplex-build/triplex-sql";
Expand Down Expand Up @@ -668,6 +669,15 @@ export function makeCloudflareAdapter(ctx: DOState): SqlStorageAdapterService {
},
SqliteDialect,
),
historyPage: makeSqlHistoryPage(
{
run: <Row extends Record<string, unknown>>(statement: string, params: readonly unknown[]) =>
Effect.try(() =>
cursorToArray(sqlStorage.exec<Row>(statement, ...(params as SqlStorageValue[]))),
),
},
SqliteDialect,
),
livePreconditionFacts: makeSqlLivePreconditionLookup(
{
run: <Row extends Record<string, unknown>>(statement: string, params: readonly unknown[]) =>
Expand Down
175 changes: 175 additions & 0 deletions packages/core/src/HistoryPage.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
import { Result } from "effect";
import { Base64Url } from "effect/encoding";

import { PaginationCursorError } from "./errors/index.js";
import * as ContentId from "./content/ContentId.js";

export type HistoryPageCursor = string & { readonly HistoryPageCursor: unique symbol };

export interface HistoryPageRequest {
/** Only this attribute's assertions. Omit for every attribute of the entity. */
readonly attribute?: string;
/** Defaults to 100; maximum 1,000. */
readonly limit?: number;
/** Opaque continuation returned by the previous page of the same entity and attribute. */
readonly cursor?: string;
/**
* Read history as recorded at this commit position. Defaults to the current head; a
* position beyond the head reads the head. A continuation must repeat the first page's
* value or omit it.
*/
readonly recordedPosition?: number;
}

/** Where a history page resumes: after this assertion in (position, ID) order. */
export interface HistoryPageAfter {
readonly position: number;
readonly id: string;
}

interface CursorEnvelope {
readonly version: 1;
readonly scope: string;
readonly entityId: string;
readonly attribute: string | null;
readonly snapshotPosition: number;
readonly afterPosition: number;
readonly afterId: string;
}

export interface PreparedHistoryPage {
readonly entityId: string;
readonly attribute?: string;
readonly limit: number;
readonly snapshotPosition: number;
readonly after?: HistoryPageAfter;
readonly scope: string;
}

const decoder = new TextDecoder();
const encoder = new TextEncoder();

const malformed = (message: string, cause?: unknown) =>
new PaginationCursorError({
reason: "malformed",
message,
...(cause === undefined ? {} : { cause }),
});

const isPosition = (value: unknown): value is number =>
typeof value === "number" && Number.isSafeInteger(value) && value >= 0;

const decode = (cursor: string): CursorEnvelope => {
try {
const bytes = Base64Url.decode(cursor);
if (Result.isFailure(bytes)) throw bytes.failure;
const value: unknown = JSON.parse(decoder.decode(bytes.success));
if (typeof value !== "object" || value === null || !("version" in value))
throw malformed("History page cursor has an invalid shape");
if (value.version !== 1)
throw new PaginationCursorError({
reason: "unsupported_version",
message: "History page cursor has an unsupported version",
});
const v = value as Partial<CursorEnvelope>;
if (
typeof v.scope !== "string" ||
typeof v.entityId !== "string" ||
(v.attribute !== null && typeof v.attribute !== "string") ||
!isPosition(v.snapshotPosition) ||
!isPosition(v.afterPosition) ||
v.afterPosition > v.snapshotPosition ||
typeof v.afterId !== "string"
)
throw malformed("History page cursor has an invalid shape");
return v as CursorEnvelope;
} catch (cause) {
if (cause instanceof PaginationCursorError) throw cause;
throw malformed("History page cursor is not valid base64url JSON", cause);
}
};

const scopeFingerprint = (scope: string): string =>
ContentId.hash(ContentId.Domain.paginationScope, `history-page\u0000${scope}`);

/**
* Validate a history page request against the current head. Throws
* `PaginationCursorError` for invalid limits, positions and cursors.
*/
export const prepareHistoryPage = (input: {
readonly entityId: string;
readonly request: HistoryPageRequest;
readonly recordedPosition: number;
readonly scope: string;
}): PreparedHistoryPage => {
const { request } = input;
const limit = request.limit ?? 100;
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 1_000)
throw malformed("History page limit must be between 1 and 1000");
if (request.recordedPosition !== undefined && !isPosition(request.recordedPosition))
throw malformed("History page recordedPosition must be a non-negative safe integer");
if (request.attribute !== undefined && typeof request.attribute !== "string")
throw malformed("History page attribute must be a string");
const scope = scopeFingerprint(input.scope);
const base = {
entityId: input.entityId,
...(request.attribute === undefined ? {} : { attribute: request.attribute }),
limit,
scope,
};
if (request.cursor === undefined)
return {
...base,
snapshotPosition: Math.min(
request.recordedPosition ?? input.recordedPosition,
input.recordedPosition,
),
};
const envelope = decode(request.cursor);
if (envelope.scope !== scope)
throw new PaginationCursorError({
reason: "scope_mismatch",
message: "History page cursor belongs to a different database scope",
});
if (envelope.entityId !== input.entityId || envelope.attribute !== (request.attribute ?? null))
throw new PaginationCursorError({
reason: "query_mismatch",
message: "History page cursor belongs to a different entity or attribute",
});
if (
request.recordedPosition !== undefined &&
Math.min(request.recordedPosition, input.recordedPosition) !== envelope.snapshotPosition
)
throw new PaginationCursorError({
reason: "basis_mismatch",
message: "History page cursor cannot be reused with a different recordedPosition",
});
if (envelope.snapshotPosition > input.recordedPosition)
throw new PaginationCursorError({
reason: "basis_mismatch",
message: "History page cursor is ahead of this database's committed head",
});
return {
...base,
snapshotPosition: envelope.snapshotPosition,
after: { position: envelope.afterPosition, id: envelope.afterId },
};
};

export const encodeHistoryPageCursor = (
prepared: PreparedHistoryPage,
after: HistoryPageAfter,
): HistoryPageCursor =>
Base64Url.encode(
encoder.encode(
JSON.stringify({
version: 1,
scope: prepared.scope,
entityId: prepared.entityId,
attribute: prepared.attribute ?? null,
snapshotPosition: prepared.snapshotPosition,
afterPosition: after.position,
afterId: after.id,
} satisfies CursorEnvelope),
),
) as HistoryPageCursor;
2 changes: 2 additions & 0 deletions packages/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ export {
type EntityPageRequest,
type EntityPageSnapshot,
} from "./EntityPage.js";
export { type HistoryPageCursor, type HistoryPageRequest } from "./HistoryPage.js";
export * from "./Value.js";
export * as Constraint from "./Constraint.js";
export * as CommitValidation from "./store/CommitValidation.js";
Expand Down Expand Up @@ -88,6 +89,7 @@ export {
type EntityTransactionPage,
type DependencyState,
type EntityPage,
type HistoryPage,
type QueryOptions,
type PagedQueryOptions,
type QueryResponse,
Expand Down
Loading
Loading