feat(public-server): U1: spec the reporting-schema pipeline by dannash100 · Pull Request #525 · beyondessential/canopy

dannash100 · 2026-08-30T23:43:05Z

Spec only, proposed. Stacked on #529 (deployment-scoped artefacts).

Reporting schemas are built against a database nobody records, so groups sit minors behind the schema they run.

Canopy derives what each central server is owed from its upgrade plan, dispatches it as a build restore intent, records the artefacts, and grades what a server reports applying.
The builder is the restore consumer, so no database is handed between actors and there is no second worklist.
A build takes an unredacted source (see comment).

dannash100 · 2026-09-02T08:43:51Z

🤖 Side finding from checking the redaction path, not part of this spec's contract.

pgro applies the masking manifest destructively (UPDATE ... SET col = anon.lorem_ipsum(...)), where bestool psql only redacts on display. The invoice tables mask their reference keys and visibility_status with a random text mask, so on a redacted replica where visibility_status = 'current' matches nothing and the masked ids stop joining. A reporting build against one emits the invoice pivots with zero columns and still reports success.

No mask kind in pgro's registry is deterministic, so masking both sides of a join can't work as intended, and the manifest is already inconsistent about it: patients.id is unmasked while patient_invoice_insurance_plans.patient_id is masked.

That's why the spec has a build take an unredacted source. The real fix is unmasking the key and visibility columns, which belongs outside this PR.

dannash100 · 2026-09-02T21:54:23Z

🤖 Broken out for the dbt side, since it lands in Maui's repos rather than here:

build_reporting_assets.py runnable headless against a database it is handed, rather than a local .env or a port-forward into tamanu-release-X-Y.
The build callable without cutting a release.
publish-artifacts.yml to stop collapsing v2.60.2 into v2.60.x, which overwrites the previous bundle at the same key and leaves nothing to grade currency against.
A version bump path for the deployment repos, which have neither create-version-branch.yml nor tamanu-upgrade.yml.

Wants a Linear card in MAUI; issues are disabled on those repos.

passcod · 2026-09-03T22:42:09Z

+
+# Reporting schemas
+
+A reporting schema is the set of database views a group's reports read from.


Ehh. Not really. It's specifically a Tamanu thing, and can't be generalised to "a group" (way too wide) or other applications/machines. Tamanu applications happen to live in groups, and reporting schemas are built against a representative Tamanu database of a group's, and applied identically to all Tamanus in that group. But that's not the same thing as "the set of database views [for] a group's reports"

passcod · 2026-09-03T22:45:10Z

+# Reporting schemas
+
+A reporting schema is the set of database views a group's reports read from.
+Canopy decides which groups owe a reporting schema for which version, supplies what a build needs to produce one, records the artefacts that result, and grades whether what a group runs is the schema it should have.


Worth actually specifying this, instead of using the "owe" phrasing. Reporting schemas are unique per (group, tamanu version) tuple. Canopy wants to be able to provide the reporting schema for every such tuple it knows about. No "owing" anything.

passcod · 2026-09-03T22:47:07Z

+A reporting schema is derived from two things no single actor holds.
+Half of it follows from a Tamanu version's database schema and is the same for every group on that version; the other half follows from the group's own configuration, and can only be produced from a database carrying it.


This is contradictory on its face, since we've just said that no single actor holds both things, and then enumerated two things that are properties of a single database system.

passcod · 2026-09-03T22:47:35Z

+A reporting schema is derived from two things no single actor holds.
+Half of it follows from a Tamanu version's database schema and is the same for every group on that version; the other half follows from the group's own configuration, and can only be produced from a database carrying it.
+
+Canopy already holds both sides.


And then immediately say that Canopy (a single actor) holds both sides

passcod · 2026-09-03T22:48:45Z

+
+An **operator** declares which groups are covered, reads the currency of each, and asks for the builds the derivation does not produce on its own.
+
+Canopy owns which group owes a schema for which version, the database that build is entitled to, the artefact that results, and the grading of what a group actually runs.


Again, Tamanus in a group want a reporting schema adapted for its particularities, it's not a question of owing.

passcod · 2026-09-03T23:14:51Z

+
+## Alerting
+
+A server whose schema is behind raises a reporting-schema check on itself, as does a central server owed a build nothing has produced (see [CHK](../monitoring/checks.md)).


Again, if this goes via bestool/normal healthchecks then there's no need for custom machinery here.

passcod · 2026-09-03T23:15:04Z

+A replica that failed to restore or come up is the restore's own health rather than a build failure, and is dispatched again as any unhealthy restore is.
+A settled pair is dispatched again by an operator asking for a build, which is what these checks exist to prompt.
+
+## Out of scope


Same as in the other spec

passcod · 2026-09-03T23:16:36Z

 ### Dispatching a migration test

-`migrate` is a semantic an intent opts into, and an intent carrying it carries no other purpose.
+`migrate` is a semantic an intent opts into, and an intent carrying it neither verifies backups nor serves a queryable replica.


Might be simpler to just remove the second clause here entirely, instead of creating a non-exhaustive list of non-applications that can go stale as soon as more purposes show up.

passcod · 2026-09-03T23:17:05Z

 It carries `check` alongside, so a single restore reports the replica's health and the migrations' outcome as two signals from one report.

-An intent carrying `migrate` is withheld from a server with no candidate version.
+An intent carrying `migrate` is withheld from a server it has no version to name.


Useless rewrite?

passcod · 2026-09-03T23:18:32Z


 A verifying intent and a migrating intent restore the same snapshot separately.
 A verifying intent restores once per snapshot, and a migrating intent's `once` is keyed to the snapshot and target version together, so it restores when a new candidate version appears rather than on every snapshot.
+An intent carrying `build` restores separately again, keyed to the pair it is owed rather than to the snapshot, so a schema is built when one is owed and not otherwise (see [RPT](reporting-schemas.md)).


If the clause in my prior comment is removed, this confusing sentence can go away, and the build semantic is just specified separately. Also, build as a word is way too generic.

…to rpt-review

passcod · 2026-09-04T00:53:23Z


 The credentials permit reading the repo and nothing else; they cannot write, overwrite, or delete.
-A consumer whose intent carries `build` additionally obtains, for the same `(group, type)`, a short-lived credential that writes to the group's artefact prefix and reaches nothing in the repo, which is what its builds publish over (see [RPT](reporting-schemas.md)).
+A consumer whose intent carries `reporting-schema` additionally obtains, for the same `(group, type)`, a short-lived credential that writes to the group's artefact prefix and reaches nothing in the repo, over which it publishes what it builds (see [RPT](reporting-schemas.md)).


We'll see how the upper thing resolves, but that might be "the consumer uses its existing canopy connection to upload the thing (and then canopy figures out where it goes and how)" rather than "we hand out credentials to S3"

passcod

I like this better!

…to rpt-review2

dannash100 mentioned this pull request Aug 31, 2026

feat: surface the reporting schema version a server reports #526

Merged

dannash100 mentioned this pull request Sep 3, 2026

feat(platform): D2: spec deployment-scoped artefacts #529

Open

dannash100 added 15 commits September 3, 2026 15:37

spec reporting schemas

70fc850

tighten reporting spec

b6eb71e

pin artefact and source

1aa1942

adopt group terminology

8c46775

settle artefact identity

c49f4b4

trim alerting

9a7995b

spec version artefacts

54b3089

demote coverage

c8028d3

build rides the migrate pathway

5d6a461

settle build contract

71b92ce

propose apply and publish

4c6ab3d

facilities run it too

14edef8

rpt review pass

dfb931e

device applies the schema

1a82c7e

name and trace artefacts

7b1e3cc

dannash100 changed the base branch from main to feature/deployment-artefacts September 3, 2026 03:39

dannash100 force-pushed the feature/reporting-schema-pipeline branch from c18ff55 to 7b1e3cc Compare September 3, 2026 03:41

passcod requested changes Sep 3, 2026

View reviewed changes

dannash100 added 2 commits September 4, 2026 11:56

Merge remote-tracking branch 'origin/feature/deployment-artefacts' in…

4fef7ab

…to rpt-review

address spec review

f682611

passcod reviewed Sep 4, 2026

View reviewed changes

passcod approved these changes Sep 4, 2026

View reviewed changes

dannash100 added 2 commits September 4, 2026 13:03

Merge remote-tracking branch 'origin/feature/deployment-artefacts' in…

23afb56

…to rpt-review2

publish through canopy

b08c287


		# Reporting schemas

		A reporting schema is the set of database views a group's reports read from.

		A reporting schema is derived from two things no single actor holds.
		Half of it follows from a Tamanu version's database schema and is the same for every group on that version; the other half follows from the group's own configuration, and can only be produced from a database carrying it.


		An operator declares which groups are covered, reads the currency of each, and asks for the builds the derivation does not produce on its own.

		Canopy owns which group owes a schema for which version, the database that build is entitled to, the artefact that results, and the grading of what a group actually runs.


		## Alerting

		A server whose schema is behind raises a reporting-schema check on itself, as does a central server owed a build nothing has produced (see [CHK](../monitoring/checks.md)).

Uh oh!

Conversation

dannash100 commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dannash100 commented Sep 2, 2026

Uh oh!

dannash100 commented Sep 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

passcod left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dannash100 commented Aug 30, 2026 •

edited

Loading