Repository navigation
fix(api): keep application logs a pending data-subject request needs - #723
Merged
Merged
Conversation
Medformatik
added a commit
that referenced
this pull request
Oct 31, 2026
…721) ## What changed and why Nothing deleted `data_manager.jobs` or `data_manager.job_stages`. Production built up about 34,000 jobs and 130,000 stages before #704. The monitoring docs said Jobs-tab rows were pruned after `ADMIN_JOB_RETENTION_DAYS`, but that was only true for admin jobs. A run an administrator started stores `api:user:<id>` in `triggered_by`, which the privacy catalogue registers as personal data (`data-manager-trigger-attribution`). The daily retention run now calls `pruneDataManagerJobs(ADMIN_JOB_RETENTION_DAYS)`, next to the admin-job prune. It deletes finished data-manager jobs, and by cascade their stages, more than that many days after they finished. It keeps: - **Running jobs.** They have no finish time. Only data-manager can tell whether one is really in flight. - **The newest finished job of each kind.** `GET /transit/state` reports the last sync from the newest row, so a rare or disabled schedule keeps it. - **Jobs of an account whose data-subject request still needs them.** The subject's id comes from `triggered_by`, in each of the three user formats the activity list reads. That last point needed a change to the shared preservation guard. This registration is an `operator_task`: on request intake its hold becomes `action_required`, because an operator preserves the rows by hand. `preservationAllowsDeletion` honoured only `held`, so a plain prune would have deleted exactly the rows the operator was asked to keep. The guard now takes the hold statuses that keep a caller's rows, and this prune passes `held` and `action_required`. The statuses render as fixed literals from a closed set. The default (`held`) produces exactly the SQL it did before, so the existing callers don't change. Also: - **Auto-bump label.** The data-manager stores scheduled automatic version bumps as `cron:data-manager-auto-bump`. The activity list only recognised `data-manager-auto-bump`, so these runs showed as "cron data manager auto bump". They now show as "Automatic version bump". - **Schema comment.** The comment on `triggered_by` now describes the stored format. ### Follow-up `application-logs` is also an `operator_task` registration with the same gap; #723, stacked on this PR, applies the same guard there. ## Verification - `data-manager-job-retention-postgres.test.ts`, new, in the `api-privacy` group, against PostgreSQL: - **Retention:** finished and interrupted jobs past the retention are deleted with their stages. A running job and the newest finished job are kept, and so is the only, 90-day-old run of a second kind. - **Request holds:** a real data-subject request is created. Old jobs triggered by that account, as `api:user:` and `manual:admin:`, are kept while its hold is `action_required`, and deleted once the hold is released. Another account's job and a scheduled job are deleted. - `preservation.test.ts` checks the guard's SQL for the new option, and that an empty status list is refused. The existing assertion `preservation.status = 'held'` still passes unchanged. - `pnpm test:database api-privacy` against a freshly migrated PostGIS: 25 files, 145 tests pass. - `npx vitest run --project node`: 18,772 tests pass. Type checks for api and `pnpm lint` pass. - Not run: the `ingestion` and `export-restore` database groups, and the daily retention inside a running app-api. ## Release and upgrade impact Product impact: ordinary On the first retention run after the upgrade, which happens at app-api startup, finished data-manager runs older than `ADMIN_JOB_RETENTION_DAYS` (default 30 days) are deleted. To keep a longer pipeline history, raise that setting before upgrading. It also applies to admin jobs.
Application logs are an operator_task registration: on request intake their hold becomes action_required, while the operator preserves the logs by hand. The daily prune honoured only held holds, so it deleted a person's logs 30 days after they were written even while their request still needed them. The privacy catalogue discloses these logs as kept for 30 days, or 30 days after the request closes. The prune now also keeps the logs through action_required holds, as the data-manager job prune does.
Medformatik
force-pushed
the
fix/app-log-retention-operator-hold
branch
from
October 31, 2026 18:27
8d4e8bb to
d16309b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #721 (merged), which added the preservation guard's
blockingStatusesoption.What changed and why
Application logs are an
operator_taskprivacy registration (application-logs). On request intake, their hold becomesaction_required, because an operator preserves the logs by hand. The daily prune used the preservation guard with its default, which honours onlyheld. So it deleted a person's logs 30 days after they were written, even while their access or erasure request still needed them.The privacy catalogue discloses these logs as kept for 30 days, or 30 days after the request closes (
daysAfterClosure: 30), so the prune contradicted the disclosure.pruneAppLogsnow passesblockingStatuses: ["held", "action_required"], as the data-manager job prune in #721 does. The logs are deleted once the hold is captured or released.The monitoring docs say so next to the log retention setting.
Verification
app-log-retention-postgres.test.ts, new, in theapi-privacygroup:action_required.expected [] to deeply equal [ 4 ]). With the fix, it passes.pnpm test:database api-privacyagainst a freshly migrated PostGIS, after rebasing ontomain: 27 files, 149 tests pass.apps/api/src/servicesandapps/api/src/privacy: 2,986 pass. Type checks for api andpnpm lintpass.Release and upgrade impact
Product impact: ordinary
No operator action. Logs about a person with an open request that still needs them are kept longer than before, until the request no longer needs them.