Repository navigation
feat(api): prune data-manager pipeline runs after the job retention - #721
Merged
Merged
Conversation
Nothing deleted data-manager jobs or their stages, although the jobs list in the admin panel said they were pruned with the admin jobs. A run an administrator started keeps their account id in triggered_by, which the privacy catalogue registers as personal data. The daily retention now deletes finished data-manager jobs, and by cascade their stages, once ADMIN_JOB_RETENTION_DAYS have passed. It keeps: - running jobs, which have no finish time; - the newest finished job of each kind, so the transit state still reports the last sync when the schedule is rare or off; - jobs of an account whose data-subject request still holds them. The registration is an operator task, whose hold becomes action_required while the operator preserves the rows by hand. The preservation guard so far honoured only held. It now takes the statuses that keep a caller's rows, and this prune passes both. Scheduled automatic version bumps are stored as cron:data-manager-auto-bump, which the activity list did not recognise. It now labels them "Automatic version bump". The schema comment on triggered_by describes the stored format.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
Nothing deleted
data_manager.jobsordata_manager.job_stages. Production built up about 34,000 jobs and 130,000 stages before #704. The monitoring docs said Jobs-tab rows were pruned afterADMIN_JOB_RETENTION_DAYS, but that was only true for admin jobs. A run an administrator started storesapi:user:<id>intriggered_by, which the privacy catalogue registers as personal data (data-manager-trigger-attribution).The daily retention run now calls
pruneDataManagerJobs(ADMIN_JOB_RETENTION_DAYS), next to the admin-job prune. It deletes finished data-manager jobs, and by cascade their stages, more than that many days after they finished. It keeps:GET /transit/statereports the last sync from the newest row, so a rare or disabled schedule keeps it.triggered_by, in each of the three user formats the activity list reads.That last point needed a change to the shared preservation guard. This registration is an
operator_task: on request intake its hold becomesaction_required, because an operator preserves the rows by hand.preservationAllowsDeletionhonoured onlyheld, so a plain prune would have deleted exactly the rows the operator was asked to keep. The guard now takes the hold statuses that keep a caller's rows, and this prune passesheldandaction_required.The statuses render as fixed literals from a closed set. The default (
held) produces exactly the SQL it did before, so the existing callers don't change.Also:
cron:data-manager-auto-bump. The activity list only recogniseddata-manager-auto-bump, so these runs showed as "cron data manager auto bump". They now show as "Automatic version bump".triggered_bynow describes the stored format.Follow-up
application-logsis also anoperator_taskregistration with the same gap; #723, stacked on this PR, applies the same guard there.Verification
data-manager-job-retention-postgres.test.ts, new, in theapi-privacygroup, against PostgreSQL:api:user:andmanual:admin:, are kept while its hold isaction_required, and deleted once the hold is released. Another account's job and a scheduled job are deleted.preservation.test.tschecks the guard's SQL for the new option, and that an empty status list is refused. The existing assertionpreservation.status = 'held'still passes unchanged.pnpm test:database api-privacyagainst a freshly migrated PostGIS: 25 files, 145 tests pass.npx vitest run --project node: 18,772 tests pass. Type checks for api andpnpm lintpass.ingestionandexport-restoredatabase groups, and the daily retention inside a running app-api.Release and upgrade impact
Product impact: ordinary
On the first retention run after the upgrade, which happens at app-api startup, finished data-manager runs older than
ADMIN_JOB_RETENTION_DAYS(default 30 days) are deleted. To keep a longer pipeline history, raise that setting before upgrading. It also applies to admin jobs.