CLI: support custom webhook paths in project.yaml - #1511
Merged
Conversation
7 tasks
elias-ba
force-pushed
the
feat/webhook-custom-path
branch
from
August 30, 2026 15:03
cdfdb81 to
68dc532
Compare
OpenFn/lightning#5105 lets a webhook trigger be named, so its URL is known before it is deployed. Deploy never sent the field. The new-trigger path builds its payload from an allowlist that does not include it, and the existing-trigger path builds its fields explicitly. A path only travels when the spec mentions it. An absent key means the server keeps what it has, so deploying a spec written before this existed does not wipe a path set through the app. Writing it blank clears it, either as an empty string or as a bare key, which YAML reads as null. Cron and kafka triggers ignore it, matching the server. Both insertion points decide on the resolved trigger type rather than on what the spec wrote, because type is optional in a spec and defaults to webhook. A trigger that names a path and nothing else is a legal spec, and guarding on the spec's own type would drop the path without saying so. Pull needs no change. It fetches the yaml from Lightning, which now exports the field.
`from-app-state` destructures the trigger keys it knows and sweeps the rest into `openfn:`, so a pulled `custom_path` landed nested there rather than beside `webhook_reply`. The v1 and v2 project formats would have disagreed about where a webhook's path lives. The workflow version hash is deliberately left alone. Lightning's `canonical_form/1` does not hash `custom_path` either, and adding it on one side only would make the two hashes disagree for every workflow holding a path, which is what `find-changed-workflows` compares. Moving it needs both sides at once.
elias-ba
force-pushed
the
feat/webhook-custom-path
branch
from
August 30, 2026 15:13
68dc532 to
8af40cd
Compare
elias-ba
added a commit
to OpenFn/lightning
that referenced
this pull request
Aug 30, 2026
A workflow whose only change was its endpoint name hashed the same as before, so anything comparing versions decided nothing had changed. The CLI's find-changed-workflows skipped it and openfn project deploy sent nothing. Costs existing workflows nothing. A nil serialises to an empty string and the parts are concatenated, so hashing the key does not move the hash of a workflow that has no path. @openfn/project skips undefined values for the same reason, which is what keeps the two implementations equal, and its parity test against a real Lightning hash still passes. Paired with OpenFn/kit#1511, which adds the key on that side.
…make Two gaps the review left open. The workflow version hash ignored custom_path, so find-changed-workflows saw no change when a webhook was renamed and openfn project deploy skipped it. Lightning hashes the same key now, in OpenFn/lightning#5105, and the parity test against a real Lightning hash still passes: both sides leave a workflow without a path hashing exactly as it did. And deploy diffs the server's project against the payload it is about to send, so every trigger field held on the server and never named in the spec showed up as a removal that was not going to happen. Absence means leave it alone, so the diff now hides what the payload does not carry. webhook_reply and webhook_response_config had this already.
…mething Three things from review. The diff mask hid fields the server was about to wipe. validate_by_type clears a path when a webhook becomes a cron, and clears the response config when the reply mode stops being after_completion, whether or not the payload names them. Masking those made the user confirm a deploy that retires a live URL with nothing in the diff saying so. The mask now stands down when the type or the reply mode is changing, and for a trigger being deleted, where the diff is the only place the user sees what goes with it. A path on a cron or kafka trigger is no longer hashed. It never served a URL, and Lightning stopped hashing one there in the paired PR, so the two would otherwise disagree. And the type check: the fixture's inferred literal type has no custom_path, so the new test needed the cast the rest of that file already uses. ava does not type check, and I ran it instead of tsc.
elias-ba
changed the title
elias-ba
requested review from
doc-han and
josephjclark
and removed request for
josephjclark
August 30, 2026 17:30
doc-han
approved these changes
Aug 31, 2026
Review point. The non-null assertion says the value is there without anything enforcing it. Nothing can reach this line with an undefined spec trigger today, since splitZip walks the union of keys and the two branches above return for spec-only and state-only, but the check costs nothing and does not depend on that staying true.
doc-han
requested changes
Aug 31, 2026
doc-han
left a comment
doc-han
left a comment
Collaborator
There was a problem hiding this comment.
The changeset looks weird to me. I mostly expect a single one if it's all patch or all minor. I see there's a bit of a mix here. but I don't get how a single PR has a patch and at the same time a minor for deploy.
Deploy had two, a minor for the feature and a patch for the diff fix. The tool flattens them to the higher bump anyway, and one entry per package per release is the shape everywhere else.
josephjclark
requested changes
Aug 31, 2026
josephjclark
left a comment
josephjclark
left a comment
Collaborator
There was a problem hiding this comment.
Please bump versions before merging to main
deploy 0.15.0, project 0.19.0, lexicon 2.4.3, and the dependents changesets pulls along with them.
elias-ba
commented
Aug 31, 2026
Collaborator
Author
There was a problem hiding this comment.
@josephjclark versions are bumped, and the changelog notes are much shorter now.
CLAUDE.md says a changeset note is a single short high-level sentence with no implementation detail. Mine were three paragraphs of it.
elias-ba
added a commit
to OpenFn/lightning
that referenced
this pull request
Aug 31, 2026
* Give webhook triggers a project and a validated custom path A webhook trigger can now hold a `custom_path`, so its endpoint can be named rather than only addressed by the trigger's generated id. The name has to be unique within a project rather than across the instance, so triggers carry a `project_id` denormalised from their workflow. A composite foreign key on `(workflow_id, project_id)` stops it drifting from the workflow's real project, and it is resolved inside the insert transaction so no caller has to remember it. It is nullable because it only scopes a path. A path is validated as a URL segment: lowercase letters, digits, hyphens and underscores, at most 255 characters, blank meaning unset, and never UUID-shaped, since those resolve against trigger ids and would be unreachable. The column has existed since 2022 with no validation, no uniqueness and no UI, so instances can already hold values these rules reject. The migration backfills the project, releases paths held by workflows deleted since, marks the triggers that already answered at a bare URL, dedupes what was never unique, and only then adds the indexes. Its data steps are gated on the trigger being disabled: webhook ingest ignores `deleted_at`, so a hidden workflow with an enabled trigger is still serving. * Resolve webhook URLs by trigger id, project path or bare path `/i/<project-id>/<custom-path>` now reaches a webhook trigger, alongside the generated `/i/<trigger-id>` URL that every trigger has always had. Resolution is tried in order and no step can match more than one row, so a request can never fail on an ambiguous path. Each step is backed by a constraint: the primary key for a trigger id, and a partial unique index for each of the two path forms. Before this the lookup resolved a path with `coalesce(custom_path, id)` through `Repo.one`, so a path set in one project could match another project's trigger id, return two rows and permanently 500 that project's live endpoint. Paths that were addressable bare at `/i/<path>` keep answering there. That set is fixed by the migration and never grows, so a path created from now on cannot claim a bare URL out from under an existing one. The ingest plug's telemetry context is narrowed to the first path segment, which is always the address. The rest is caller data: `/i/<trigger-id>/Patient/123` carries record identifiers. * Carry a trigger's custom path through provisioning, export and merge A path set in the app now survives a pull, and one written in `project.yaml` now deploys, so a project can be described end to end without opening the editor. An absent key means unchanged, so deploying a spec written before this existed cannot wipe a path. `project_id` and the bare-URL marker are refused in a trigger payload: they are server-owned, and `validate_extraneous_params/2` builds its allow-list from the schema, so adding them as fields would have silently started accepting them. Provisioning releases the paths of workflows that are hidden and disabled, both before the write, so one document can drop a workflow and claim its path in the same breath, and after it project-wide, since a checked-in yaml can still name the path of a workflow deleted since. Export, the JSON endpoint and the merge all drop a path the current rules would reject, so a document written before them cannot fail the import it is deployed into. A sandbox clone keeps its parent's path verbatim and answers on its own project's URL, so promoting one never moves the parent's live URL. * Name a webhook's URL in the trigger panel The webhook wizard gains a Custom URL path field. Typing a name shows what it will be stored as, the way the project form treats a project name, and the show panel's URL updates once the trigger is saved. The field only judges a path the user actually changed, so a value written before these rules existed does not block edits to anything else. Becoming a webhook counts as a change, since a path that meant nothing on a cron row is about to become a live URL and the server will check it. A path the server refused is not the trigger's, and for a duplicate it resolves to whichever workflow legitimately owns the name, so the panel falls back to the generated URL rather than offering one that posts into another workflow. The same goes for a legacy path holding a slash, which is stored verbatim but is not addressable. Applying a YAML or AI answer that never mentions the path keeps what the trigger holds, matching the way the provisioner reads an absent key. * Add the changelog entry * Hash a webhook's custom path in the workflow version A workflow whose only change was its endpoint name hashed the same as before, so anything comparing versions decided nothing had changed. The CLI's find-changed-workflows skipped it and openfn project deploy sent nothing. Costs existing workflows nothing. A nil serialises to an empty string and the parts are concatenated, so hashing the key does not move the hash of a workflow that has no path. @openfn/project skips undefined values for the same reason, which is what keeps the two implementations equal, and its parity test against a real Lightning hash still passes. Paired with OpenFn/kit#1511, which adds the key on that side. * Hash only a path the app would actually export Two problems with hashing custom_path verbatim, both found in review. ProvisioningJSON and ExportUtils drop a path the naming rules reject, so the CLI never receives one. Hashing it here left the app hash and the CLI hash disagreeing forever for exactly the grandfathered rows the migration exists to keep working, and every openfn project diff would have reported the workflow as changed with nothing to show for it. And Map.take is type blind, so a stale path on a cron or kafka row, which never served a URL and was never validated, moved the hash of a workflow nobody had touched. Both are gated now, and both directions are covered by a test that fails when the gate is removed.
josephjclark
reviewed
Sep 1, 2026
josephjclark
left a comment
josephjclark
left a comment
Collaborator
There was a problem hiding this comment.
Added a new test for v2 deploy, otherwise looks good.
I wanna hold release by a couple of hours while I look into somehing for Tuch
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short Description
Adds
custom_pathto the webhook trigger payload@openfn/deploysends, so a webhook endpoint can be named inproject.yaml. This is the CLI half of OpenFn/lightning#5105.Fixes #1516
Implementation Details
@openfn/deploynever sentcustom_path, so a name written inproject.yamlwas dropped on the way up. It carries it now.A path only travels when the spec mentions it. Leaving the key out means the server keeps whatever it has, so deploying a spec written before this existed will not wipe a path somebody set in the app. Writing it blank clears it, either as
''or as a barecustom_path:, which YAML reads as null. That is the rule the provisioner applies on the Lightning side.Two other things had to move or the feature does not work. The workflow version hash ignored the path, so a workflow whose only change was its endpoint name looked unchanged and
openfn project deployskipped it; both sides hash it now, and a workflow without a path hashes exactly as it did. Anddeployshowed a removal for every trigger field the payload does not carry, which it was never going to remove.The v2 project format swept a pulled path under
openfn:instead of keeping it besidewebhook_reply, so the two formats disagreed about where a webhook's path lives.QA Notes
Nothing here works until OpenFn/lightning#5105 ships. An older CLI against a newer Lightning is fine, and a newer CLI against an older Lightning sends a key the server ignores.
Once both are up: put
custom_path: facility-001on a webhook trigger inproject.yaml, runopenfn deploy, andPOST /i/<project-id>/facility-001should create a work order. Thenopenfn pulland the path should come back down.AI Usage
Please disclose whether you've used AI anywhere in this PR (it's cool, we just want to know!):
You can read more details in our Responsible AI Policy