From 8acdfca76c6b8950fddbd6a620bc6e78561d193c Mon Sep 17 00:00:00 2001 From: "Elias W. BA" Date: Fri, 4 Sep 2026 22:53:03 +0000 Subject: [PATCH] Kafka triggers are gone Removed in v2.18.2, but the page still walked people through configuring one and the intro still promised three trigger types. A short note stands where the section was, rather than nothing. People will search for this, and anyone who had a Kafka trigger needs to know it became a disabled webhook trigger rather than disappearing. Self-hosters get the environment variable warning from the changelog, since that one bites on upgrade. The Kafka input state section in the jobs docs goes too, along with the link that pointed at it. --- docs/build/triggers.md | 101 +++++------------------------------------ docs/jobs/state.md | 27 ----------- 2 files changed, 11 insertions(+), 117 deletions(-) diff --git a/docs/build/triggers.md b/docs/build/triggers.md index f447f1cbc131..8c2f4c0da04f 100644 --- a/docs/build/triggers.md +++ b/docs/build/triggers.md @@ -3,7 +3,7 @@ title: Triggers --- Triggers allow you to start the execution of Workflows automatically. They come -in three types: Cron triggers, Webhook Event, and Kafka triggers. +in two types: Cron triggers and Webhook Event triggers. ## Webhook Event Triggers @@ -241,98 +241,19 @@ fn(state => { }); ``` -## Kafka Triggers (beta) 🚧 +## Kafka Triggers -The Kafka trigger allows OpenFn users to build Workflows triggered by messages -published by a Kafka cluster. The triggers make use of Kafka consumer groups -that are set up on-demand to receive messages from a defined cluster then -converts them to `Input` dataclips that are used to initialize a Work Order. +Kafka triggers were removed in **v2.18.2**. A workflow can no longer be started +by consuming from a Kafka cluster. -:::info For self-hosted OpenFn deployments +Existing Kafka triggers were converted to **disabled webhook triggers** rather +than deleted, so the workflows they belonged to are intact. To keep one running, +point the sending system at the trigger's webhook URL and enable it. -Instance administrators have to enable Kafka for their instance by setting -`KAFKA_TRIGGERS_ENABLED=yes` in the environment variable. +:::caution For self-hosted OpenFn deployments -::: - -![Configuring Kafka Trigger](/img/configuring-kafka.webp) - -:::info What is Kafka? - -Apache Kafka® is an event streaming platform designed to handle high volumes of -data. Check out -[Kafka Docs](https://kafka.apache.org/documentation/#gettingStarted) to learn -more. +The `KAFKA_*` environment variables no longer do anything. If you're running +with `KAFKA_TRIGGERS_ENABLED` switched on, take a backup and switch it off +before upgrading. Stay on the previous release if you still need Kafka. ::: - -### Configuring a Kafka trigger for your workflow - -1. Create a new Workflow or open an existing Workflow in your Project -2. Click on the workflow's Trigger and change the trigger type to - `Kafka Consumer` in the `Trigger type` dropdown. -3. Fill out the required connection details: - -- **Hosts**: Provide the URL of the host(s) your trigger should listen to for - messages. -- **Topics**: Enter the topics your Kafka consumers should subscribe to. You - need at least one topic for a successful connection. -- **SSL**: Some Kafka clusters require SSL connection. If you are connecting to - an environment that requires SSL connection, select `Enable SSL`. -- **SSL Authentication**: Select the type of Authentication required for the - Kafka cluster. -- **Initial offset policy**: The intial offset dictates where the consumer - starts reading messages from a topic when it subscribes for the first time. - There are three possible options: `earliest` messages available, `latest` - messages available, or messages with a specific `timestamp` (e.g., - `1721889238000`). -- **Connect timeout**: The connect timeout specified in seconds (e.g., `30`) - represents how long the consumer should wait before timing out when attempting - to connect with a Kafka cluster. - -4. If you have not finished designing your Workflow or you're not ready to start - receiving messages from the Kafka cluster, please check the box to **disable - the trigger** until you're ready for message ingestion to begin. - -:::warning Disable the trigger during workflow design - -Once the required connection information is provided via the modal, the trigger -will _immediately_ start attempting to connect to the Kafka cluster. We advise -that the trigger is disabled until your workflow is ready to receive data from -the cluster for processing. **To stop the trigger from receiving and processing -messages, check the `Disable this trigger` checkbox at the bottom of the trigger -configuration modal.** - -::: - -Learn how the initial `state` (and `Input`) for Kafka-triggered Workflows gets -built [here](../jobs/state#kafka-triggered-runs). - -### Known "sharp edges" on the Kafka trigger feature - -We'll monitor bug/exception reports, perform user interviews and collect feature -requests during the beta to determine which of these rough-edges are worth -ironing out, and how to do so assuming that the community wants to support Kafka -triggers going forward. Please don't hesitate to reach out on -[community.openfn.org](https://community.openfn.org) to make your voice heard! - -1. Performance settings are out of the _end-user's_ control and can only be set - at instance-level. As there is quite a close relationship between cluster and - consumer settings, this may prove to be an obstacle as users will not be able - to tune their consumers to align with their individual clusters in large - multi-tenant deployments. -2. If a message could not be turned into a work order (due to persistence errors - or hitting the hard limit), these will not be visible to the end-user and may - be lost forever (i.e. the cluster thinks Lightning has them but it doesn't - and they eventually rotate off the cluster) -3. Errors are written to the log and to Sentry; nothing is visible to the - end-user. -4. Reprocessing dropped messages isn't practical if you are not writing failed - messages to some sort of persistent file storage. -5. If a consumer group is disconnected from a cluster long enough for the - cluster to forget the last message, we may not be able to prevent duplicates - making it through (as a result of instance-wide de-duplication settings). -6. We cannot provide concurrency _and_ honour message sequence that is based on - the Kafka message key mechanism. If a user wants to guarantee the Kafka - message sequence, they must enable 1-at-a-time processing on OpenFn by - turning their workflow concurrency down to 1. diff --git a/docs/jobs/state.md b/docs/jobs/state.md index e96962e4288f..74a29905209a 100644 --- a/docs/jobs/state.md +++ b/docs/jobs/state.md @@ -154,33 +154,6 @@ The input state will look something like this: } ``` -### Kafka triggered runs - -When a Kafka message is fetched by the trigger, the input state contains the -message body and information that can be used for auditing or recovery if -connections are lost or workorders fail. - -The input state looks like this: - -```js -{ - data: { // the message value - formId: "patient_enrollment", - name: "John Doe" - }, - request: { - "headers": [ - // kafka headers can be used to provide additional metadata - ], - "key": "", // the key assigned to the message by the publisher - "offset": 168321, - "partition": 1, - "topic": "fhir-data-pipes", - "ts": 1721889238000 // the kafka message timestamp - } -} -``` - ### Cron triggered runs When a run is triggered by a cron job, its input state will be the final state