diff --git a/docs/build/triggers.md b/docs/build/triggers.md index f447f1cbc13..8c2f4c0da04 100644 --- a/docs/build/triggers.md +++ b/docs/build/triggers.md @@ -3,7 +3,7 @@ title: Triggers --- Triggers allow you to start the execution of Workflows automatically. They come -in three types: Cron triggers, Webhook Event, and Kafka triggers. +in two types: Cron triggers and Webhook Event triggers. ## Webhook Event Triggers @@ -241,98 +241,19 @@ fn(state => { }); ``` -## Kafka Triggers (beta) 🚧 +## Kafka Triggers -The Kafka trigger allows OpenFn users to build Workflows triggered by messages -published by a Kafka cluster. The triggers make use of Kafka consumer groups -that are set up on-demand to receive messages from a defined cluster then -converts them to `Input` dataclips that are used to initialize a Work Order. +Kafka triggers were removed in **v2.18.2**. A workflow can no longer be started +by consuming from a Kafka cluster. -:::info For self-hosted OpenFn deployments +Existing Kafka triggers were converted to **disabled webhook triggers** rather +than deleted, so the workflows they belonged to are intact. To keep one running, +point the sending system at the trigger's webhook URL and enable it. -Instance administrators have to enable Kafka for their instance by setting -`KAFKA_TRIGGERS_ENABLED=yes` in the environment variable. +:::caution For self-hosted OpenFn deployments -::: - -![Configuring Kafka Trigger](/img/configuring-kafka.webp) - -:::info What is Kafka? - -Apache Kafka® is an event streaming platform designed to handle high volumes of -data. Check out -[Kafka Docs](https://kafka.apache.org/documentation/#gettingStarted) to learn -more. +The `KAFKA_*` environment variables no longer do anything. If you're running +with `KAFKA_TRIGGERS_ENABLED` switched on, take a backup and switch it off +before upgrading. Stay on the previous release if you still need Kafka. ::: - -### Configuring a Kafka trigger for your workflow - -1. Create a new Workflow or open an existing Workflow in your Project -2. Click on the workflow's Trigger and change the trigger type to - `Kafka Consumer` in the `Trigger type` dropdown. -3. Fill out the required connection details: - -- **Hosts**: Provide the URL of the host(s) your trigger should listen to for - messages. -- **Topics**: Enter the topics your Kafka consumers should subscribe to. You - need at least one topic for a successful connection. -- **SSL**: Some Kafka clusters require SSL connection. If you are connecting to - an environment that requires SSL connection, select `Enable SSL`. -- **SSL Authentication**: Select the type of Authentication required for the - Kafka cluster. -- **Initial offset policy**: The intial offset dictates where the consumer - starts reading messages from a topic when it subscribes for the first time. - There are three possible options: `earliest` messages available, `latest` - messages available, or messages with a specific `timestamp` (e.g., - `1721889238000`). -- **Connect timeout**: The connect timeout specified in seconds (e.g., `30`) - represents how long the consumer should wait before timing out when attempting - to connect with a Kafka cluster. - -4. If you have not finished designing your Workflow or you're not ready to start - receiving messages from the Kafka cluster, please check the box to **disable - the trigger** until you're ready for message ingestion to begin. - -:::warning Disable the trigger during workflow design - -Once the required connection information is provided via the modal, the trigger -will _immediately_ start attempting to connect to the Kafka cluster. We advise -that the trigger is disabled until your workflow is ready to receive data from -the cluster for processing. **To stop the trigger from receiving and processing -messages, check the `Disable this trigger` checkbox at the bottom of the trigger -configuration modal.** - -::: - -Learn how the initial `state` (and `Input`) for Kafka-triggered Workflows gets -built [here](../jobs/state#kafka-triggered-runs). - -### Known "sharp edges" on the Kafka trigger feature - -We'll monitor bug/exception reports, perform user interviews and collect feature -requests during the beta to determine which of these rough-edges are worth -ironing out, and how to do so assuming that the community wants to support Kafka -triggers going forward. Please don't hesitate to reach out on -[community.openfn.org](https://community.openfn.org) to make your voice heard! - -1. Performance settings are out of the _end-user's_ control and can only be set - at instance-level. As there is quite a close relationship between cluster and - consumer settings, this may prove to be an obstacle as users will not be able - to tune their consumers to align with their individual clusters in large - multi-tenant deployments. -2. If a message could not be turned into a work order (due to persistence errors - or hitting the hard limit), these will not be visible to the end-user and may - be lost forever (i.e. the cluster thinks Lightning has them but it doesn't - and they eventually rotate off the cluster) -3. Errors are written to the log and to Sentry; nothing is visible to the - end-user. -4. Reprocessing dropped messages isn't practical if you are not writing failed - messages to some sort of persistent file storage. -5. If a consumer group is disconnected from a cluster long enough for the - cluster to forget the last message, we may not be able to prevent duplicates - making it through (as a result of instance-wide de-duplication settings). -6. We cannot provide concurrency _and_ honour message sequence that is based on - the Kafka message key mechanism. If a user wants to guarantee the Kafka - message sequence, they must enable 1-at-a-time processing on OpenFn by - turning their workflow concurrency down to 1. diff --git a/docs/jobs/state.md b/docs/jobs/state.md index e96962e4288..74a29905209 100644 --- a/docs/jobs/state.md +++ b/docs/jobs/state.md @@ -154,33 +154,6 @@ The input state will look something like this: } ``` -### Kafka triggered runs - -When a Kafka message is fetched by the trigger, the input state contains the -message body and information that can be used for auditing or recovery if -connections are lost or workorders fail. - -The input state looks like this: - -```js -{ - data: { // the message value - formId: "patient_enrollment", - name: "John Doe" - }, - request: { - "headers": [ - // kafka headers can be used to provide additional metadata - ], - "key": "", // the key assigned to the message by the publisher - "offset": 168321, - "partition": 1, - "topic": "fhir-data-pipes", - "ts": 1721889238000 // the kafka message timestamp - } -} -``` - ### Cron triggered runs When a run is triggered by a cron job, its input state will be the final state