DOCS-3028: Update log collector docs for the Fluentd to Fluent Bit migration - #3004
Merged
Merged
Conversation
…-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The operator now folds a user-supplied CA from a splunk-ca ConfigMap into the log collector's trusted bundle, the same way syslog-ca works for syslog, so Splunk HEC endpoints with self-signed certificates can be used. Add the step to the Splunk archiving instructions and a release-note line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move the Fluentd filter migration into a Before you begin section on both filtering pages, so a reader who already has filters sees it before the task rather than after logs go missing. Name the command that surfaces the operator warning, and note that filters apply to Linux nodes only. List the four filters the log collector ships as an exhaustive set, and link each one, so a reader does not write a filter the image cannot load. Document the operational detail on the pages that own it: filesystem buffering and its per-destination caps in the log collection overview, the S3 object key layout and the syslog packet-size default in Archive logs, and the deprecated override field and container-name aliases in Configure resource requests and limits.
Use the ConfigMap and DaemonSet spellings Vale expects, and put the resource and field names in the private-CA steps into code formatting. The syslog step gets the same treatment as the new Splunk step beside it, so the two adjacent steps read the same way.
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview succeeded!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A few documentation details are misleading/unclear (notably link text vs. anchor targets and note wording) and should be corrected to avoid confusing readers during the Fluentd → Fluent Bit transition.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates Calico Enterprise and Calico Cloud next documentation to reflect the migration from the Fluentd log collector to Fluent Bit (calico-fluent-bit in calico-system), aligning examples, component names, metrics/alerts, and operational procedures with the new collector.
Changes:
- Rewrites flow/DNS log filtering docs from Fluentd
<filter>syntax to Fluent Bit YAML filter lists and renames the filters ConfigMap tofluent-bit-filters. - Updates Prometheus/BYO Prometheus and metrics pages for Fluent Bit (HTTP metrics endpoint on port 2020, updated alert metrics and names).
- Sweeps related docs to reflect namespace/resource/secret/pod renames and updated operational procedures (diags, packet capture retrieval, architecture).
File summaries
| File | Description |
|---|---|
| calico-enterprise/reference/component-resources/configure-resources.mdx | Updates LogCollector override examples and verification commands for calico-fluent-bit/calico-system, including alias/deprecation notes. |
| calico-enterprise/reference/clis/calicoctl/cluster/diags.mdx | Updates sample calicoctl diags output to show Fluent Bit pods in calico-system. |
| calico-enterprise/reference/architecture/overview.mdx | Renames Fluentd section to Fluent Bit and adjusts related component descriptions/formatting. |
| calico-enterprise/operations/monitor/prometheus/byo-prometheus.mdx | Updates BYO Prometheus instructions for Fluent Bit metrics scraping (HTTP endpoint, new ServiceMonitor name, network policy note). |
| calico-enterprise/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx | Retitles and updates metrics/alerts from Fluentd to Fluent Bit, including new alert and metric names. |
| calico-enterprise/operations/license-options.mdx | Updates license-expiration behavior wording to reference Fluent Bit scaling down. |
| calico-enterprise/operations/comms/index.mdx | Updates secret table (notably calico-fluent-bit-tls) and formats owner references consistently. |
| calico-enterprise/observability/review-unused-network-policies.mdx | Updates unused-policy examples for the new Fluent Bit policy/namespace. |
| calico-enterprise/observability/packetcapture.mdx | Updates packet capture retrieval steps to use calico-system/calico-node pods and Kubernetes resource name casing. |
| calico-enterprise/observability/kube-audit.mdx | Updates audit-log instructions to reference the Fluent Bit log collector. |
| calico-enterprise/observability/elastic/overview.mdx | Updates default log collector description to Fluent Bit and documents filesystem buffering behavior/caps. |
| calico-enterprise/observability/elastic/flow/filtering.mdx | Adds “Before you begin” migration guidance and rewrites filter examples for Fluent Bit YAML. |
| calico-enterprise/observability/elastic/dns/filtering-dns.mdx | Adds “Before you begin” migration guidance and rewrites DNS filter examples for Fluent Bit YAML. |
| calico-enterprise/observability/elastic/archive-storage.mdx | Documents S3 key layout changes, syslog defaults, and adds Splunk private-CA ConfigMap step. |
| calico-enterprise/_includes/content/_default-install.mdx | Updates default install summary to reference Fluent Bit log collection. |
| calico-enterprise/_includes/components/ReqsSys.js | Updates requirements table wording to remove Fluentd datastore reference. |
| calico-cloud/reference/component-resources/configure-resources.mdx | Mirrors LogCollector override example updates for Cloud docs. |
| calico-cloud/operations/monitor/prometheus/byo-prometheus.mdx | Mirrors BYO Prometheus Fluent Bit updates for Cloud docs. |
| calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx | Mirrors Fluent Bit metrics/alerts updates for Cloud docs. |
| calico-cloud/operations/comms/index.mdx | Updates secret table (notably calico-fluent-bit-tls) and formats owner references consistently. |
| calico-cloud/observability/packetcapture.mdx | Mirrors packet capture retrieval updates for Cloud docs. |
| calico-cloud/observability/kube-audit.mdx | Mirrors audit-log wording update for Cloud docs. |
| calico-cloud/observability/elastic/overview.mdx | Mirrors Fluent Bit default collector + buffering behavior documentation for Cloud docs. |
| calico-cloud/observability/elastic/flow/filtering.mdx | Mirrors Fluent Bit flow filtering migration guidance and YAML examples for Cloud docs. |
| calico-cloud/observability/elastic/dns/filtering-dns.mdx | Mirrors Fluent Bit DNS filtering migration guidance and YAML examples for Cloud docs. |
| calico-cloud/observability/elastic/archive-storage.mdx | Mirrors S3 key layout and syslog default packet size updates for Cloud docs. |
| calico-cloud/get-started/operator-checklist.mdx | Updates troubleshooting/checklist examples for Fluent Bit pods/namespaces and related terminology. |
| calico-cloud/get-started/cc-arch-diagram.mdx | Updates architecture diagram table to replace Fluentd with Fluent Bit and updates associated ports. |
| calico-cloud/_includes/components/ReqsSys.js | Updates requirements table wording to remove Fluentd datastore reference. |
Review details
- Files reviewed: 29/29 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+553
to
+557
| The [LogCollector](../../reference/installation/api.mdx#logcollector) CR provides a way to configure resources for CalicoFluentBitDaemonSet, EKSLogForwarderDeployment. | ||
|
|
||
| ### FluentdDaemonSet. | ||
| ### CalicoFluentBitDaemonSet. | ||
|
|
||
| To configure resource specification for the [FluentdDaemonSet](../../reference/installation/api.mdx#fluentddaemonset), patch the LogCollector CR using the below command: | ||
| To configure resource specification for the [CalicoFluentBitDaemonSet](../../reference/installation/api.mdx#logcollector), patch the LogCollector CR using the below command: |
Comment on lines
+238
to
+242
| :::note | ||
|
|
||
| Skip this step if publicCA bundle is good enough to verify the server certificates. | ||
|
|
||
| ::: |
Comment on lines
+586
to
+590
| The [LogCollector](../../reference/installation/api.mdx#logcollector) CR provides a way to configure resources for CalicoFluentBitDaemonSet, EKSLogForwarderDeployment. | ||
|
|
||
| ### FluentdDaemonSet. | ||
| ### CalicoFluentBitDaemonSet. | ||
|
|
||
| To configure resource specification for the [FluentdDaemonSet](../../reference/installation/api.mdx#fluentddaemonset), patch the LogCollector CR using the below command: | ||
| To configure resource specification for the [CalicoFluentBitDaemonSet](../../reference/installation/api.mdx#logcollector), patch the LogCollector CR using the below command: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Calico Enterprise 3.24 EP3 replaces the Fluentd log collector with Fluent Bit, deployed as calico-fluent-bit in calico-system. This updates the Calico Enterprise and Calico Cloud next docs to match.
This supersedes #2815, which Jiawei opened in July. Jiawei's two commits are cherry-picked with his authorship intact, and tianfeng92's verification of the commands carries over. The release notes and upgrade notes are split out into a stacked follow-up, so this PR is only the documentation changes.
What is here:
What changed from #2815:
Every resource, label, metric, port, and default in these pages was re-verified against merged code rather than the pre-merge branch #2815 was written from: tigera/operator#4910 and tigera/calico-private#12538.
Not touched: versioned docs, releases.json, and the generated installation API reference. _api.mdx regenerates from the operator release at the EP3 cut.
Validation: yarn build-next completes with no warnings and no broken links. Vale reports nothing on the changed lines. The five new links to the Fluent Bit documentation all resolve.
Pages to review on the deploy preview:
The Calico Cloud twins of these pages carry the same changes, so one product is enough for the repeated renames.
Issue: https://tigera.atlassian.net/browse/DOCS-3028