Update log collector docs for the fluentd to fluent-bit migration (EV-6164) - #2815
Closed
hjiawei wants to merge 3 commits into
Closed
Update log collector docs for the fluentd to fluent-bit migration (EV-6164)#2815hjiawei wants to merge 3 commits into
hjiawei wants to merge 3 commits into
Conversation
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview succeeded!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the next-version Calico Enterprise and Calico Cloud documentation to reflect the migration from the Fluentd log collector to Fluent Bit (calico-fluent-bit in calico-system), including updated filtering configuration examples, monitoring/metrics guidance, and component naming/namespace changes.
Changes:
- Replace Fluentd-specific resource names, namespaces, and architecture references with Fluent Bit equivalents across Enterprise and Cloud docs.
- Rewrite flow/DNS log filtering docs from Fluentd
<filter>syntax to Fluent Bit YAML filter lists, including the ConfigMap rename tofluent-bit-filtersand upgrade notes. - Update Prometheus monitoring docs and alert examples to Fluent Bit metrics and ServiceMonitor manifests, including new/different alert signals.
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| calico-enterprise/reference/component-resources/configure-resources.mdx | Updates LogCollector resource-override examples and verification commands to Fluent Bit. |
| calico-enterprise/reference/clis/calicoctl/cluster/diags.mdx | Updates diags output example to include calico-fluent-bit pods in calico-system. |
| calico-enterprise/reference/architecture/overview.mdx | Replaces Fluentd architecture section/link with Fluent Bit. |
| calico-enterprise/operations/monitor/prometheus/byo-prometheus.mdx | Updates BYO Prometheus scraping instructions for Fluent Bit and new ServiceMonitor reference. |
| calico-enterprise/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx | Retitles and updates alert examples from Fluentd metrics to Fluent Bit metrics. |
| calico-enterprise/operations/license-options.mdx | Updates license-expiration behavior text to reference Fluent Bit collector scale-down. |
| calico-enterprise/operations/comms/index.mdx | Updates certificates/services table entries to Fluent Bit-related names/namespaces. |
| calico-enterprise/observability/review-unused-network-policies.mdx | Updates example output to Fluent Bit-related policy naming. |
| calico-enterprise/observability/packetcapture.mdx | Updates packet capture retrieval steps to use calico-node pods in calico-system. |
| calico-enterprise/observability/kube-audit.mdx | Updates wording to reflect audit logs being consumed by Fluent Bit collector. |
| calico-enterprise/observability/elastic/overview.mdx | Updates Elastic logging overview to reference Fluent Bit installation/behavior. |
| calico-enterprise/observability/elastic/flow/filtering.mdx | Rewrites flow log filtering from Fluentd to Fluent Bit YAML list format; renames filters ConfigMap. |
| calico-enterprise/observability/elastic/dns/filtering-dns.mdx | Rewrites DNS log filtering docs to Fluent Bit YAML list format; renames filters ConfigMap and adds upgrade note. |
| calico-enterprise/_includes/content/_default-install.mdx | Updates default install table to mention Fluent Bit for log collection. |
| calico-enterprise/_includes/components/ReqsSys.js | Updates system requirements table wording for log storage. |
| calico-cloud/reference/component-resources/configure-resources.mdx | Updates LogCollector resource-override examples and verification commands to Fluent Bit. |
| calico-cloud/operations/monitor/prometheus/byo-prometheus.mdx | Updates BYO Prometheus scraping instructions for Fluent Bit and new ServiceMonitor reference. |
| calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx | Retitles and updates alert examples from Fluentd metrics to Fluent Bit metrics. |
| calico-cloud/operations/comms/index.mdx | Updates certificates/services table entries to Fluent Bit-related names/namespaces. |
| calico-cloud/observability/packetcapture.mdx | Updates packet capture retrieval steps to use calico-node pods in calico-system. |
| calico-cloud/observability/kube-audit.mdx | Updates wording to reflect audit logs being consumed by Fluent Bit collector. |
| calico-cloud/observability/elastic/overview.mdx | Updates Elastic logging overview to reference Fluent Bit installation/behavior. |
| calico-cloud/observability/elastic/flow/filtering.mdx | Rewrites flow log filtering from Fluentd to Fluent Bit YAML list format; renames filters ConfigMap. |
| calico-cloud/observability/elastic/dns/filtering-dns.mdx | Rewrites DNS log filtering docs to Fluent Bit YAML list format; renames filters ConfigMap and adds upgrade note. |
| calico-cloud/get-started/operator-checklist.mdx | Updates operator checklist steps/output to Fluent Bit pods/namespace and per-node component list. |
| calico-cloud/get-started/cc-arch-diagram.mdx | Updates Cloud architecture diagram table entries (component name + metrics port) for Fluent Bit. |
| calico-cloud/_includes/components/ReqsSys.js | Updates system requirements table wording for log storage. |
hjiawei
force-pushed
the
EV-6164-fluent-bit-docs
branch
from
July 2, 2026 06:30
5e860df to
f923e00
Compare
hjiawei
force-pushed
the
EV-6164-fluent-bit-docs
branch
2 times, most recently
from
July 15, 2026 23:56
af96fd7 to
9c9143a
Compare
hjiawei
marked this pull request as ready for review
July 15, 2026 23:56
Merged
5 tasks
hjiawei
force-pushed
the
EV-6164-fluent-bit-docs
branch
from
July 16, 2026 23:12
9c9143a to
c43021e
Compare
tianfeng92
approved these changes
Jul 17, 2026
tianfeng92
left a comment
tianfeng92
left a comment
Contributor
There was a problem hiding this comment.
Went through all of the command changes and verified them. LGTM!
Left some neat.
| | | NAME | AVAILABLE | PROGRESSING | DEGRADED | SINCE | | ||
| | --- | ----------------------------- | --------- | ----------- | -------- | ----- | | ||
| | 1 | apiserver | TRUE | FALSE | FALSE | 10m | | ||
| | 1 | `apiserver` | TRUE | FALSE | FALSE | 10m | |
Contributor
There was a problem hiding this comment.
Any reason to add ` to quote here? Seems to be redundant format.
Contributor
Author
There was a problem hiding this comment.
No reason — a stray formatting change from the edit. Reverted to plain apiserver in a6cd6ec.
| $[prodname] packet capture is implemented in a Kubernetes-native way so you can troubleshoot service/application connectivity issues and performance issues. You can start a packet capture in the web console Service Graph, or using the CLI. | ||
|
|
||
| Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, daemonset, statefulset, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. | ||
| Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, DaemonSet, StatefulSet, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. |
Contributor
There was a problem hiding this comment.
Suggested change
| Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, DaemonSet, StatefulSet, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. | |
| Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, ReplicaSet, DaemonSet, StatefulSet, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. |
| ### Packet capture in Service Graph | ||
|
|
||
| 1. Select an endpoint from the service graph (for example, namespace, service, replica set, daemonset, statefulset, or pod), right-click, and select **Initiate packet capture**. | ||
| 1. Select an endpoint from the service graph (for example, namespace, service, replica set, DaemonSet, StatefulSet, or pod), right-click, and select **Initiate packet capture**. |
Contributor
There was a problem hiding this comment.
Suggested change
| 1. Select an endpoint from the service graph (for example, namespace, service, replica set, DaemonSet, StatefulSet, or pod), right-click, and select **Initiate packet capture**. | |
| 1. Select an endpoint from the service graph (for example, namespace, service, ReplicaSet, DaemonSet, StatefulSet, or pod), right-click, and select **Initiate packet capture**. |
| $[prodname] packet capture is implemented in a Kubernetes-native way so you can troubleshoot service/application connectivity issues and performance issues. You can start a packet capture in the web console Service Graph, or using the CLI. | ||
|
|
||
| Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, daemonset, statefulset, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. | ||
| Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, DaemonSet, StatefulSet, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. |
Contributor
There was a problem hiding this comment.
Suggested change
| Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, DaemonSet, StatefulSet, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. | |
| Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, ReplicaSet, DaemonSet, StatefulSet, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. |
| ### Packet capture in Service Graph | ||
|
|
||
| 1. Select an endpoint from the service graph (for example, namespace, service, replica set, daemonset, statefulset, or pod), right-click, and select **Initiate packet capture**. | ||
| 1. Select an endpoint from the service graph (for example, namespace, service, replica set, DaemonSet, StatefulSet, or pod), right-click, and select **Initiate packet capture**. |
Contributor
There was a problem hiding this comment.
Suggested change
| 1. Select an endpoint from the service graph (for example, namespace, service, replica set, DaemonSet, StatefulSet, or pod), right-click, and select **Initiate packet capture**. | |
| 1. Select an endpoint from the service graph (for example, namespace, service, ReplicaSet, DaemonSet, StatefulSet, or pod), right-click, and select **Initiate packet capture**. |
| * *Breaking change:* S3 archive object keys change from Fluentd's flat layout (`<bucketPath>/flows20260101_<n>.gz`) to directory-style keys (`<bucketPath>/flows/20260101_<uuid>.gz`), and non-cluster-host flows are archived under a separate `non_cluster_flows/` directory. Update downstream tooling that relies on the old key patterns. | ||
| * Log buffering moves from Fluentd's in-memory buffers to Fluent Bit filesystem storage under `/var/log/calico/calico-fluent-bit/` on each node. Buffered-but-unsent chunks now survive pod restarts, and node disk usage grows accordingly (capped per output by `storage.total_limit_size`). | ||
| * Syslog forwarding no longer applies Fluentd's 1024-byte packet-size default. When `spec.additionalStores.syslog.packetSize` is unset, Fluent Bit uses a 2048-byte default for the RFC5424 messages the operator renders, which reduces truncation of longer log lines. Set `packetSize` to cap the message size explicitly; messages larger than the cap are truncated. | ||
|
|
Contributor
There was a problem hiding this comment.
Possibly missing following breaking changes:
- Windows DaemonSet rename — tigera-fluentd-windows → calico-fluent-bit-windows. The rename bullet only names the Linux resources; Windows users grep/monitor by the old name too.
- Metrics service rename — fluentd-metrics/fluentd-metrics-windows → single calico-fluent-bit-metrics service with label k8s-app: calico-fluent-bit. Anyone with a custom ServiceMonitor/scrape config needs the new service name and label, not just new metric names/port (the current bullet only says dashboards/alerts).
- EKS log forwarder init container removed — old deployment had an eks-log-forwarder-startup init container; the new one has only the main container. Resource overrides in eksLogForwarderDeployment targeting the init container name silently stop applying. Niche (EKS + overrides), but it's a silent behavior change.
Contributor
Author
There was a problem hiding this comment.
Good catch. Verified all three against the operator render code and added them in 4586a5b: the rename bullet now names calico-fluent-bit-windows, the metrics bullet covers the metrics service renames, and a new bullet covers the removed eks-log-forwarder-startup init container.
One correction: the operator still renders one metrics service per OS (metricsService() in pkg/render/logcollector/daemonset.go), so the services are calico-fluent-bit-metrics and calico-fluent-bit-metrics-windows rather than a single service. The release-notes bullet names both.
Contributor
Author
|
This change is targeting Calico Enterprise v3.24 EP3 release. |
hjiawei
force-pushed
the
EV-6164-fluent-bit-docs
branch
2 times, most recently
from
August 6, 2026 04:49
f8e7863 to
4d32aa2
Compare
…-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Document the fluentd -> Fluent Bit migration (tigera/operator#4910) in the next-version Calico Enterprise release notes: - New feature entry for the Fluent Bit log collector. - Deprecated: the fluentdDaemonSet override, superseded by calicoFluentBitDaemonSet. - Upgrade notes with the breaking changes: tigera-fluentd namespace/resource rename to calico-system, fluentd-filters -> fluent-bit-filters ConfigMap, fluentbit_* Prometheus metrics on port 2020, and the new S3 archive key layout, plus the non-breaking moves to filesystem buffering and the larger syslog packet-size default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The operator now folds a user-supplied CA from a splunk-ca ConfigMap into the log collector's trusted bundle, the same way syslog-ca works for syslog, so Splunk HEC endpoints with self-signed certificates can be used. Add the step to the Splunk archiving instructions and a release-note line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hjiawei
force-pushed
the
EV-6164-fluent-bit-docs
branch
from
August 18, 2026 17:11
4d32aa2 to
1c47342
Compare
This was referenced Sep 3, 2026
Collaborator
|
Thanks for this, Jiawei, and sorry it sat for so long. I parked it in July while I was cutting the 3.24 docs, and then it slipped. I have picked the work up and re-cut it, so I am closing this one as superseded:
Both commits are cherry-picked, so you are still the author, and I have carried over @tianfeng92's verification of the commands. Tracking under https://tigera.atlassian.net/browse/DOCS-3028. The split is the review comment I had drafted but never submitted: the release notes carried an unusually large amount of specific information for one feature, so keeping the documentation changes and the release-specific content in separate PRs makes both reviewable. Then in the release notes I gave each upgrade note one action and one link, and moved the mechanism behind it onto the page that documents it. Filesystem buffering and its per-destination caps now live in the log collection overview, the S3 object key layout and the syslog packet-size default in Archive logs, and the deprecated override field and its container-name aliases in Configure resource requests and limits. The other change worth flagging is the filter migration note. It was a note at the end of each filtering page, which a reader would only reach after their filters had already stopped working. It is now a Before you begin section that says who it affects, that the rewrite happens before the upgrade, and which command surfaces the warning your controller raises for a key that does not parse. I also listed the four filters the image ships as an exhaustive set with a link each, since FLB_MINIMAL means anything else will not load, and noted that filters apply to Linux nodes only. Two things I could not settle from the repos, and would like your confirmation on:
Everything else I re-verified against the merged code rather than the pre-merge branch, since both PRs landed after you opened this: the resource, container, secret, service, and policy names, the metrics port and path, the ServiceMonitor selector and label, the S3 key format, the removed EKS init container, the buffer path and its caps, and the parse-failure warning. All of it held. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Calico Enterprise v3.24 EP3 will replace the fluentd log collector with Fluent Bit (
calico-fluent-bitincalico-system; tigera/operator#4910, tigera/calico-private#12538). This updates the next-version Calico Enterprise and Calico Cloud docs to match:<filter>syntax to Fluent Bit YAML filter lists; the user-filters ConfigMap is renamedfluentd-filters→fluent-bit-filters; an upgrade note explains that old fluentd-syntax filters are not read or translated and that the operator raises atigera statuswarning naming any key that fails to parse.fluentbit_output_chunk_available_capacity_percent, plus a new dropped-chunks alert onfluentbit_output_retries_failed_total. (File path kept for URL stability — happy to rename + add a redirect if preferred.)/api/v2/metrics/prometheus, guarded by theallow-calico-fluent-bitpolicy) and points at the newfluent-bit-metrics-service-monitor.yamlsample.tigera-fluentd→calico-system, daemonset/label/secret renames, packet capture retrieval via the calico-node pod,calicoFluentBitDaemonSetresource-override examples, diags output, architecture overviews, Calico Cloud arch diagram ports and operator checklist.Not touched: versioned docs,
releases.json(historical), and the generated installation API reference (_api.mdxregenerates from the operator release — until then the#fluentbitdaemonsetanchor is dangling, which the build ignores peronBrokenAnchors: ignore).Validation:
yarn build-nextreports exactly the same three broken links asmain— this PR introduces no new ones. Every resource/label/metric name was verified against the operator branch (hjiawei/operator:fluent-bit-deploy) or upstream Fluent Bit docs.Should land together with the calico-private/operator PRs for the release that ships the migration.
🤖 Generated with Claude Code