OpenTelemetry on Kubernetes: deploy node, cluster, and identity in that order

You install the OpenTelemetry Collector Helm chart, see the Pods enter Running, and assume Kubernetes is now observable.

It is not.

Kubernetes monitoring works best when Collector topology follows where each signal actually exists:

• Node-local data such as kubelet metrics, host metrics, and container logs belongs on every node.
• Cluster-wide state such as Pod phases and node conditions belongs to one active cluster-level Collector.
• Application telemetry should be enriched with workload identity before network hops make association ambiguous.

Ask where each signal exists before you ask which Collector shape to run.

For most production setups, that leads to a simple baseline:

1. a DaemonSet for node-local collection;
2. a Deployment for cluster state and events;
3. workload identity verified at the node layer;
4. an optional gateway only when you can name the policy it centralizes.

This article uses the opentelemetry-collector chart 0.167.0, which ships Collector 0.157.0, with the opentelemetry-collector-k8s distribution image.

1. Start with signal locality

The Collector chart gives you a Kubernetes workload and a starting pipeline. It does not decide:

• which data you need;
• where that data exists;
• how it reaches your backend; or
• whether telemetry can be connected to the workload that emitted it.

A useful mental model is:

The split is straightforward:

• Node-local
  • kubeletstats
  • hostmetrics
  • filelog
  • local OTLP reception
• Cluster-wide
  • k8s_cluster
  • Kubernetes events and selected API objects
• Application-emitted
  • traces
  • metrics
  • logs sent by SDKs or exporters

The OpenTelemetry component deployment matrix is worth reading before adding more receivers.

2. Run a DaemonSet for data that lives on a node

A DaemonSet puts one Collector on every eligible node. That makes it the natural home for telemetry that is physically or logically node-local:

• kubeletstats for node, Pod, and container metrics;
• hostmetrics for host-level signals;
• filelog for Kubernetes container logs on the local filesystem; and
• OTLP receivers when applications send to a Collector on the same node.

A baseline values file looks like this:

# values-agent.yaml
# chart 0.167.0, Collector 0.157.0

mode: daemonset

image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s

presets:
  kubeletMetrics:
    enabled: true
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true

hostNetwork: true

service:
  enabled: false

ports:
  otlp:
    hostPort: null
  otlp-http:
    hostPort: null
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

extraEnvsFrom:
  - secretRef:
      name: otel-backend-credentials

config:
  receivers:
    jaeger: null
    zipkin: null
    kubeletstats:
      insecure_skip_verify: true

  exporters:
    debug: null
    otlp_http:
      endpoint: ${env:BACKEND_OTLP_ENDPOINT}
      headers:
        authorization: ${env:BACKEND_API_KEY}

  service:
    pipelines:
      traces:
        receivers: [otlp]
        exporters: [otlp_http]
      metrics:
        exporters: [otlp_http]
      logs:
        exporters: [otlp_http]

Two chart behaviors matter here.

First, null removes a default component. If you remove a component that a default pipeline still references, you must update that pipeline too.

Second, Helm replaces lists rather than merging them. If you touch a pipeline list, spell out the complete list you want.

A useful rule is:

Remove a default component and you own every pipeline that referenced it.

Put the memory limiter first

With the Kubernetes attributes preset enabled, the rendered processor order may place k8s_attributes before memory_limiter.

That is not ideal.

The memory limiter should run early so it can reject data and apply backpressure before downstream processing allocates more memory. At the same time, Kubernetes enrichment must happen before batch, because batching can destroy the per-connection context used for workload association.

Both constraints fit:

config:
  service:
    pipelines:
      traces:
        processors: [memory_limiter, k8s_attributes, batch]
      metrics:
        processors: [memory_limiter, k8s_attributes, batch]
      logs:
        processors: [memory_limiter, k8s_attributes, batch]

The limiter runs first, enrichment still happens before batching, and association context survives.

Treat kubelet TLS as something to verify, not assume

On the test cluster used for this article, the kubelet served a certificate the Collector did not trust. The result was deceptively quiet: the Collector Pod was Running, health checks were green, RBAC was correct, and kubelet metrics were still absent.

This setting made collection work in that environment:

config:
  receivers:
    kubeletstats:
      insecure_skip_verify: true

Do not treat that as a universal production recommendation. Some managed clusters present certificates that are already trusted.

The lesson is simpler: verify that kubelet metrics actually arrive. Do not use Collector Pod status as proof that the scrape works.

Expose only the ports you intend to use

A DaemonSet deserves careful port handling because a host-level bind can create listeners on every node.

If you want more granular control than hostNetwork: true, bind only the OTLP ports you need:

mode: daemonset

hostNetwork: false

service:
  enabled: false

ports:
  otlp:
    enabled: true
    containerPort: 4317
    hostPort: 4317
  otlp-http:
    enabled: true
    containerPort: 4318
    hostPort: 4318

Applications can then send to the Collector on their own node:

env:
  - name: NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

  - name: POD_UID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid

  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://$(NODE_IP):4318

  - name: OTEL_RESOURCE_ATTRIBUTES
    value: k8s.pod.uid=$(POD_UID)

Explicitly setting k8s.pod.uid gives the attributes processor a stable workload identifier that does not depend on what happened to the source IP on the network path.

3. Run one active cluster Collector for cluster state

A node Collector cannot tell you how many Pods are Pending or which nodes are NotReady.

That state lives in the Kubernetes API.

The k8s_cluster receiver reads cluster-level metrics and entity updates. It should have one active owner per cluster; otherwise multiple active instances can emit duplicate data.

A separate Deployment keeps that responsibility explicit:

# values-cluster.yaml
# chart 0.167.0, Collector 0.157.0

mode: deployment
replicaCount: 1

image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s

presets:
  clusterMetrics:
    enabled: true
  kubernetesEvents:
    enabled: true

extraEnvsFrom:
  - secretRef:
      name: otel-backend-credentials

config:
  exporters:
    debug: null
    otlp_http:
      endpoint: ${env:BACKEND_OTLP_ENDPOINT}
      headers:
        authorization: ${env:BACKEND_API_KEY}

  service:
    pipelines:
      traces:
        exporters: [otlp_http]
      metrics:
        exporters: [otlp_http]
      logs:
        exporters: [otlp_http]

The apparently pointless traces pipeline is intentional.

The chart defines traces, metrics, and logs pipelines by default, and those pipelines reference the default debug exporter. Once debug: null removes that exporter, every pipeline that referenced it must be updated or the Collector will not start.

Keep node-local receivers out of the Deployment

Do not put kubeletMetrics or logsCollection on this Deployment.

A Deployment lands on one node. Collecting node-local data from there would give you one node’s view while pretending to be a cluster-level workload.

Likewise, workload association belongs at the node layer. Cluster-state records are already about Kubernetes objects and carry identity from the receiver that read them.

Scale with leader election, not duplicate collection

If you increase the cluster Collector replica count, only one replica should actively collect cluster state.

Conceptually:

The chart can use leader election for this role. The important invariant is not “one Pod.” It is one active cluster-state owner.

4. Prove workload identity before adding more topology

Kubernetes metadata is what lets a trace, metric, and log answer the same operational question.

The Kubernetes attributes processor enriches telemetry only after it associates that telemetry with a Pod.

That association can come from a resource attribute such as a Pod IP or UID, or from the incoming connection address.

The connection is the fragile option because it is only useful when the sender address still identifies the original workload.

Different signal paths behave differently:

• File logs: filelog reads bytes from disk, so there is no sender connection to inspect. Identity comes from the Kubernetes log file path.
• Application telemetry: a local node Collector can associate the workload while the connection still represents the application.
• Gateway-forwarded telemetry: the gateway sees the forwarding Collector as the sender, not the original application.

That is why enrichment belongs close to the workload.

If an agent forwards telemetry to a gateway, attach workload identity at the agent first and let those resource attributes travel with the record.

For one real workload, verify that a representative trace, metric, and log all carry the expected:

• service.name and environment;
• namespace and Pod name or UID;
• workload identity such as Deployment or StatefulSet; and
• node identity where relevant.

If those records do not connect, you have ingested data that will be much harder to use during an incident.

Sidecars deserve special care. Do not assume a sidecar Collector can reconstruct sibling-container identity after the fact. When needed, inject known identity with the Kubernetes Downward API.

5. Add a gateway only when it has a job

Everything above works without a gateway.

That is a feature.

A gateway is a central Collector that receives telemetry from node Collectors or applications, applies shared processing, and exports to the backend.

Add one when you can name the policy it centralizes.

Two common reasons are:

• Shared policy across workloads or backends: filtering, redaction, routing, or common processing belongs in one central pipeline.
• One egress and credential boundary: export credentials and backend access live in one place instead of on every node.

Tail-based sampling is another valid reason because all spans for a trace must reach the same decision-making Collector. That introduces its own routing and scaling constraints and deserves separate treatment.

A gateway changes the data path:

The important rule is unchanged:

Identity must be settled before the gateway.

A gateway that tries to infer workload identity from the forwarding agent’s connection sees the wrong sender.

Gateway replicas are a scaling decision. They have nothing to do with the single-active-owner rule for cluster-state receivers.

6. Five checks that catch a bad baseline early

Do these before adding more receivers, labels, discovery rules, or dashboards.

The first one does not require a running cluster:

helm template otel-agent open-telemetry/opentelemetry-collector \
  --version 0.167.0 \
  -f values-agent.yaml

Read the rendered ConfigMap and verify that every component named in a pipeline is defined.

Then test the running architecture:

1. Node coverage
   Verify that every intended node has a node Collector. Account for taints and scheduling rules rather than assuming the DaemonSet runs everywhere.

2. Cluster-state uniqueness
   Verify that only one k8s_cluster receiver is active, or that leader election has selected one active owner.

3. Backend delivery
   Prove that your configured exporter delivers a representative trace, metric, and log. A healthy Collector process is not the same thing as successful export.

4. Workload association
   Verify that the trace, metric, and log from one real workload carry the expected Kubernetes identity.

5. Optional feature scope
   If you later enable discovery, object watches, or additional receivers, prove the exact resource or workload they are meant to cover and nothing broader.

These checks validate the architecture rather than any individual Collector component.

If a signal disappears after it has entered a working architecture, debug it hop by hop across receivers, processors, queues, and exporters.

The baseline to remember

The simplest reliable OpenTelemetry topology on Kubernetes follows signal locality:

• DaemonSet: collect data that lives on each node and attach workload identity while the local context still exists.
• Deployment: collect cluster-wide state under one active owner.
• Gateway: add only when you can name a shared policy, scaling need, or egress boundary that justifies another hop.
• Verification: prove telemetry delivery and workload association instead of trusting Pod status.

The Collector chart gives you primitives.

The architecture comes from deciding where the data lives.

Sources

• OpenTelemetry Kubernetes Getting Started
• OpenTelemetry with Kubernetes
• OpenTelemetry Collector Chart
• Collector Configuration
• Collector internal telemetry
• Agent deployment pattern
• Gateway deployment pattern
• OpenTelemetry Demo Docs
• OTLP exporter configuration
• Telemetrygen README