Logs Implementation Guide

Overview

This path takes a service from uncollected log lines to logs that investigators can search, parse, and pivot to the trace that produced them. Follow the levels in order. First prove that logs arrive once with stable identity. Then standardize their route and library output. Use Grok for legacy text formats and where string contains relevant parsable data. Finally, add trace correlation when traces already exist.

Application stdout / file / journald
                 │
                 ▼
      OpenTelemetry Collector
                 │
                 ├─► Tsuga Logs: search, patterns, and investigation
                 │
Active trace context ── trace_id + span_id ──► correlated log records
                                               │
                                               ▼
                                           Tsuga Traces

Level 5 assumes traces already reach Tsuga. Instrument the application before using that level if they do not. Deploy the collection tier through the route that owns the host or cluster running the log source.

Before starting

• Agree the services, environments, log sources, and owners in scope.
• Decide which collection boundary owns each source: node agent for Kubernetes container logs, host Collector for VM logs, or an application/sidecar Collector for direct OTLP logs.
• Confirm the customer’s data-handling rules. Suppress secrets and sensitive values at the application wherever possible. Collector redaction is a safety net, not the primary control.
• Decide the stable resource identity used on all signals: service.name, deployment.environment.name, and, where already used, service.namespace and service.version.
• Do not collect an identical source through two routes. Duplicate log ingestion inflates error counts and creates confusing investigations.

Level 1 - Collect logs

Start with the smallest possible proof: one known source, one service, and one environment. The aim is not full parsing or correlation yet. It is a searchable test record with correct identity and a healthy collection path.

Choose the receiver according to the source that already owns the logs:

• Kubernetes container stdout and stderr: collect from a node agent. Preserve cluster, namespace, pod, container, node, and workload context.
• VM logs: collect files or journald through a host Collector. Choose one where journald mirrors an application file log. Collecting both duplicates events.
• Applications that can emit OpenTelemetry logs: send them to a local or sidecar Collector. Do not expose an OTLP receiver publicly merely for log collection.
• Existing platform routes such as Fluent Bit, FireLens, or a cloud event stream: receive the forwarded records in a gateway Collector when that is the platform standard.

Set service.name and deployment.environment.name before rollout. For platform-collected logs, enrich the source context that makes the record actionable, then apply any filtering or redaction, and batch near the exporter.

Validate from the sender outward:

1. Confirm the Collector is healthy and has no persistent receiver, queue, or export errors.
2. Emit one unique, harmless log message after the Collector is ready.
3. Search for the exact string in Tsuga.
4. Confirm its timestamp, context.service.name, environment, and source context.

Exit criteria: a unique log line is searchable within the expected delivery delay. It has the agreed identity. The source is collected exactly once.

References: Collector configuration and the OpenTelemetry logs data model.

Level 2 - Routes and libraries

Once a source is reliable, make its format and route intentional. Use structured JSON so timestamps, levels, messages, and fields survive transport. Keep the existing logging library unless a change has a clear operational benefit.

Configure one JSON object per line with:

• a timezone-aware timestamp.
• severity or level.
• stable message text.
• structured business and diagnostic fields.
• trace_id and span_id if correlation is already enabled.

Use separate routes for application JSON, access logs, system logs, and multiline exceptions. Route on stable metadata such as file path, container identity, namespace, source tag, or a known JSON field. Do not route on message text that can change in a release.

Routing happens in two places, and each source needs one deliberate owner for each:

• In the Collector, where receivers and processors assemble, drop, or redact records before they leave the customer’s network. Suppress sensitive values here or earlier.
• In Tsuga log routes, which apply ordered, team-owned processing at ingest. Matching routes run in rank order, and later routes see earlier changes. Review rank and ownership before adding a route to a shared source.

Document each route’s owner, schema, data-handling policy, and multiline behavior. A stack trace must remain one log event. Test representative info and error messages through the same route.

When the service also sends traces, initialize OpenTelemetry before the logger and framework. Test worker threads, scheduled jobs, and queue consumers separately because they may not inherit request context.

Exit criteria: each source has one named route and owner. Application logs are structured and distinguishable from platform logs. Multiline errors stay together. Sensitive values have an agreed suppression or redaction policy.

Level 3 - Grok parsing

Use Grok parsing only for legacy applications, infrastructure logs, access logs, or vendor products that cannot emit structured logs. New services should use a stable structured log contract.

Before writing a pattern, collect approved samples for normal, error, exception, optional, and malformed lines. Keep secrets and sensitive customer data out of documentation and tickets. Document the route, timestamp and timezone, level vocabulary, message boundary, optional fields, and pattern owner.

Extract only the fields that investigators need: timestamp, severity, message, and a small number of stable diagnostic attributes. Preserve the raw body for diagnosis. Avoid extracting unbounded values such as IDs, raw URLs, user input, and payload fragments as indexed fields. They increase cardinality and data-handling risk.

Run the pattern in a Tsuga log route when the owning team can parse at ingest. Use a Collector parser when the record needs reshaping or redaction before it leaves the customer’s network. Do not parse the same field in both places.

Parsing fails open. An unmatched line keeps its original body, remains searchable, and contributes to a parse-failure signal. Do not drop unmatched errors. Use Preview, then test valid, invalid, and multiline input after the save. Route changes affect only newly ingested logs. Review parse failures after releases.

Exit criteria: each legacy format has an owner and documented contract. Representative records parse into the agreed fields. Unmatched records remain searchable. Parse failures are visible. New services default to structured logs rather than new Grok dependencies.

Level 4 - Add infrastructure metrics and validate the relevant view

Add the metrics and identity attributes that populate the inventory for the runtime. Logs alone do not create Kubernetes, Hosts, or service views. Validate the relevant log pivot before adding trace correlation.

Kubernetes logs ──► Kubernetes metrics + object events ──► Kubernetes inventory
VM / node logs ──► host metrics ─────────────────────────► Hosts
Lambda logs ─────► function metrics + traces ────────────► Services, Logs, and Traces

Kubernetes

For Kubernetes workloads, collect container logs, host and kubelet metrics at the node level, plus cluster metrics and entity events. Set a stable k8s.cluster.name and preserve workload identity on logs. Confirm that the same cluster and workload appear in the Kubernetes view and pivot to scoped logs, spans, and Analytics.

Kubernetes inventory needs Kubernetes metrics and object events. Kubernetes-looking log fields do not populate it. Fix missing node agents, cluster receivers, RBAC permissions, or cluster identity in the collection layer.

Hosts and VMs

For VMs and Kubernetes nodes, collect supported host metrics as well as logs. Hosts needs a usable host identifier on the appropriate CPU metric. Check that View logs still identifies the intended host when its name can appear elsewhere in a record.

Lambda and other serverless runtimes

For Lambda, collect function logs through the Telemetry API or the established cloud route. Send function metrics and traces through the OpenTelemetry Lambda path when tracing is in scope. Lambda execution environments do not appear in Hosts or Kubernetes inventory. Validate through the service, Logs, and Traces views. Use Cloud Resources only for a connected cloud account and an account-backed investigation.

Validate the relationship

For each runtime, test one known workload or function and confirm:

1. The relevant inventory or service view contains the expected resource identity.
2. Its scoped logs show the known test record without duplicate sources.
3. The metric context and log resource attributes agree on environment and infrastructure identity.
4. A user can move from the relevant view to its log evidence without widening to unrelated infrastructure.

Exit criteria: the chosen runtime has the required infrastructure metrics. The corresponding Kubernetes, Hosts, or service view is populated. The view’s logs pivot is both present and correctly scoped.

Level 5 - Correlation with traces

Add this level only when traces already reach Tsuga. The application must write the active span identifiers with the log. The Collector can preserve and map those fields but cannot reconstruct missing context.

Logs and traces must share the same service.name and deployment resource attributes. Prefer a language SDK, agent, or logger bridge because it can read the active context directly:

• Java: enable the agent or logger integration and include MDC or Log4j2 context fields in the structured pattern.
• Python: initialize logging instrumentation before handlers or framework logging are configured.
• Node.js: initialize OpenTelemetry before the logger and framework, then test worker threads separately.
• Go: pass context.Context to the logging call and read the current span context.
• .NET: export ILogger logs through OpenTelemetry while Activity.Current is active.

For an unsupported library, add trace_id and span_id in the formatter, processor, or wrapper. Map those fields into the OpenTelemetry log record. Do not infer context from a request ID or message text.

Validate with one known request that writes a unique log line while a span is active:

1. Confirm that the trace and log share the exact same service.name.
2. Confirm the log record contains non-empty trace and span IDs.
3. Confirm the trace ID opens a trace in Tsuga.
4. Confirm the relevant span shows its correlated logs.
5. Repeat the test for each in-scope background job, queue consumer, or asynchronous execution path.

For asynchronous work, propagate W3C context across the boundary and restore it before creating consumer spans and logs. Validate the consumer against its own active span.

Exit criteria: known request logs and traces link in both directions. Correlation works for agreed asynchronous paths. Logs retain correct service and environment identity. Treat missing correlation first as a context-propagation or logger-initialization issue.

Make it operational

Add operational coverage only after collection, parsing, and service identity are reliable. Otherwise alerts will not be trusted.

• Alert on new failure shapes, not on log volume. Use a new-error-pattern monitor on the application log route so an unfamiliar error introduced by a release becomes visible without someone writing a query for it first. Use a threshold monitor when the customer has an agreed error-count or error-rate limit for a known pattern, with a floor that stops a single line from paging.
• Alert on collection health. Persistent Collector export errors, queue pressure, or drops mean the log evidence is incomplete during an incident. Route that to the team that owns the Collector, not to the application team.
• Give investigators a starting view. A small dashboard per owning team with error counts by service, top patterns, parse failures, and log volume by source is enough. Build widgets from an Analytics query you have already validated.
• Agree retention and access. Set the retention policy per source with the customer, and record who may read logs that contain regulated or customer data. Retention is a data-handling decision, not a cost setting.
• Review parse failures and volume after each release. A release that changes a log format shows up as a parse-failure increase, not as an outage.

Scope monitors to one owning team and environment. Test the notification route before handover.

Troubleshooting path

Validate in source order: source emission, Collector acceptance and export, then a searchable Tsuga record with the expected identity. For wrong fields, check the structured logger output and route before changing a parser. For missing correlation, check active context, matching service.name, trace_id, and span_id before changing collection infrastructure.

Handover

The log path is handed over when:

• Every in-scope source has one collection owner, one route, one documented format, and one retention policy.
• The Collector configuration, ingestion key, and any Tsuga route have named owners, and the key has a rotation owner.
• Levels 1 to 3 have met their exit criteria for each source. Level 4 has met its exit criteria for the runtime that owns them. Level 5 has met its exit criteria for every service where traces exist.
• The data-handling decision is recorded: what is suppressed at the source, what is redacted in the Collector, who may read the result.
• The monitors and dashboard from Make it operational have owners and a tested notification route.

Intentional exclusions: logs prove what the application recorded. Request-level causality needs trace correlation. Kubernetes and Hosts views need metrics. Grok remains a compatibility layer for formats the team cannot change. New services should emit structured logs.

Completion criterion

A known log line appears in Tsuga exactly once, with the agreed service.name and environment, its parsed fields, and a trace_id that opens the trace that produced it.

Validate with the Tsuga CLI

# Confirm that one known log arrived with the expected service identity.
tsuga logs search \
  --query "context.service.name:<service> <unique-log-token>" \
  --from -15m \
  --to now \
  --max-results 10

# Confirm the trace that produced it is searchable with the same identity.
tsuga traces search \
  --query "context.service.name:<service>" \
  --from -15m \
  --to now \
  --max-results 10

The first command returns the test log once. Two identical records mean duplicate collection. Check its identity, parsed fields, and non-empty trace_id. If trace_id is empty, check the logger or context propagation. The second command confirms that the trace side of the pivot has the same service.name.