Instrumentation: The price you pay for observability

Production systems must report their own behavior. A service that cannot answer what it is doing, how long its operations take, and where its failures originate is not a finished is a liability awaiting an incident. Instrumentation is the discipline that closes this gap. It is the deliberate emission of telemetry from running software so that its behavior can be measured, queried, and reasoned about without attaching a debugger or redeploying code.

This article establishes the core concepts of instrumentation, makes the case for why it is a first-class engineering requirement and demonstrates a complete implementation on a Python/FastAPI stack using OpenTelemetry, Prometheus, and structured logging.

Part 1: Core Concepts

Instrumentation is the act of emitting telemetry from code. Such telemetry can come in the form of a histogram observation around a request, the structured log record or the trace span wrapping a database query. It occurs inside the application and is the responsibility of the engineers who write it.

Monitoring is the practice of watching known signals for known failure modes. Examples include sending alerts exceeds a defined threshold or when latency breaches an SLO. Monitoring answers questions formulated in advance.

Observability is a system property that can be defined as the ability to answer questions that were not formulated in advance. Observability can only be achieved when instrumentation is sufficiently contextual, correlated, and high-fidelity.

Monitoring consumes instrumentation; observability emerges from it. A system with poor instrumentation cannot be monitored effectively and cannot be made observable at all.

The three pillars of telemetry

Telemetry takes three canonical forms: Metrics, logs and traces.

Metrics — magnitude, frequency, and distribution

Metrics are numeric measurements aggregated over time: request throughput, latency distributions, queue depth, resource utilization. They are compact, cheap to retain, and fast to query, which makes them the correct substrate for dashboards and alerting.

The Prometheus data model defines the standard metric types:

• Counter : A counter is a monotonically increasing value. This can be the number of requests served, errors raised, payments processed. Counters are consumed as rates (rate(http_requests_total[5m])), never as raw values.
• Gauge: A guage is a value that may rise and fall. e.g active connections, queue depth, memory in use.
• Histogram: These are observations bucketed into ranges, enabling percentile computation. Request latency is a very classical example.
• Summary: A summary is a pre-computed quantile. Histograms have superseded summaries in practice because histograms aggregate correctly across instances; summaries do not.

Logs: Sytem events

Logs record descrete events in a system. They capture things such as the specific user transacting, the specific invoice issued or the exact error returned. They are the evidentiary record of the system.

There is clear distincion between structured and unstractured logging:

# Unstructured — a sentence, greppable at best
"Payment failed for user 4521 on invoice INV-2024-0091: timeout"

# Structured — a queryable event
{"event": "payment_failed", "user_id": 4521, "invoice_id": "INV-2024-0091",
 "reason": "timeout", "gateway": "daraja", "duration_ms": 30012,
 "trace_id": "7f9a2b...", "timestamp": "2026-08-06T09:14:22Z"}

The structured form supports filtering, aggregation, and — critically — correlation with traces via the trace ID. The unstructured form supports text search, contingent on the exact phrasing surviving every future refactor. Production systems should this emit structured logs.

Traces: The anatomy of a request

A distributed trace records the path of a single request through the system. Each unit of work, such as an HTTP handler, a database query or an outbound API call is known as a span, carrying its own start time, duration, and attributes. Spans nest to form a tree; every span in a request shares a trace ID.

Traces can help you answer questions such as:

• Of the 800ms a request consumed, where did the time go?

It is possible to see the costs in application code, in Postgres, or in a third-party payment gateway as a waterfall chart.

Cardinality is a hard constraint, not a guideline

Every unique combination of label values on a metric materializes a separate time series. The following is well-formed:

http_requests_total{method="POST", route="/api/invoices", status="201"}

The following will destroy a metrics backend at scale:

http_requests_total{method="POST", route="/api/invoices", status="201", user_id="4521"}

At 100,000 users, that single label multiplies the series count by five orders of magnitude. The rule is absolute: metric labels carry low-cardinality dimensions — route templates, status codes, plan tiers. High-cardinality detail belongs in logs and trace attributes, which are engineered for it. Violating this rule is the most common instrumentation failure, and it fails silently until the metrics infrastructure collapses under series growth.

Context propagation binds the pillars together

Telemetry is valuable in proportion to how well it correlates. The binding mechanism for this is known as context propagation. In practice, the trace ID travels with the request through function calls, across await boundaries, into background tasks, and across service boundaries via the W3C traceparent HTTP header.

When propagation is implemented correctly, incident investigation follows a fixed, fast path from an alert (metric), to the traces for the affected endpoint, to a representative slow trace, to every log line emitted during that exact request.

Part 2: Why Instrumentation Is Non-Negotiable

Production cannot be debugged any other way

Instrumentation is the only mechanism for investigating behavior that manifests in production systems under real load, real data, real users and real concurrency, which is where the consequential failures live. Race conditions, connection pool exhaustion, lock contention, and thundering herds do not reproduce on a laptop. Debuggers do not attach to pods serving live traffic. New 'Print statements' require a redeploy, by which point the incident has passed and the evidence is gone.Systems that are not instrumented are, in the only environment that matters, systems that cannot be debugged.

Data terminates arguments

With traces, arguments such as "The database is the bottleneck." vs "No — the external API is." can quicky be resolved by a five-second query. Without telemetry, this dispute is resolved by seniority.

Instrumentation restructures engineering culture around evidence. As such, capacity planning, performance work, and incident review all move from opinion to measurement.

Deployment safety is built on telemetry

Error rates and latency percentiles before and after a rollout constitute the canary signal. Every mature deployment practice, such as canary releases, blue/green deployments and automated rollback presupposes instrumentation. Without metrics, there is no rollback trigger and no definition of "healthy."

Accountability requires records

Any system handling payments or personal data will eventually be asked to account for its behavior by auditors, regulators or by customers after an incident. "We do not know what happened" is not an acceptable answer. Audit trails, breach forensics, and SLA reporting all presuppose that the system records what it does.

Distributions expose what averages conceal

An uninstrumented system is evaluated by anecdote: "it feels slow." An instrumented system is evaluated by distribution. The distinction is commercially material. A checkout endpoint with an acceptable mean and a degraded p99 is failing its highest-value users, which might be large orders and complex requests, and these populate the tail. A tail that is not measured is a tail that will not be fixed.

The cost curve rewards early adoption

Retrofitting telemetry into a mature codebase is no easy feat. You'll be dealing with hundreds of handlers, inconsistent logging, no correlation identifiers. Instrumenting a young codebase is a couple hundred lines of middleware code and a set of habits. As with testing, the cost is front-loaded in the adopter's favor, and the penalty for deferral compounds.

Part 3: Implementation

The reference stack is FastAPI, OpenTelemetry for tracing, Prometheus for metrics, and structlog for structured logging. The concepts should transfer directly to Go, Node.js, or the JVM; only the library names change.

Step 0: Decide what to measure — RED and USE

Instrumentation without a framework produces noise. Two established methods cover the essential surface:

RED — applied to every request-driven service and endpoint:

• Rate — requests per second
• Errors — failed requests per second
• Duration — latency distribution, as a histogram

USE — applied to every resource:

• Utilization — how busy the resource is
• Saturation — how much work is queued against it
• Errors — failure events

RED on the API surface and USE on the database connection pool will surface the majority of production issues a typical service encounters. Implement these first and refine later.

Step 1: Structured logging

Configure structlog to emit JSON in production:

# logging_setup.py
import logging
import structlog

def configure_logging(json_output: bool = True):
    processors = [
        structlog.contextvars.merge_contextvars,   # request-scoped context
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
    ]
    if json_output:
        processors.append(structlog.processors.JSONRenderer())
    else:
        processors.append(structlog.dev.ConsoleRenderer())  # local development

    structlog.configure(
        processors=processors,
        wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
        cache_logger_on_first_use=True,
    )

Log events, not prose:

import structlog
log = structlog.get_logger()

# Incorrect
log.info(f"Processing payment of {amount} for invoice {invoice_id}")

# Correct — every field is queryable
log.info("payment_processing_started",
         invoice_id=invoice_id,
         amount=amount,
         currency="KES",
         gateway="daraja")

Use this convention: the first argument is a stable, snake_case event name that survives refactors and supports filtering indefinitely; all remaining detail is key-value data.

Step 2: Request context middleware

Assign every request an identifier and bind it into the logging context, so that every log record within the request carries it without per-call effort:

# middleware.py
import time
import uuid

import structlog
from starlette.middleware.base import BaseHTTPMiddleware

log = structlog.get_logger()

class RequestContextMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        request_id = request.headers.get("x-request-id") or uuid.uuid4().hex

        structlog.contextvars.clear_contextvars()
        structlog.contextvars.bind_contextvars(
            request_id=request_id,
            method=request.method,
            path=request.url.path,
        )

        start = time.perf_counter()
        response = await call_next(request)
        elapsed_ms = (time.perf_counter() - start) * 1000

        log.info("request_completed",
                 status_code=response.status_code,
                 duration_ms=round(elapsed_ms, 2))

        response.headers["x-request-id"] = request_id
        return response

bind_contextvars is built on Python's contextvars, so the bound fields survive await boundaries and remain isolated between concurrent requests. From this point forward, every log record emitted during a request includes request_id, method, and path by construction.

Step 3: Metrics with Prometheus

pip install prometheus-client

Define the RED metrics once, as module-level singletons:

# metrics.py
from prometheus_client import Counter, Histogram, Gauge

HTTP_REQUESTS = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "route", "status"],
)

HTTP_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency",
    ["method", "route"],
    # Buckets tuned for a web API: 5ms to 10s
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)

DB_POOL_IN_USE = Gauge(
    "db_pool_connections_in_use",
    "Database connections currently checked out",
)

PAYMENTS_PROCESSED = Counter(
    "payments_processed_total",
    "Payments processed, by gateway and outcome",
    ["gateway", "outcome"],  # outcome: success | failed | timeout
)

Record them in the middleware. Note the cardinality discipline: the label is the route template (/invoices/{invoice_id}), never the concrete path (/invoices/8812) — the concrete path would materialize one time series per invoice:

# in RequestContextMiddleware.dispatch, after call_next:
route = request.scope.get("route")
route_path = route.path if route else "unmatched"

HTTP_REQUESTS.labels(
    method=request.method,
    route=route_path,
    status=str(response.status_code),
).inc()
HTTP_DURATION.labels(method=request.method, route=route_path).observe(
    elapsed_ms / 1000
)

Expose the scrape endpoint:

# main.py
from fastapi import FastAPI
from prometheus_client import make_asgi_app

app = FastAPI()
app.add_middleware(RequestContextMiddleware)
app.mount("/metrics", make_asgi_app())

A Prometheus server scraping /metrics at a 15-second interval now provides rate, errors, and duration for every endpoint. The queries that belong on the primary dashboard:

# Requests per second, by route
sum by (route) (rate(http_requests_total[5m]))

# Error ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# p99 latency per route
histogram_quantile(0.99,
  sum by (route, le) (rate(http_request_duration_seconds_bucket[5m])))

Business metrics warrant the same rigor as technical metrics. A spike in payments_processed_total{gateway="daraja", outcome="timeout"} is a superior early-warning signal to any CPU graph: it measures user impact directly rather than machine load.

Step 4: Distributed tracing with OpenTelemetry

OpenTelemetry is the vendor-neutral standard: instrument once, export to Jaeger, Tempo, Honeycomb, or any commercial backend. Auto-instrumentation covers the majority of the surface:

pip install opentelemetry-api opentelemetry-sdk \
    opentelemetry-exporter-otlp \
    opentelemetry-instrumentation-fastapi \
    opentelemetry-instrumentation-sqlalchemy \
    opentelemetry-instrumentation-httpx \
    opentelemetry-instrumentation-redis

# tracing.py
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor

def configure_tracing(app, engine, service_name: str = "invoice-api"):
    provider = TracerProvider(
        resource=Resource.create({"service.name": service_name})
    )
    provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
    )
    trace.set_tracer_provider(provider)

    FastAPIInstrumentor.instrument_app(app)          # span per request
    SQLAlchemyInstrumentor().instrument(engine=engine.sync_engine)  # span per query
    HTTPXClientInstrumentor().instrument()           # span per outbound call
    RedisInstrumentor().instrument()                 # span per Redis command

This configuration alone produces a complete waterfall per request: the handler span, each SQL statement, each Redis command, each outbound HTTP call with status and duration. The traceparent header propagates automatically on outbound httpx requests; an OTel-instrumented downstream service continues the same trace across the network boundary.

Manual spans belong around business logic that auto-instrumentation cannot see:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def process_payment(invoice_id: str, amount: int):
    with tracer.start_as_current_span("process_payment") as span:
        span.set_attribute("invoice.id", invoice_id)     # high cardinality is
        span.set_attribute("payment.amount", amount)     # correct on spans
        span.set_attribute("payment.gateway", "daraja")

        result = await initiate_stk_push(invoice_id, amount)

        span.set_attribute("payment.outcome", result.status)
        if result.status == "failed":
            span.record_exception(result.error)
        return result

Trace attributes carry the invoice ID and amount without penalty — the high-cardinality detail deliberately excluded from metric labels belongs precisely here.

Step 5: Correlate logs with traces

As the final requirement, stamp the active trace ID onto every log record, enabling bidirectional navigation between traces and logs. A single structlog processor accomplishes this:

# add to the processors list in configure_logging()
from opentelemetry import trace

def add_trace_context(logger, method_name, event_dict):
    span = trace.get_current_span()
    ctx = span.get_span_context()
    if ctx.is_valid:
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict

The investigative loop is now closed:

1. The alert fires: p99 latency on /api/payments has breached 2 seconds (metric).
2. The trace view, filtered to that route and sorted by duration, shows the slow traces sharing a 1.9-second span on the Daraja STK Push call (trace).
3. The trace ID, queried against the logs, yields the exact gateway request and response, with invoice and tenant attached (log).

The investigation takes minutes. No SSH sessions, no speculation.

Step 6: Alert on symptoms, not causes

Alerts must target what users experience. These are things like error ratio, SLO burn rate or payment success rate and not internal states such as CPU utilization or pod restarts. Cause-based alerts page engineers for conditions users never noticed; symptom-based alerts page precisely when impact exists, and the instrumentation identifies the cause once investigation begins.

The minimum viable alert set for a small service:

• Error ratio above 1% over 5 minutes
• p99 latency above SLO for 10 minutes
• Payment success rate below baseline
• The /metrics endpoint itself unreachable — the alert that verifies telemetry is alive

Failure Modes to Engineer Against

High-cardinality metric labels. Stated above; restated because it is the most common failure. User IDs, invoice IDs, and raw URLs do not belong on metrics. Monitor the time-series count itself.

Sensitive data in logs. Structured logging makes it trivially easy to serialize entire objects. Data like phone numbers, national IDs, tokens, and card data must be redacted at the processor level. Under any data-protection regime, logs are personal data.

Telemetry without consumers. A metric that feeds no dashboard and no alert is an infrastructure bill with no return. Every signal should exist to answer a question. Audit quarterly and delete what nothing consumes.

Deferring sampling strategy. Tracing every request is viable at 10 rps and ruinous at 10,000. Head sampling is simple; tail sampling is superior and supported by the OTel Collector. The strategy must precede the traffic.

Averages on dashboards. Any panel displaying mean latency should be replaced with p50/p95/p99.

Instrumentation as an afterthought. A complete instrumentation setup can take less thanof 150 lines. Written in the first week of a project, it makes every subsequent feature observable by default.

Conclusion

Instrumentation reduces to three enforced practices:

1. Emit structured events, not strings. Every log record is a queryable object with a stable event name.
2. Measure RED signals with histograms. Rate, errors, and duration on every endpoint; percentiles over averages; low-cardinality labels only.
3. Trace the request path and propagate context. Every log, metric, and span from a single request must join on its trace ID.