# OpenTelemetry, From Zero > The vendor-neutral standard for telemetry: traces, metrics, and logs from one instrumentation, exported anywhere via the OTel collector. --- # OpenTelemetry, From Zero You picked a monitoring vendor, sprinkled their SDK through your code, and shipped. Eighteen months later the bill tripled, you want to switch, and you realize their agent is welded into every service. That trap is exactly what OpenTelemetry exists to disarm. You instrument your code once against an open standard, and you point the data wherever you want - today's vendor, tomorrow's vendor, or your own stack. This guide gives you the mental model first: what the three signals are, how a trace stitches itself across services, and what the collector actually does. Then you wire it up and live with it. ## How to read this Read phase 1 even if you're in a hurry - the whole thing makes sense once you see *why* "instrument once, export anywhere" is the entire point. Phase 2 is the hands-on core: SDK, auto-instrumentation, the collector pipeline. Phase 3 is the part nobody warns you about: sampling, cost, and the failures that look like OTel's fault but aren't. If you're fuzzy on what traces, metrics, and logs even are as concepts, read [the observability primer](/guides/observability-logs-metrics-traces) first, then come back here for the standard that ties them together. ## The phases 1. [What OpenTelemetry actually is](01-what-otel-actually-is.md) - the standard, the three signals, and why it won. 2. [Instrumenting and exporting](02-instrumenting-and-exporting.md) - SDK, auto vs manual, the collector pipeline. 3. [Sampling, cost, and reality](03-sampling-cost-and-reality.md) - what breaks, what it costs, and how to keep it sane. --- # What OpenTelemetry actually is Here's the situation OTel was born into. Every monitoring vendor used to ship its own SDK. You'd import their library, call their tracer, format logs their way, and name your metrics by their convention. Your telemetry - the data that describes how your system behaves - was written in *their* dialect. Switching vendors meant re-instrumenting every service by hand. The data was hostage, and everyone knew it. OpenTelemetry (almost always written "OTel") flips that. It's an open standard for how telemetry is *produced* and *shaped*, governed by the CNCF, the same foundation behind Kubernetes. You instrument your code against OTel's API. Where the data finally lands - a SaaS backend, an open-source stack, a file - is a configuration choice you make later, and can change without touching code. That sentence is the whole guide: **instrument once, export anywhere.** ## The three signals OTel organizes telemetry into three "signals." You don't have to adopt all three at once, but they're designed to fit together. - **Traces** - the path of a single request as it moves through your system. A trace is a tree of **spans**, where each span is one unit of work (an HTTP handler, a DB query, a call to another service). Traces answer "*why was this one request slow?*" - **Metrics** - numbers aggregated over time: request rate, error count, queue depth, p99 latency. Metrics answer "*is the system healthy right now, in aggregate?*" - **Logs** - timestamped text records of discrete events. Logs answer "*what exactly happened at 14:03?*" If those three categories are new to you, the [observability primer](/guides/observability-logs-metrics-traces) walks through each in depth. Here we care about what makes OTel special: it gives all three a *common* data model and a *common* way to ship them, so they can reference each other. An exemplar on a latency metric can point straight at the trace of a slow request. ## A span, concretely A trace is the star of the show, so let's make a span real. Conceptually, one span carries a name, a start and end time, a status, and a bag of key/value **attributes**. Here's what a single span looks like when you print it: ```text Span: "GET /checkout" trace_id: 4bf92f3577b34da6a3ce929d0e0e4736 span_id: 00f067aa0ba902b7 parent: (none - this is the root) start: 14:03:12.114 end: 14:03:12.461 (347 ms) status: OK attributes: http.request.method = "GET" http.route = "/checkout" http.response.status_code = 200 user.id = "u_8812" ``` *What just happened:* one request became one span with a unique `span_id`, grouped under a `trace_id` that ties it to every other span in the same request. The attributes are the searchable context - later you can ask "show me checkout spans for user u_8812 that took over 300 ms." ## How a trace crosses service boundaries A single span is fine, but the magic is a trace that spans *services*. When service A calls service B, A has to tell B "you're part of trace `4bf9…`, and your parent span is `00f0…`." This handoff is called **context propagation**, and it rides along in request headers - for HTTP, a standard header called `traceparent` (the W3C Trace Context format OTel adopts). ```text Service A (web) Service B (payments) span: GET /checkout span: POST /charge trace_id: 4bf9... trace_id: 4bf9... (same!) span_id: 00f0... parent: 00f0... (A's span) | ^ | HTTP request with header | +--- traceparent: 00-4bf9...-00f0...-01 ---+ ``` *What just happened:* B read the `traceparent` header, saw it belonged to trace `4bf9…`, and made its own span a *child* of A's span. Now both spans share one `trace_id`, so your backend can draw the full waterfall - A waited on B - across two separate processes. No shared database, no manual correlation IDs. The propagation is the thing that turns isolated spans into one distributed trace. > Lose context propagation and your traces shatter into disconnected single-service fragments. Most "my traces aren't connected" bugs come down to a hop where the headers weren't forwarded - a queue, a background job, a proxy that strips headers. ## Why OTel won Standards usually lose to whoever has the biggest install base. OTel won anyway, for a few grounded reasons: - **It merged the two main rivals.** OpenTracing and OpenCensus were competing open projects splitting the community. OTel is their merger, so the obvious alternatives folded into it instead of fighting it. - **It's vendor-neutral by design, and vendors back it anyway.** The backends still compete on storage, querying, and UI - the parts that are genuinely hard. Owning the SDK was never their moat, so supporting a shared standard cost them little and won them goodwill. - **The collector decouples everything.** Because there's a separate component that receives, transforms, and forwards telemetry, your apps never need to know who the backend is. (That's phase 2.) - **It rides existing standards.** W3C Trace Context for propagation, a stable wire protocol (OTLP). It didn't reinvent the web's plumbing. The practical payoff: the OTel API in your code is stable and neutral, so the "rip out the vendor SDK" project that used to eat a sprint becomes a config change. ```quiz [ { "q": "What does 'instrument once, export anywhere' mean in OpenTelemetry?", "choices": [ "You can only export to one backend at a time", "Your code emits telemetry against a neutral standard, and the destination backend is a config choice you can change without editing code", "Instrumentation is automatic and never needs code", "You write separate instrumentation for each vendor" ], "answer": 1, "explain": "OTel separates how telemetry is produced from where it lands, so switching backends is configuration, not re-instrumentation." }, { "q": "What ties multiple spans across different services into one distributed trace?", "choices": [ "A shared database table of request IDs", "All services writing to the same log file", "Context propagation - a shared trace_id and parent span_id passed in headers like traceparent", "The collector guessing which spans belong together by timestamp" ], "answer": 2, "explain": "Context propagation carries the trace_id and parent span_id (e.g. via the W3C traceparent header) so a downstream span becomes a child of the upstream one." }, { "q": "Which is NOT one of OpenTelemetry's three signals?", "choices": [ "Traces", "Metrics", "Logs", "Dashboards" ], "answer": 3, "explain": "The three signals are traces, metrics, and logs. Dashboards are something a backend builds on top of that data, not a signal OTel produces." } ] ``` --- # Instrumenting and exporting You understand the model. Now the question is the one you actually showed up with: *how do I get spans out of my service and onto a screen?* There are exactly two jobs. First, your application has to **produce** telemetry - that's instrumentation. Second, that telemetry has to **travel** somewhere - that's exporting, and the piece in the middle is the collector. We'll do them in that order. ## Auto vs manual instrumentation You have two ways to produce telemetry, and you'll use both. **Auto-instrumentation** wires up the libraries you already use - your web framework, HTTP client, database driver - without you editing application code. Each language ships its own auto-instrumentation, and the activation differs by ecosystem. In some it's a packaged agent you attach at startup; in others you install instrumentation packages and call a setup function. The shape varies, but the idea is identical: known libraries get spans for free. ```bash # Python: install the instrumentation packages, let the tooling # detect your libraries, then run your app under it. pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-bootstrap -a install # detects installed libs, adds matching instrumentation # Point it at a collector and run your app through the wrapper: export OTEL_SERVICE_NAME=checkout-api export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 opentelemetry-instrument python app.py ``` *What just happened:* with zero changes to `app.py`, your Flask/Django/FastAPI routes, your `requests` calls, and your DB queries now emit spans, already connected by context propagation. `OTEL_SERVICE_NAME` is the label that tells your traces apart from other services; the endpoint is where spans go. > Auto-instrumentation gets you 80% of the value for roughly zero effort. Start here every time. Reach for manual spans only where auto-instrumentation can't see - your own business logic. **Manual instrumentation** is you creating spans by hand for the work that matters to *your* domain - the loop auto-instrumentation has no way to know is important. ```python from opentelemetry import trace tracer = trace.get_tracer("checkout") def apply_discounts(cart): with tracer.start_as_current_span("apply_discounts") as span: span.set_attribute("cart.item_count", len(cart.items)) total = run_discount_rules(cart) # your real logic span.set_attribute("cart.discount_total", total) return total ``` *What just happened:* you created a span named `apply_discounts` that automatically becomes a child of whatever span is currently active (the HTTP handler auto-instrumentation already started). Because it's the "current span," any DB call inside `run_discount_rules` nests under it too. The attributes turn this into something you can query: "discount runs where `item_count > 50`." ## The SDK: what's actually running The OTel **API** is the surface you call (`get_tracer`, `start_as_current_span`). The OTel **SDK** is the implementation that does the real work: it batches spans, applies sampling, attaches resource info (service name, host, version), and hands finished spans to an **exporter**. The standard exporter speaks **OTLP** - the OpenTelemetry Protocol - which is the native wire format every OTel-aware backend and the collector understand. That API/SDK split matters: a library author can call the API and emit spans, and if no SDK is configured, those calls are harmless no-ops. The application decides whether and how telemetry actually flows. ## The collector: receive, process, export You *can* export straight from your app to a backend. For anything beyond a toy, don't - put the **collector** in between. The collector is a standalone binary (or sidecar, or cluster service) that does three things, in a pipeline, for each signal: ```text ┌──────────────── OpenTelemetry Collector ────────────────┐ your apps │ RECEIVERS → PROCESSORS → EXPORTERS │ backends (OTLP) ───┼──► otlp → batch → otlp/ ──────────┼──► Vendor A │ → filter → prometheus ──────────┼──► Grafana stack │ → attributes → logging (debug) ────────┼──► stdout └──────────────────────────────────────────────────────────┘ ``` *What just happened:* the collector **receives** telemetry (commonly over OTLP on ports 4317 for gRPC and 4318 for HTTP), **processes** it (batch it for efficiency, drop noisy spans, scrub a PII attribute, add metadata), and **exports** it to one or more destinations at once. Your apps only ever know the collector's address. Here's a minimal collector config - three blocks defined, then a `pipelines` section that wires them together: ```yaml receivers: otlp: protocols: grpc: # listens on :4317 http: # listens on :4318 processors: batch: # group spans before export - fewer, bigger sends attributes: actions: - key: user.email # never let PII reach the backend action: delete exporters: otlp/vendor: endpoint: ingest.example-backend.com:443 debug: # print to the collector's own logs while testing verbosity: detailed service: pipelines: traces: receivers: [otlp] processors: [attributes, batch] exporters: [otlp/vendor, debug] ``` *What just happened:* defining a receiver, processor, or exporter does nothing on its own - it has to be listed in a pipeline under `service`. This `traces` pipeline takes OTLP in, deletes the `user.email` attribute, batches, and fans out to both the vendor and the debug log. Want to add a second backend or send metrics somewhere else? You edit this YAML and restart the collector - your application code never changes. That's the decoupling from phase 1, made concrete. ## Why the collector earns its keep It looks like an extra moving part, and it is - but it pays for itself fast: - **One place to switch backends.** Re-point the exporter, restart the collector, done. No redeploy of N services. - **One place to scrub and shape.** PII deletion, attribute renaming, dropping health-check spans - centralized, not copy-pasted into every service. - **A buffer.** The collector batches and retries, so a backend hiccup doesn't back up into your application. - **Protocol translation.** Receive OTLP, export Prometheus-format metrics for a [Prometheus and Grafana](/guides/prometheus-and-grafana) setup, or speak a vendor's dialect - the collector adapts so your code doesn't. For builders: a common production shape is a lightweight collector running as an **agent** next to each app (or as a sidecar), forwarding to a horizontally-scaled **gateway** collector pool that does the heavy processing and talks to backends. Start with one collector; split into agent + gateway only when volume demands it. ```quiz [ { "q": "When should you reach for manual instrumentation instead of auto-instrumentation?", "choices": [ "Always - auto-instrumentation is unreliable", "For your own business logic that auto-instrumentation can't see, after auto-instrumentation covers the standard libraries", "Only when you have no framework", "Never - manual spans are deprecated" ], "answer": 1, "explain": "Start with auto-instrumentation for frameworks/clients/DBs, then add manual spans for domain logic the auto layer has no way to know matters." }, { "q": "What are the three stages of a collector pipeline, in order?", "choices": [ "Export, process, receive", "Receive, process, export", "Ingest, store, query", "Sample, batch, drop" ], "answer": 1, "explain": "A collector pipeline receives telemetry, processes it (batch, filter, scrub), then exports it to one or more backends." }, { "q": "In a collector config, what makes a defined exporter actually do anything?", "choices": [ "Defining it under the exporters block is enough", "It must be listed in a pipeline under the service section", "It activates automatically on restart", "You must set its verbosity to detailed" ], "answer": 1, "explain": "Receivers, processors, and exporters are inert until wired into a pipeline under service.pipelines - that's what connects them." } ] ``` --- # Sampling, cost, and reality The first month with OTel is a honeymoon. Traces light up, you find a slow query you'd been blind to for a year, everyone's thrilled. Then the bill arrives, or a trace shows up half-empty, and you learn the parts the getting-started guides skip. This phase is those parts: keeping the volume sane, the failures that masquerade as OTel bugs, and the habits that keep traces trustworthy. ## Sampling: you can't keep every trace A busy service produces an astonishing number of spans, and most traces are boring - a fast, successful request that looks like a million others. Storing all of them is expensive and tells you nothing extra. **Sampling** is deciding which traces to keep. There are two strategies, and the difference is the most consequential choice you'll make. **Head sampling** decides at the *start* of a trace, before you know how it turns out. It's cheap and simple - flip a coin at the root span, keep 10%. ```yaml # In the SDK / via env: keep ~10% of traces, chosen at the root. export OTEL_TRACES_SAMPLER=parentbased_traceidratio export OTEL_TRACES_SAMPLER_ARG=0.1 ``` *What just happened:* the sampler keeps roughly one trace in ten. `parentbased_*` is the important half of the name - it means a child service respects the parent's decision, so an entire distributed trace is kept-or-dropped together. Without that, service A keeps the trace and service B drops it, and you get the shattered, half-missing traces that drive people up the wall. The catch: head sampling can't know the request will error or be slow, because it decides *first*. You'll throw away exactly the traces you most wanted to see. **Tail sampling** decides at the *end*, once the whole trace is complete and you can see latency and status. It runs in the collector, which buffers all spans of a trace, then applies rules: keep everything that errored, everything slow, plus a small percentage of the normal ones. ```yaml processors: tail_sampling: decision_wait: 10s # hold spans this long, waiting for the trace to finish policies: - name: keep-errors type: status_code status_code: { status_codes: [ERROR] } - name: keep-slow type: latency latency: { threshold_ms: 500 } - name: keep-some-normal type: probabilistic probabilistic: { sampling_percentage: 5 } ``` *What just happened:* the collector waits up to 10 seconds for a trace's spans, then keeps it if *any* policy matches - every error, every request over 500 ms, and 5% of the rest. You keep the interesting traces and a representative baseline, and drop the boring bulk. The price is memory and complexity: the collector has to hold spans in flight, which constrains where and how you can run it (all spans of one trace must reach the same collector instance). > Rule of thumb: start with head sampling because it's trivial. Move to tail sampling when you realize you're missing the errors - which you will, because head sampling drops them blind. ## The cost trap nobody mentions OTel itself is free; the data it produces is not. Most observability bills scale with **volume**, and the silent budget-killers are usually: - **High-cardinality attributes.** Putting something unbounded - `user.id`, a request UUID, a full URL with query string - as a *metric* dimension explodes the number of time series and can dwarf everything else. Attributes on *spans* are fine and useful; the danger is unbounded values on *metrics*. - **Over-instrumented spans.** Auto-instrumentation plus eager manual spans can produce dozens of spans per request. Most are noise. - **Logs you forgot you forwarded.** Piping debug-level logs through OTel at full volume is a fast way to a surprising invoice. The collector is your cost-control panel. The `filter`, `attributes`, and sampling processors let you drop health checks, strip high-cardinality dimensions, and thin volume *before* it hits the metered backend - and you tune it without redeploying a single service. ## Failures that look like OTel's fault (but aren't) When traces go wrong, the cause is almost always one of these, and almost never a bug in OTel: - **Broken traces across a hop.** A queue, a cron job, an outbound HTTP call, or a proxy didn't carry the `traceparent` context. The trace splits into disconnected fragments. Fix: ensure context is propagated (or manually re-attached) across every async boundary - message brokers especially, since they don't forward headers for you. - **Nothing shows up at all.** Ninety percent of the time it's the endpoint or the protocol port: gRPC OTLP is `4317`, HTTP OTLP is `4318`, and mixing them up means your spans sail into a closed door. Add a `debug` exporter to the collector and watch whether spans even arrive. - **Clock skew makes the waterfall look wrong.** Spans from a machine with a drifting clock render with impossible overlaps or negative gaps. The traces are real; the clocks lied. Fix it at the host with NTP, not in OTel. - **Missing spans after enabling sampling.** Working as designed - you asked it to drop traces. Confirm your sampler ratio before assuming data loss. ## Semantic conventions: the boring habit that pays off OTel publishes **semantic conventions** - standard names for common attributes, like `http.request.method`, `db.system`, `service.name`. They feel pedantic until the moment a backend's dashboard, alert, or service map *just works* because your attributes matched the names it expected. Custom-named attributes (`my_http_method`) leave you wiring everything by hand. Follow the conventions for anything standard; invent names only for genuinely domain-specific attributes (`cart.discount_total` is yours to name). In the wild: teams that succeed with OTel treat it as a product with an owner, not a one-time install. Someone owns the collector config, the sampling policy, and the conventions - because telemetry that nobody curates degrades into expensive noise. Start small (auto-instrumentation, head sampling, one collector), then evolve sampling and processing as your volume and your questions grow. ```quiz [ { "q": "Why does head sampling tend to lose the traces you most want?", "choices": [ "It keeps too many traces and hides errors in the noise", "It decides at the start of a trace, before it knows whether the request errored or was slow", "It only works inside the collector", "It always keeps errors but drops normal traces" ], "answer": 1, "explain": "Head sampling decides at the root span, before the outcome is known, so it can drop a trace that later turned out to be a slow error." }, { "q": "Which is the classic silent driver of a high observability bill?", "choices": [ "Following OTel's semantic conventions", "Running a collector", "Putting high-cardinality values like user IDs as metric dimensions", "Using head sampling" ], "answer": 2, "explain": "Unbounded values as metric dimensions explode the number of time series. (On spans, attributes are fine; the danger is on metrics.)" }, { "q": "A trace splits into disconnected single-service fragments after passing through a message queue. The most likely cause is:", "choices": [ "OpenTelemetry has a bug in its trace model", "Context (traceparent) wasn't propagated across the async boundary", "The backend rejected the spans", "Sampling dropped the middle spans" ], "answer": 1, "explain": "Queues and async boundaries don't forward trace context automatically; without propagating it, downstream spans start a new, disconnected trace." } ] ```