OpenTelemetry Instrumentation

Trace agent execution, LLM calls, tool usage, and handoffs using OpenTelemetry. Compatible with Langfuse and any OTel-compatible backend.

Overview

The Agents::Instrumentation module produces OTel spans that give you full visibility into agent execution:

• LLM generation spans with model name, token counts, and input/output
• Tool execution spans with arguments and results
• Agent container spans grouping related LLM and tool calls
• Handoff events recording agent-to-agent transfers

Generation inputs include MIME types for RubyLLM message attachments. Their URLs, local paths, and image data are omitted. Structured image inputs also omit the image URL.

Spans follow the GenAI semantic conventions and include Langfuse-specific attributes for rich rendering in the Langfuse dashboard.

RubyLLM 2.0 events and Langfuse

RubyLLM 2.0 emits chat.ruby_llm, tool_call.ruby_llm, and usage.ruby_llm events. In Rails, these use ActiveSupport::Notifications. Outside Rails, configure a RubyLLM instrumenter to receive them. The events are useful for logging and per-attempt usage, but RubyLLM does not turn them into this runner’s Langfuse trace. Keep Agents::Instrumentation when you need the following mapping:

Runner span or event	Data in Langfuse	RubyLLM event alone
Root span	Overall input and output, session, user, tags, and Chatwoot metadata	No runner-level input, output, or Chatwoot context
Agent span	Agent name and the generations and tools it owns	No agent handoff boundary
Generation span	Request messages, response, model, temperature, and response tokens	chat.ruby_llm has the model and response, but needs an OTLP and Langfuse attribute adapter
Tool span	Tool arguments, result, and error status	tool_call.ruby_llm has the call, but needs the same adapter
Handoff event	Source agent, target agent, and reason	No RubyLLM handoff event for this runner

RubyLLM’s usage.ruby_llm event reports each provider attempt, including failed and cancelled attempts. The generation spans here use tokens from completed messages. Use a separate usage.ruby_llm subscriber for attempt-level accounting; do not add its tokens to the same generation spans or costs may be counted twice. See the RubyLLM instrumentation guide for event payloads.

For an app outside Rails, add activesupport and configure RubyLLM before creating a chat:

require "active_support"
require "active_support/notifications"

RubyLLM.configure do |config|
  config.instrumenter = ActiveSupport::Notifications
end

ActiveSupport::Notifications.subscribe("usage.ruby_llm") do |event|
  puts event.payload.inspect # Replace with your per-attempt usage collector.
end

This enables RubyLLM’s native events. The runner spans still require Agents::Instrumentation.install and an OTel exporter, as shown below.

Setup

1. Install dependencies

Add to your Gemfile:

gem "opentelemetry-sdk"
gem "opentelemetry-exporter-otlp"

Then run bundle install.

2. Configure the OTel SDK

require "opentelemetry-sdk"
require "opentelemetry-exporter-otlp"

OpenTelemetry::SDK.configure do |c|
  c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
      OpenTelemetry::Exporter::OTLP::Exporter.new(
        endpoint: "https://your-otel-endpoint/v1/traces",
        headers: { "Authorization" => "Bearer YOUR_TOKEN" }
      )
    )
  )
end

3. Install on a runner

require "agents/instrumentation"

tracer = OpenTelemetry.tracer_provider.tracer("my-app")
runner = Agents::Runner.with_agents(triage, billing, support)

Agents::Instrumentation.install(runner, tracer: tracer)

That’s it. Every runner.run(...) call now produces OTel spans.

Span Hierarchy

root (agents.run)
├── agent.Calculator           # container span per agent (no model attr)
│   ├── agents.run.generation  # GENERATION: model, tokens, I/O
│   ├── agents.run.tool.add    # TOOL: arguments + result
│   └── agents.run.generation  # second LLM call after tool result
├── agent.Support              # after handoff
│   └── agents.run.generation
└── agents.run.handoff         # point event on root span

Only GENERATION spans carry gen_ai.request.model. This prevents Langfuse from double-counting costs when it sums token usage across spans with a model attribute.

Configuration Options

trace_name

Custom name for the root span (default: "agents.run"):

Agents::Instrumentation.install(runner,
  tracer: tracer,
  trace_name: "customer_support.run"
)

Child spans derive their names: customer_support.run.generation, customer_support.run.tool.add_numbers, etc.

span_attributes

Static attributes applied to the root span:

Agents::Instrumentation.install(runner,
  tracer: tracer,
  span_attributes: {
    "langfuse.trace.tags" => '["production","v2"]',
    "langfuse.session.id" => session_id
  }
)

attribute_provider

A lambda that receives the context wrapper and returns dynamic attributes:

Agents::Instrumentation.install(runner,
  tracer: tracer,
  attribute_provider: ->(ctx) {
    {
      "langfuse.user.id" => ctx.context[:user_id].to_s,
      "langfuse.session.id" => ctx.context[:session_id].to_s
    }
  }
)

Langfuse Integration

Endpoint and Authentication

Langfuse accepts OTel traces at {LANGFUSE_HOST}/api/public/otel/v1/traces. Authentication uses HTTP Basic with your public and secret keys:

require "base64"

langfuse_host = ENV["LANGFUSE_HOST"] # e.g. "https://cloud.langfuse.com"
langfuse_pk   = ENV["LANGFUSE_PUBLIC_KEY"]
langfuse_sk   = ENV["LANGFUSE_SECRET_KEY"]

auth_token = Base64.strict_encode64("#{langfuse_pk}:#{langfuse_sk}")

OpenTelemetry::SDK.configure do |c|
  c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
      OpenTelemetry::Exporter::OTLP::Exporter.new(
        endpoint: "#{langfuse_host}/api/public/otel/v1/traces",
        headers: { "Authorization" => "Basic #{auth_token}", "x-langfuse-ingestion-version" => "4" }
      )
    )
  )
end

Attribute Mapping

The instrumentation sets Langfuse-specific attributes that map to the Langfuse UI:

Attribute	Set On	Langfuse Display
langfuse.trace.input	Root span	Legacy trace input for existing evaluators
langfuse.trace.output	Root span	Legacy trace output for existing evaluators
langfuse.observation.input	All spans	Observation input (sidebar click)
langfuse.observation.output	All spans	Observation output (sidebar click)
langfuse.observation.type	Tool spans	"tool" type indicator
langfuse.user.id	Root span (via attribute_provider)	User filter/display
langfuse.session.id	Root span (via attribute_provider)	Session grouping
langfuse.trace.tags	Root span (via span_attributes)	Trace tags
gen_ai.request.model	Generation spans only	Model name + cost calculation
gen_ai.usage.input_tokens	Generation spans	Token usage
gen_ai.usage.output_tokens	Generation spans	Token usage
gen_ai.usage.reasoning.output_tokens	Generation spans, when reported	Reasoning tokens within output tokens
langfuse.observation.metadata.reasoning_summary	Responses generation spans with thinking: { display: :summarized }	Returned summary, when provided

Langfuse v4 reads overall input and output from langfuse.observation.input and langfuse.observation.output on the root span. This integration also keeps the deprecated trace input and output attributes for existing consumers. The Langfuse v4 ingestion path needs the x-langfuse-ingestion-version: 4 exporter header. See the Langfuse migration guide.

Reasoning tokens are already part of output tokens, so do not add the two counts. The summary field records only text returned for a requested Responses summary. Encrypted reasoning signatures stay in conversation history for stateless replay and are not added to Langfuse spans. Treat persisted conversation history as sensitive data.

EU vs US Cloud

• US: https://cloud.langfuse.com
• EU: https://eu.cloud.langfuse.com

Set LANGFUSE_HOST accordingly. Self-hosted instances use your own URL.

Complete Example

require "agents"
require "agents/instrumentation"
require "opentelemetry-sdk"
require "opentelemetry-exporter-otlp"
require "base64"

# --- Configure Agents ---
Agents.configure do |config|
  config.openai_api_key = ENV["OPENAI_API_KEY"]
  config.default_model = "gpt-4o-mini"
end

# --- Configure OTel with Langfuse ---
langfuse_host = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
auth_token = Base64.strict_encode64(
  "#{ENV["LANGFUSE_PUBLIC_KEY"]}:#{ENV["LANGFUSE_SECRET_KEY"]}"
)

OpenTelemetry::SDK.configure do |c|
  c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
      OpenTelemetry::Exporter::OTLP::Exporter.new(
        endpoint: "#{langfuse_host}/api/public/otel/v1/traces",
        headers: { "Authorization" => "Basic #{auth_token}", "x-langfuse-ingestion-version" => "4" }
      )
    )
  )
end

tracer = OpenTelemetry.tracer_provider.tracer("my-app")

# --- Build agents ---
triage = Agents::Agent.new(name: "Triage", instructions: "Route users...")
billing = Agents::Agent.new(name: "Billing", instructions: "Handle billing...")
support = Agents::Agent.new(name: "Support", instructions: "Technical support...")

triage.register_handoffs(billing, support)
billing.register_handoffs(triage)
support.register_handoffs(triage)

# --- Create runner with instrumentation ---
runner = Agents::Runner.with_agents(triage, billing, support)

Agents::Instrumentation.install(runner,
  tracer: tracer,
  trace_name: "customer_support",
  attribute_provider: ->(ctx) {
    {
      "langfuse.user.id" => ctx.context[:user_id].to_s,
      "langfuse.session.id" => ctx.context[:session_id].to_s
    }
  }
)

# --- Run conversations ---
result = runner.run("I have a billing question",
  context: { user_id: "user_123", session_id: "sess_456" })

puts result.output

# Ensure spans are flushed before exit
at_exit { OpenTelemetry.tracer_provider.force_flush }

Troubleshooting

“undefined” values in Langfuse

Langfuse renders empty string attributes as “undefined”. The instrumentation guards against this by not setting attributes when values are nil or empty. If you see “undefined”, check that your agents are producing output content.

Double-counted costs

If token costs appear inflated, verify that gen_ai.request.model is only set on GENERATION spans, not on container or root spans. Agents::Instrumentation handles this by default. If you set custom span_attributes that include gen_ai.request.model, costs will be double-counted.

Empty spans / missing data

• Ensure opentelemetry-sdk is installed (not just opentelemetry-api)
• Call OpenTelemetry.tracer_provider.force_flush before process exit
• Verify your OTLP endpoint is reachable and credentials are correct
• Check that Agents::Instrumentation.install returns the runner (returns nil if OTel is unavailable)

Spans not appearing in Langfuse

• Verify the endpoint includes /api/public/otel/v1/traces
• Check that the Authorization header uses Basic (not Bearer) with base64-encoded pk:sk
• Use BatchSpanProcessor for production; SimpleSpanProcessor can be useful for debugging
• SSL CRL errors on Ruby 3.4+: The OTLP exporter silently fails when SSL certificate revocation list (CRL) checks fail. The exporter reports SUCCESS but no data arrives. Fix by passing ssl_verify_mode: OpenSSL::SSL::VERIFY_NONE to the exporter in development, or ensure your system CA certificates are up to date

End-to-end check

The optional Langfuse check stubs the model response, exports real OTel spans to your Langfuse project, and reads them back through the observations API. It covers a handoff, token usage, session and user attributes, and a failed tool. Set LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, and LANGFUSE_SECRET_KEY, then run:

RUN_LANGFUSE_E2E=1 bundle exec rspec spec/integration/langfuse_e2e_spec.rb

The model call is stubbed, so this check does not require a provider API key. Use the separate live LLM smoke suite when you also need to check provider access.