Setup Sentry AI Agent Monitoring

Configure Sentry to track LLM calls, agent executions, tool usage, and token consumption.

Invoke This Skill When User asks to "monitor AI/LLM calls" or "track OpenAI/Anthropic usage" User wants "AI observability" or "agent monitoring" User asks about token usage, model latency, or AI costs Prerequisites

AI monitoring requires tracing enabled (tracesSampleRate > 0).

Detection First

Always detect installed AI SDKs before configuring:

JavaScript

grep -E '"(openai|@anthropic-ai/sdk|ai|@langchain|@google/genai)"' package.json

Python

grep -E '(openai|anthropic|langchain|huggingface)' requirements.txt pyproject.toml 2>/dev/null

Supported SDKs JavaScript Package Integration Min Sentry SDK Auto? openai openAIIntegration() 10.2.0 Yes @anthropic-ai/sdk anthropicAIIntegration() 10.12.0 Yes ai (Vercel) vercelAIIntegration() 10.6.0 Node only @langchain/ langChainIntegration() 10.22.0 Yes @langchain/langgraph langGraphIntegration() 10.25.0 Yes @google/genai googleGenAIIntegration() 10.14.0 Yes

*Vercel AI requires explicit setup for Edge runtime and experimental_telemetry per-call.

Python Package Install Min SDK openai pip install "sentry-sdk[openai]" 2.41.0 anthropic pip install "sentry-sdk[anthropic]" 2.x langchain pip install "sentry-sdk[langchain]" 2.x huggingface_hub pip install "sentry-sdk[huggingface_hub]" 2.x JavaScript Configuration Auto-enabled integrations (OpenAI, Anthropic, Google GenAI, LangChain)

Just ensure tracing is enabled. To capture prompts/outputs:

Sentry.init({ dsn: "YOUR_DSN", tracesSampleRate: 1.0, integrations: [ Sentry.openAIIntegration({ recordInputs: true, recordOutputs: true }), ], });

Next.js OpenAI (additional step required)

For Next.js projects using OpenAI, you must wrap the client:

import OpenAI from "openai"; import * as Sentry from "@sentry/nextjs";

const openai = Sentry.instrumentOpenAiClient(new OpenAI()); // Use 'openai' client as normal

LangChain / LangGraph (explicit) integrations: [ Sentry.langChainIntegration({ recordInputs: true, recordOutputs: true }), Sentry.langGraphIntegration({ recordInputs: true, recordOutputs: true }), ],

Vercel AI SDK

Add to sentry.edge.config.ts for Edge runtime:

integrations: [Sentry.vercelAIIntegration()],

Enable telemetry per-call:

await generateText({ model: openai("gpt-4o"), prompt: "Hello", experimental_telemetry: { isEnabled: true, recordInputs: true, recordOutputs: true }, });

Python Configuration import sentry_sdk from sentry_sdk.integrations.openai import OpenAIIntegration # or anthropic, langchain

sentry_sdk.init( dsn="YOUR_DSN", traces_sample_rate=1.0, send_default_pii=True, # Required for prompt capture integrations=[OpenAIIntegration(include_prompts=True)], )

Manual Instrumentation

Use when no supported SDK is detected.

Span Types op Value Purpose gen_ai.request Individual LLM calls gen_ai.invoke_agent Agent execution lifecycle gen_ai.execute_tool Tool/function calls gen_ai.handoff Agent-to-agent transitions Example (JavaScript) await Sentry.startSpan({ op: "gen_ai.request", name: "LLM request gpt-4o", attributes: { "gen_ai.request.model": "gpt-4o" }, }, async (span) => { span.setAttribute("gen_ai.request.messages", JSON.stringify(messages)); const result = await llmClient.complete(prompt); span.setAttribute("gen_ai.usage.input_tokens", result.inputTokens); span.setAttribute("gen_ai.usage.output_tokens", result.outputTokens); return result; });

Key Attributes Attribute Description gen_ai.request.model Model identifier gen_ai.request.messages JSON input messages gen_ai.usage.input_tokens Input token count gen_ai.usage.output_tokens Output token count gen_ai.agent.name Agent identifier gen_ai.tool.name Tool identifier PII Considerations

Prompts/outputs are PII. To capture:

JS: recordInputs: true, recordOutputs: true per-integration Python: include_prompts=True + send_default_pii=True Troubleshooting Issue Solution AI spans not appearing Verify tracesSampleRate > 0, check SDK version Token counts missing Some providers don't return tokens for streaming Prompts not captured Enable recordInputs/include_prompts Vercel AI not working Add experimental_telemetry to each call