From e27e83f64552b974ef36d510d1e3c4e72a78c0af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jannik=20Maierh=C3=B6fer?= <48529566+jannikmaierhoefer@users.noreply.github.com> Date: Wed, 12 Feb 2025 19:08:05 +0100 Subject: [PATCH] docs: add Langfuse OpenTelemetry guide (#601) --- docs/source/en/tutorials/inspect_runs.md | 101 ++++++++++++++++++++++- 1 file changed, 97 insertions(+), 4 deletions(-) diff --git a/docs/source/en/tutorials/inspect_runs.md b/docs/source/en/tutorials/inspect_runs.md index 7727bbf..cb53eed 100644 --- a/docs/source/en/tutorials/inspect_runs.md +++ b/docs/source/en/tutorials/inspect_runs.md @@ -20,7 +20,7 @@ rendered properly in your Markdown viewer. > [!TIP] > If you're new to building agents, make sure to first read the [intro to agents](../conceptual_guides/intro_agents) and the [guided tour of smolagents](../guided_tour). -### Why log your agent runs? +## Why log your agent runs? Agent runs are complicated to debug. @@ -32,7 +32,7 @@ So using instrumentation to record agent runs is necessary in production for lat We've adopted the [OpenTelemetry](https://opentelemetry.io/) standard for instrumenting agent runs. -This means that you can just run some instrumentation code, then run your agents normally, and everything gets logged into your platform. +This means that you can just run some instrumentation code, then run your agents normally, and everything gets logged into your platform. Below are some examples of how to do this with different OpenTelemetry backends. Here's how it then looks like on the platform: @@ -41,7 +41,7 @@ Here's how it then looks like on the platform: -### Setting up telemetry with Arize AI Phoenix +## Setting up telemetry with Arize AI Phoenix First install the required packages. Here we install [Phoenix by Arize AI](https://github.com/Arize-ai/phoenix) because that's a good solution to collect and inspect the logs, but there are other OpenTelemetry-compatible platforms that you could use for this collection & inspection part. ```shell @@ -106,4 +106,97 @@ You can then navigate to `http://0.0.0.0:6006/projects/` to inspect your run! -You can see that the CodeAgent called its managed ToolCallingAgent (by the way, the managed agent could be have been a CodeAgent as well) to ask it to run the web search for the U.S. 2024 growth rate. Then the managed agent returned its report and the manager agent acted upon it to calculate the economy doubling time! Sweet, isn't it? \ No newline at end of file +You can see that the CodeAgent called its managed ToolCallingAgent (by the way, the managed agent could be have been a CodeAgent as well) to ask it to run the web search for the U.S. 2024 growth rate. Then the managed agent returned its report and the manager agent acted upon it to calculate the economy doubling time! Sweet, isn't it? + +## Setting up telemetry with Langfuse + +This part shows how to monitor and debug your Hugging Face **smolagents** with **Langfuse** using the `SmolagentsInstrumentor`. + +> **What is Langfuse?** [Langfuse](https://langfuse.com) is an open-source platform for LLM engineering. It provides tracing and monitoring capabilities for AI agents, helping developers debug, analyze, and optimize their products. Langfuse integrates with various tools and frameworks via native integrations, OpenTelemetry, and SDKs. + +### Step 1: Install Dependencies + +```python +%pip install smolagents +%pip install opentelemetry-sdk opentelemetry-exporter-otlp openinference-instrumentation-smolagents +``` + +### Step 2: Set Up Environment Variables + +Set your Langfuse API keys and configure the OpenTelemetry endpoint to send traces to Langfuse. Get your Langfuse API keys by signing up for [Langfuse Cloud](https://cloud.langfuse.com) or [self-hosting Langfuse](https://langfuse.com/self-hosting). + +Also, add your [Hugging Face token](https://huggingface.co/settings/tokens) (`HF_TOKEN`) as an environment variable. + +```python +import os +import base64 + +LANGFUSE_PUBLIC_KEY="pk-lf-..." +LANGFUSE_SECRET_KEY="sk-lf-..." +LANGFUSE_AUTH=base64.b64encode(f"{LANGFUSE_PUBLIC_KEY}:{LANGFUSE_SECRET_KEY}".encode()).decode() + +os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "https://cloud.langfuse.com/api/public/otel" # EU data region +# os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "https://us.cloud.langfuse.com/api/public/otel" # US data region +os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = f"Authorization=Basic {LANGFUSE_AUTH}" + +# your Hugging Face token +os.environ["HF_TOKEN"] = "hf_..." +``` + +### Step 3: Initialize the `SmolagentsInstrumentor` + +Initialize the `SmolagentsInstrumentor` before your application code. Configure `tracer_provider` and add a span processor to export traces to Langfuse. `OTLPSpanExporter()` uses the endpoint and headers from the environment variables. + + +```python +from opentelemetry.sdk.trace import TracerProvider + +from openinference.instrumentation.smolagents import SmolagentsInstrumentor +from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter +from opentelemetry.sdk.trace.export import SimpleSpanProcessor + +trace_provider = TracerProvider() +trace_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter())) + +SmolagentsInstrumentor().instrument(tracer_provider=trace_provider) +``` + +### Step 4: Run your smolagent + +```python +from smolagents import ( + CodeAgent, + ToolCallingAgent, + DuckDuckGoSearchTool, + VisitWebpageTool, + HfApiModel, +) + +model = HfApiModel( + model_id="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" +) + +search_agent = ToolCallingAgent( + tools=[DuckDuckGoSearchTool(), VisitWebpageTool()], + model=model, + name="search_agent", + description="This is an agent that can do web search.", +) + +manager_agent = CodeAgent( + tools=[], + model=model, + managed_agents=[search_agent], +) +manager_agent.run( + "How can Langfuse be used to monitor and improve the reasoning and decision-making of smolagents when they execute multi-step tasks, like dynamically adjusting a recipe based on user feedback or available ingredients?" +) +``` + +### Step 5: View Traces in Langfuse + +After running the agent, you can view the traces generated by your smolagents application in [Langfuse](https://cloud.langfuse.com). You should see detailed steps of the LLM interactions, which can help you debug and optimize your AI agent. + +![smolagents example trace](https://langfuse.com/images/cookbook/integration-smolagents/smolagent_example_trace.png) + +_[Public example trace in Langfuse](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/ce5160f9bfd5a6cd63b07d2bfcec6f54?timestamp=2025-02-11T09%3A25%3A45.163Z&display=details)_