Skip to content

observability.md

Latest commit

 

History

History
223 lines (160 loc) · 8.13 KB

observability.md

File metadata and controls

223 lines (160 loc) · 8.13 KB
Raw

Observability & Distributed Tracing

Monitoring and debugging distributed workflows is critical for production systems. The Durable Task Scheduler ecosystem provides multiple layers of observability: a built-in dashboard for business-level monitoring and OpenTelemetry support for infrastructure-level tracing.

Table of Contents

  • Built-in Dashboard
  • Distributed Tracing with OpenTelemetry
  • Durable Functions Distributed Tracing
  • Durable Task SDKs Tracing
  • Exporter Options
  • Next Steps

Built-in Dashboard

Every Durable Task Scheduler instance (including the local emulator) comes with a monitoring dashboard out of the box.

What you can do:

  • View all orchestrations — filter by status, name, time range
  • Drill into execution history — see each activity, sub-orchestration, and event
  • Monitor timing — identify slow activities and bottlenecks
  • Manage instances — pause, terminate, restart, or purge orchestration instances
  • Multi-agent visualization — trace complex AI agent workflows across multiple orchestrations

Access the dashboard:

Local (Emulator):

http://localhost:8082

Azure: Navigate to your Durable Task Scheduler resource → Task Hub → Dashboard URL, or go to dashboard.durabletask.io and register your endpoint.

📖 Dashboard documentation →

Distributed Tracing with OpenTelemetry

For infrastructure-level observability — latency analysis, cross-service correlation, and performance profiling — you can use OpenTelemetry (OTel) distributed tracing.

How it complements the dashboard

Aspect Built-in Dashboard OpenTelemetry Tracing
Focus Business logic (orchestration state) Infrastructure (latency, errors, dependencies)
Granularity Orchestration/activity level Span-level (including HTTP, DB calls)
Cross-service Within task hub Across all services (end-to-end)
Storage Managed by DTS Your choice (App Insights, Jaeger, etc.)
Best for "What happened in this orchestration?" "Where is the bottleneck across my system?"

Durable Functions Distributed Tracing

Durable Functions supports Distributed Tracing V2, which correlates orchestrations, entities, and activities into unified traces.

Setup

  1. Update host.json:
{
  "extensions": {
    "durableTask": {
      "tracing": {
        "distributedTracingEnabled": true,
        "version": "V2"
      }
    }
  }
}

  1. Requirements:

    • .NET Isolated: Microsoft.Azure.Functions.Worker.Extensions.DurableTask >= v1.4.0
    • Non-.NET: Microsoft.Azure.WebJobs.Extensions.DurableTask >= v3.2.0

  2. Configure Application Insights — If your Function app has Application Insights enabled, traces will appear automatically.

Viewing traces in Application Insights

  1. Navigate to your Application Insights resource
  2. Go to Transaction Search
  3. Filter for Request and Dependency events with Durable Functions prefixes (orchestration:, activity:)
  4. Click on an event to see the end-to-end Gantt chart

The Gantt chart shows the full orchestration flow — when each activity started, how long it took, and the data flow between them.

📖 Durable Functions diagnostics →

Durable Task SDKs Tracing

The Durable Task SDKs integrate with OpenTelemetry. Some spans come from your app and some come from DTS, depending on the SDK. Configure the tracer provider and exporter for your language.

.NET

Add OpenTelemetry packages to your project:

<PackageReference Include="OpenTelemetry" Version="1.*" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.*" />
<PackageReference Include="OpenTelemetry.Exporter.OtlpProtocol" Version="1.*" />

Configure tracing in your worker:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddSource("Microsoft.DurableTask")
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri("http://localhost:4317");
            });
    });

Python

Install OpenTelemetry packages:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

Configure tracing:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

Go

The Go SDK (github.com/microsoft/durabletask-go v1.0.0-beta3, Go 1.25.0+) passes W3C trace context from the caller through DTS to activities. Set up an OpenTelemetry tracer provider and exporter in your app, start a caller span, and pass its context when you schedule an orchestration. Activities can create their own spans from that context.

DTS creates the orchestration, activity, and timer spans. The Go worker does not export these spans locally. Matching trace IDs show that context was passed; they do not show the full DTS trace.

Start with the Go OpenTelemetry sample. With the emulator running:

cd samples/durable-task-sdks/go
go mod download
go run ./opentelemetry-tracing

To send application spans to an OTLP/HTTP collector or Jaeger, set the endpoint:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 go run ./opentelemetry-tracing

Use port 4318 (OTLP/HTTP), not 4317 (OTLP/gRPC). If the collector is not available, the sample fails. This setting does not export DTS spans.

See the SDK tracing example and the OpenTelemetry Go documentation for exporter setup.

Local Development with Jaeger

For local development, you can use Jaeger to visualize traces alongside the DTS emulator.

Docker Compose setup

version: '3.8'
services:
  dts-emulator:
    image: mcr.microsoft.com/dts/dts-emulator:latest
    ports:
      - "8080:8080"
      - "8082:8082"

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP
    environment:
      - COLLECTOR_OTLP_ENABLED=true

After starting both services:

Exporter Options

Exporter Best For Setup Complexity
Application Insights Azure production workloads Low (built-in for Azure Functions)
Jaeger Local development, self-hosted Low (Docker)
Zipkin Lightweight tracing Low (Docker)
Grafana Tempo Grafana ecosystem users Medium
OTLP (generic) Any OTel-compatible backend Varies

For Azure production workloads, use Application Insights with the Azure Monitor OpenTelemetry Distro in supported languages. The distro is not a Go integration. For Go, use an OpenTelemetry Go exporter and a collector.

Next Steps