Repository F# setup
open System
open System.IO
open System.Threading
open System.Threading.Tasks
open Axial
open Axial.Layers
open Axial.Console
open Axial.FileSystem
open Axial.Hosting
open Axial.Hosting.Browser
open Axial.Hosting.Node
open Axial.PlatformService
open Axial.State
open Axial.Telemetry
open Axial.Telemetry.JavaScript

Trace workflows and inspect them in Aspire

Telemetry answers three different questions:

  • What happened? A trace shows the spans that ran and how long each one took.
  • Why did it fail? Span status and attributes distinguish typed failures, defects, and interruption.
  • What was still running? Fiber metrics and dumps expose background work that an ordinary task trace can miss.

Axial publishes standard .NET ActivitySource spans and Meter instruments. OpenTelemetry collects those signals and exports them to a backend. The .NET Aspire dashboard is the fastest way to see the result locally.

Use Axial.Telemetry on .NET. For Node and browser applications compiled with Fable, use Axial.Telemetry.JavaScript. Both adapters read the same ambient Context and emit the same axial.flow.* vocabulary.

Before you begin

Install the .NET adapter and the OpenTelemetry host packages:

dotnet add package Axial.Telemetry
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore

You need an OTLP receiver such as the Aspire dashboard, an OpenTelemetry Collector, Jaeger, Tempo, or a hosted observability service. Axial does not choose an exporter or send telemetry by itself. The runnable Axial.ReferenceApp configures the SDK and OTLP exporters, launches a local Aspire dashboard, and generates traces, metrics, logs, and a fiber dump from one endpoint.

Configure OpenTelemetry once

Create an application-owned ActivitySource, then subscribe to both that source and Axial's runtime source at the host boundary:

The builder below is the WebApplicationBuilder from an ASP.NET Core Program.fs; this block is a fragment of that host setup, not a standalone snippet.

Fragment
open System.Diagnostics

let applicationActivitySource = new ActivitySource("Checkout.Api")

builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(fun resource ->
        resource.AddService("checkout-api") |> ignore)
    .WithTracing(fun tracing ->
        tracing
            .AddSource(applicationActivitySource.Name, "Axial")
            .AddAspNetCoreInstrumentation()
            .AddOtlpExporter()
        |> ignore)
    .WithMetrics(fun metrics ->
        metrics
            .AddMeter("Axial")
            .AddOtlpExporter()
        |> ignore)
|> ignore

These names answer different questions:

  • checkout-api is the service name. Aspire groups telemetry by the deployed application or service that produced it.
  • Checkout.Api is the application's instrumentation scope. Workflow spans describing checkout behavior should use this source rather than appearing to be operations owned by Axial.
  • Axial is Axial's runtime instrumentation scope. Automatic fiber spans and fiber metrics remain here because they describe the Flow runtime.
  • checkout.submit is the span name for one operation.

An ActivitySource only publishes spans. The OpenTelemetry SDK listens, samples, and exports them. Register every source used by Activity.traceOn, Activity.traceWithSource, or captured in an ActivityTracer; otherwise .NET returns null from StartActivity and the workflow runs without a span.

Trace a workflow

Wrap a workflow at a boundary that has an operational meaning. Pass the application source because this span describes user code; Axial supplies the Flow-aware tracing behavior around it:

The examples below use this checkout workflow, and a listener that records each finished span with its tags, as the OpenTelemetry SDK would export them:

Show page setup
// Setup for the checked examples on this page.
open System
open Axial
open Axial.PlatformService
open Axial.Telemetry
open Axial.Telemetry.JavaScript
/// Fails the docs test when an example's result differs from the value shown.
let shouldEqual expected actual =
    if actual <> expected then failwithf "Expected %A but got %A" expected actual
Verified
open System.Diagnostics

type Order = { OrderId: int; Total: decimal }

type CheckoutError =
    | CardDeclined

    static member describe(error: CheckoutError) =
        match error with
        | CardDeclined -> "card declined"

let applicationActivitySource = new ActivitySource("Checkout.Api")

let checkout (order: Order) : Flow<unit, CheckoutError, int> =
    if order.Total > 0m then Flow.ok order.OrderId else Flow.fail CardDeclined

let order = { OrderId = 42; Total = 19.95m }
let userId = "user-7"
let tenantId = "acme"

/// Runs a flow and returns the spans that finished on the application source, with their tags.
let spansOf (workflow: Flow<unit, CheckoutError, int>) : (string * Map<string, string>) list =
    FlowStream.fromCallback 64 (fun emit ->
        Flow.delay (fun () ->
            let listener =
                new ActivityListener(
                    ShouldListenTo = (fun source -> source.Name = "Checkout.Api"),
                    Sample = SampleActivity<ActivityContext>(fun _ -> ActivitySamplingResult.AllData),
                    ActivityStopped =
                        (fun activity ->
                            let tags = activity.TagObjects |> Seq.map (fun tag -> tag.Key, string tag.Value) |> Map.ofSeq
                            emit.Value(activity.DisplayName, tags) |> ignore))
            ActivitySource.AddActivityListener listener
            workflow |> Flow.run () |> ignore
            emit.End()
            Flow.ok (listener :> IDisposable)))
    |> FlowStream.runCollect
    |> Flow.run ()
    |> function Exit.Success spans -> spans | exit -> failwithf "Span collection failed: %A" exit
Requires & types
Requires: open Axial; open System; open System.Diagnostics (packages: Axial)
val applicationActivitySource: ActivitySource
val checkout: Order -> Flow<unit,CheckoutError,int>
val order: Order
val userId: string
val tenantId: string
val spansOf: Flow<unit,CheckoutError,int> -> (string * Map<string,string>) list
Verified
let submitted =
    checkout order
    |> Activity.traceWithSource applicationActivitySource CheckoutError.describe "checkout.submit"
Requires & types
Requires: open Axial.Telemetry (packages: Axial.Telemetry)
val submitted: Flow<unit,CheckoutError,int>
Verified
spansOf submitted |> List.map fst |> shouldEqual [ "checkout.submit" ]

Activity.traceWithSource accepts the source and a renderer for the workflow's typed error. Use Activity.traceOn when string is an acceptable representation:

Verified
let submittedOn =
    checkout order
    |> Activity.traceOn applicationActivitySource "checkout.submit"
Requires & types
Requires: open Axial.Telemetry (packages: Axial.Telemetry)
val submittedOn: Flow<unit,CheckoutError,int>

Trace without repeating the source at every call site

Capture applicationActivitySource once as an ActivityTracer, then call .Trace wherever you need a span:

Verified
let checkoutTracer = ActivityTracer.create applicationActivitySource

let submittedWithTracer =
    checkout order
    |> checkoutTracer.Trace "checkout.submit"
Requires & types
Requires: open Axial.Telemetry (packages: Axial.Telemetry)
val checkoutTracer: ActivityTracer
val submittedWithTracer: Flow<unit,CheckoutError,int>

For a workflow tree that traces from many places, install the tracer once near the composition root instead, and call the ambient Activity.trace/Activity.traceWith from anywhere underneath it, without passing the source at each call site:

Verified
// deep inside the workflow tree, in code that has no reference to checkoutTracer:
let innerStep = checkout order |> Activity.trace "checkout.submit"

let application = innerStep |> Activity.withTracer checkoutTracer
Requires & types
Requires: open Axial.Telemetry (packages: Axial.Telemetry)
val innerStep: Flow<unit,CheckoutError,int>
val application: Flow<unit,CheckoutError,int>
Verified
spansOf application |> List.map fst |> shouldEqual [ "checkout.submit" ]

Activity.trace and Activity.traceWith read the tracer installed by the nearest enclosing Activity.withTracer; forked child fibers inherit it too. There is no default source to fall back to: calling Activity.trace outside a withTracer scope throws immediately, so a workflow can never end up silently tagged under the wrong instrumentation scope. Use the ambient form for workflow trees under one application source, and traceOn/traceWithSource (or a second ActivityTracer) when a call site needs a different source than the one currently installed.

The span starts when the workflow runs and stops when its asynchronous execution settles. Axial records:

Exit Span status Attributes
success Ok axial.flow.outcome = success
typed failure Error axial.flow.outcome = fail, axial.flow.error
defect Error axial.flow.outcome = die, exception.*
interruption unset axial.flow.outcome = interrupt, axial.flow.interrupted = true
composite cause dominant outcome axial.flow.cause with the rendered cause tree

Do not trace every combinator. Trace operations that you would search for in an incident: checkout.submit, invoice.generate, or outbox.deliver.

Add attributes

An attribute adds searchable context to the active span. Axial stores attributes in an immutable ambient Context. The context is separate from the workflow environment because it is execution metadata, not an application service. Nested scopes restore the previous value when they finish, and forked fibers inherit the context present at the fork.

Use a curated OpenTelemetry helper for a common semantic attribute:

Verified
let submittedByUser =
    checkout order
    |> Context.withEndUserId userId
    |> Activity.traceWithSource applicationActivitySource CheckoutError.describe "checkout.submit"
Requires & types
Requires: open Axial.Telemetry (packages: Axial, Axial.Telemetry)
val submittedByUser: Flow<unit,CheckoutError,int>
Verified
spansOf submittedByUser |> List.map (fun (_, tags) -> Map.tryFind "enduser.id" tags) |> shouldEqual [ Some "user-7" ]

enduser.id is an OpenTelemetry semantic-convention attribute, currently marked development by OpenTelemetry. It can contain identifying information; use a stable opaque identifier only when your privacy policy permits exporting it. Axial documents the stability and scope of every convention for which it provides a convenience helper. Configure resource attributes such as service identity in the OpenTelemetry SDK instead of copying them onto every span.

Define typed keys for application attributes:

Verified
module CheckoutAttributes =
    let tenantId = AttributeKey.string "example.tenant.id"
    let retryCount = AttributeKey.int64 "example.checkout.retry_count"
Requires & types
Requires: open Axial.Telemetry (packages: Axial)
val tenantId: AttributeKey<string>
val retryCount: AttributeKey<int64>

Attach values without boxing or runtime conversion:

Verified
let submittedForTenant =
    checkout order
    |> Context.withAttributes [
        Context.attribute CheckoutAttributes.tenantId tenantId
        Context.attribute CheckoutAttributes.retryCount 2L
    ]
    |> Activity.traceWithSource applicationActivitySource CheckoutError.describe "checkout.submit"
Requires & types
Requires: open Axial.Telemetry (packages: Axial, Axial.Telemetry)
val submittedForTenant: Flow<unit,CheckoutError,int>
Verified
spansOf submittedForTenant
|> List.map (fun (_, tags) -> Map.tryFind "example.tenant.id" tags, Map.tryFind "example.checkout.retry_count" tags)
|> shouldEqual [ Some "acme", Some "2" ]

A typed key prevents attaching an integer to a string attribute. Supported values are strings, Booleans, 64-bit integers, floating-point values, and homogeneous lists of those types.

Build a context once when several workflows share the same metadata:

Verified
let requestContext =
    Context.empty
    |> Context.addEndUserId userId
    |> Context.add (Context.attribute CheckoutAttributes.tenantId tenantId)

let inRequest = application |> Context.withContext requestContext
Requires & types
Requires: open Axial.Telemetry (packages: Axial)
val requestContext: TelemetryContext
val inRequest: Flow<unit,CheckoutError,int>

Read the currently scoped context from a workflow when an integration needs to inspect it:

Verified
let exportContext (context: TelemetryContext) = Context.tryFind CheckoutAttributes.tenantId context

let exportedTenant : Flow<unit, CheckoutError, string option> =
    flow {
        let! telemetryContext = Context.current
        return exportContext telemetryContext
    }
    |> Context.withContext requestContext
Requires & types
Requires: open Axial; open Axial.Telemetry (packages: Axial)
val exportContext: TelemetryContext -> string option
val exportedTenant: Flow<unit,CheckoutError,string option>
Verified
exportedTenant |> Flow.run () |> shouldEqual (Exit.Success(Some "acme"))
Requires & types
Requires: open Axial (packages: Axial)

Most workflows should scope attributes rather than read the whole context. Adapters use Context.current at integration boundaries; application dependencies still belong in the Flow environment.

Attribute names are contracts with dashboards and alerts. Follow OpenTelemetry semantic conventions when one applies. Use an application-owned prefix otherwise. Do not attach secrets, unrestricted personal information, or high-cardinality values to metrics. A correlation value that must cross process boundaries may belong in OpenTelemetry baggage; trace identity itself belongs in trace and span context, not in a duplicate attribute.

Observe fibers

A successful root workflow can still have failed background work. Install fiber telemetry once around the application:

Verified
let withDefectSpans = application |> FiberTelemetry.observe
Requires & types
Requires: open Axial.Telemetry.JavaScript (packages: Axial.Telemetry.JavaScript)
val withDefectSpans: Flow<unit,CheckoutError,int>

This records defect spans for failed fibers and for defects that no code can observe. To create a span for every forked fiber, use FiberTelemetry.observeWithSpans:

Verified
let withFiberSpans = application |> FiberTelemetry.observeWithSpans
Requires & types
Requires: open Axial.Telemetry.JavaScript (packages: Axial.Telemetry.JavaScript)
val withFiberSpans: Flow<unit,CheckoutError,int>

Span-per-fiber mode provides more detail and more data. Use defect-only observation by default, and enable span-per-fiber when you need fork-to-settle timing or a complete concurrency tree.

Add runtime metrics independently:

Verified
let withMetrics =
    application
    |> FiberMetrics.observe
    |> FiberTelemetry.observe
Requires & types
Requires: open Axial.Telemetry; open Axial.Telemetry.JavaScript (packages: Axial.Telemetry, Axial.Telemetry.JavaScript)
val withMetrics: Flow<unit,CheckoutError,int>

The Axial meter records starts, live fibers, settlements, duration, and unobserved defects. A rising live-fiber count without matching settlements indicates stuck or leaked work.

Watch queues and hub subscriptions

Report a queue or a hub subscription under a name with QueueMetrics.observe. It is reported until the scope that registered it closes, so register a subscription in the fiber that consumes it:

Verified
let historian (readings: Hub<float>) (record: float -> Flow<unit, Never, unit>) : Flow<unit, Never, unit> =
    flow {
        let! history = readings |> Hub.subscribe (QueueStrategy.BackPressure 10_000)
        do! history |> QueueMetrics.observe "historian"
        do! history |> FlowStream.fromDequeue |> FlowStream.runForEachFlow record
    }
Requires & types
Requires: open Axial; open Axial.Telemetry (packages: Axial, Axial.Telemetry)
val historian: Hub<float> -> (float -> Flow<unit,Never,unit>) -> Flow<unit,Never,unit>

The figures are read only when the metrics pipeline collects, so an observed queue costs nothing extra per value. Each measurement carries an axial.queue.name tag. axial.queue.size, axial.queue.capacity, axial.queue.waiting_takers, and axial.queue.waiting_offerers are gauges; axial.queue.accepted, axial.queue.dropped, and axial.queue.evicted are counters. Alert when a lossless subscriber's size approaches its capacity, before it fills and holds up the publisher, and on the rate of drops or evictions for a lossy one.

A MeterListener can sample the Axial meter to check the observed figures, and to confirm a queue stops being reported after its scope closes:

Verified
open System.Diagnostics.Metrics

type QueueCapture() =
    let gate = obj ()
    let latest = Collections.Generic.Dictionary<string * string, int64>()
    let listener = new MeterListener()

    do
        listener.InstrumentPublished <- fun instrument metricListener ->
            if instrument.Meter.Name = "Axial" && instrument.Name.StartsWith "axial.queue." then
                metricListener.EnableMeasurementEvents instrument

        listener.SetMeasurementEventCallback<int64>(fun instrument value tags _ ->
            let queueName =
                tags.ToArray()
                |> Array.tryFind (fun tag -> tag.Key = "axial.queue.name")
                |> Option.map (fun tag -> string tag.Value)
                |> Option.defaultValue ""

            lock gate (fun () -> latest[(instrument.Name, queueName)] <- value))

        listener.Start()

    member _.Read(instrument: string, queueName: string) =
        lock gate (fun () -> latest.Clear())
        listener.RecordObservableInstruments()

        lock gate (fun () ->
            match latest.TryGetValue((instrument, queueName)) with
            | true, value -> Some value
            | _ -> None)

    interface IDisposable with
        member _.Dispose() = listener.Dispose()

let observeQueue (capture: QueueCapture) : Flow<unit, Never, (int64 option list) * int64 option> =
    flow {
        let! readings =
            flow {
                let! (display: Queue<int>) = Queue.sliding 2
                do! display |> QueueMetrics.observe "queue-metrics-display"
                do! display |> Queue.offerAll [ 1..5 ] |> Flow.ignore

                return
                    [ for instrument in
                          [ "axial.queue.size"
                            "axial.queue.capacity"
                            "axial.queue.accepted"
                            "axial.queue.evicted"
                            "axial.queue.dropped" ] ->
                          capture.Read(instrument, "queue-metrics-display") ]
            }
            |> Flow.scoped

        let afterScope = capture.Read("axial.queue.size", "queue-metrics-display")
        return readings, afterScope
    }
Requires & types
Requires: open Axial; open Axial.Telemetry; open System; open System.Diagnostics.Metrics (packages: Axial, Axial.Telemetry)
val observeQueue: QueueCapture -> Flow<unit,Never,(int64 option list * int64 option)>
Verified
use capture = new QueueCapture()

observeQueue capture
|> Flow.run ()
|> shouldEqual
    (Exit.Success([ Some 2L; Some 2L; Some 5L; Some 3L; Some 0L ], None)
     : Exit<(int64 option list) * int64 option, Never>)
Requires & types
Requires: open Axial (packages: Axial)

The sliding queue has capacity 2, accepted all five offers, evicted the first three, and ended with two values buffered; after its scope closed, axial.queue.size no longer reports the queue (None).

Capture a fiber dump

A trace explains completed and timed operations. A fiber dump shows the work that is live now.

Install a registry at the application edge:

Verified
let registry = FiberRegistry()

let observedApplication =
    application
    |> Flow.withFiberRegistry registry
    |> FiberMetrics.observe
    |> FiberTelemetry.observe
Requires & types
Requires: open Axial; open Axial.Telemetry; open Axial.Telemetry.JavaScript (packages: Axial, Axial.Telemetry, Axial.Telemetry.JavaScript)
val registry: FiberRegistry
val observedApplication: Flow<unit,CheckoutError,int>

Name long-lived work so the dump is readable:

Verified
let pollOutbox : Flow<ClockEnvironment, CheckoutError, unit> = Flow.sleep (TimeSpan.FromSeconds 5.0)

let startPoller : Flow<ClockEnvironment, CheckoutError, Fiber<CheckoutError, unit>> = Flow.forkNamed "outbox-poller" pollOutbox
Requires & types
Requires: open Axial; open System (packages: Axial)
val pollOutbox: Flow<ClockEnvironment,CheckoutError,unit>
val startPoller: Flow<ClockEnvironment,CheckoutError,Fiber<CheckoutError,unit>>

registry.DumpAt and FiberDumpTelemetry.record both take an IClock to timestamp the snapshot. Inside a workflow, read the environment's injected clock (Flow.envWith (fun (env: ClockEnvironment) -> env.Clock)) so tests and repeated runs get a deterministic timestamp. At the edge there is no environment to read: call registry.DumpAt(Clock.live) from a protected diagnostics endpoint, write it during a stuck shutdown, or attach it to the current trace, all with the live clock:

Verified
let recordDump () = FiberDumpTelemetry.record Clock.live registry
Requires & types
Requires: open Axial.PlatformService; open Axial.Telemetry (packages: Axial.PlatformService, Axial.Telemetry)
val recordDump: unit -> unit

FiberDumpTelemetry.record adds an axial.flow.fiber.dump event to the current activity. If no activity is current, it creates a standalone span. The event records which fibers were running when you captured the dump. See Supervision and fiber observability for registry and dump details.

View Axial in the Aspire dashboard

Aspire's dashboard accepts OTLP telemetry. The quickest complete example is Axial.ReferenceApp: run dotnet run --file apphost.cs from its directory, then call its /observability/demo endpoint as described in the example README.

If your Aspire service uses AddServiceDefaults(), keep that setup and add the application source, Axial's runtime source, and Axial's meter:

Fragment
builder.Services
    .AddOpenTelemetry()
    .WithTracing(fun tracing ->
        tracing.AddSource(applicationActivitySource.Name, "Axial") |> ignore)
    .WithMetrics(fun metrics -> metrics.AddMeter("Axial") |> ignore)
|> ignore

Run the application and open the dashboard:

  1. Open Traces and select an incoming request.
  2. Expand the request span to find checkout.submit or another Activity.trace span.
  3. Inspect its status and attributes, including enduser.id, application attributes, and axial.flow.outcome.
  4. Enable FiberTelemetry.observeWithSpans to see named child fibers in the same trace.
  5. Trigger FiberDumpTelemetry.record and inspect the axial.flow.fiber.dump event on the active span.
  6. Open Metrics and chart axial.flow.fibers.live and axial.flow.fibers.unobserved_defects.

Choose attributes, annotations, or logs

  • Use telemetry context attributes for values intentionally exported to tracing backends and queried across spans.
  • Use Flow.annotate for Axial runtime diagnostics. Telemetry exports these under axial.flow.annotation.*, preserving their separate namespace.
  • Use ILog for messages and exceptions that belong in the host logging pipeline.

The mechanisms share ambient scoping, but they are different operational contracts.

Next steps