Axial
Repository F# setup
open System
open System.IO
open System.Threading
open System.Threading.Tasks
open Axial
open Axial.Layers
open Axial.Console
open Axial.FileSystem
open Axial.Hosting
open Axial.Hosting.Browser
open Axial.Hosting.Node
open Axial.PlatformService
open Axial.State
open Axial.Telemetry
open Axial.Telemetry.JavaScript

Torture Tests

Most of Axial's concurrency guarantees are about races. An interrupted take never loses a value. A lossless subscriber sees every value in order. A scope releases what it acquired, however its fibers end. Each guarantee is easy to state and easy to break by accident, and a race that breaks it only now and then will not show up in a small example.

Each torture scenario runs one area of the library hard: many fibers, random interruptions, failures, and shutdowns in the middle of the work. Then it checks the invariants that must hold whatever the interleaving was. A check never depends on timing, such as "this finished within 10 ms", so a violated check is a defect, not a slow machine.

Run them yourself

The scenarios are a runnable example in examples/Axial.TortureTest. From a clone of the repository, run every scenario for 10 rounds:

dotnet run --project examples/Axial.TortureTest

Name a scenario, a round count, or both:

dotnet run --project examples/Axial.TortureTest -- queues 200
dotnet run --project examples/Axial.TortureTest -- all 100

The program prints each invariant with a check mark and the number of rounds it held on. It exits with a non-zero code if any invariant was ever violated, so it can gate a CI job or run as a soak test on your own hardware.

Rounds and seeds

A scenario runs once per round, and each round has a seed. The seed fixes every random choice the scenario makes: which fiber to interrupt, which strategy a subscriber uses, how long an operation waits. A violated invariant is reported with the seeds it failed on, so the same choices can be run again. The interleaving of fibers still differs from run to run, which is what the rounds are for.

The runner around the scenarios:

/// Runs one round, turning a round that fails outright into a violated check.
let runRound (scenario: Scenario) (seed: int) : Check list =
    match (scenario.Run(Round(seed, 100))).RunSynchronously(clockEnvironment) with
    | Exit.Success checks -> checks
    | Exit.Failure cause ->
        let reason = Cause.prettyPrint (fun (_: Never) -> "") cause
        [ check $"the round completed (it ended with {reason})" false ]

/// Runs every round of a scenario, prints each invariant with the rounds it held on and the seeds it failed with,
/// and returns whether every invariant held on every round.
let report (scenario: Scenario) (rounds: int) : bool =
    let results = [ for seed in 1..rounds -> seed, runRound scenario seed ]
    let invariants = results |> List.collect (snd >> List.map _.Invariant) |> List.distinct
    printfn "%s: %s" scenario.Name scenario.Title

    for invariant in invariants do
        let failedSeeds =
            [ for seed, checks in results do
                  if checks |> List.exists (fun check -> check.Invariant = invariant && not check.Held) then seed ]

        match failedSeeds with
        | [] -> printfn "  ✔ %s (%d/%d)" invariant rounds rounds
        | seeds ->
            let listed = seeds |> List.map string |> String.concat ", "
            printfn "  ✗ %s (%d/%d; failed with seeds %s)" invariant (rounds - seeds.Length) rounds listed

    results |> List.forall (snd >> List.forall _.Held)

/// Usage: <c>dotnet run -- [scenario | all] [rounds]</c>. Runs every scenario for 10 rounds by default, and exits
/// with a non-zero code if any invariant was violated.
[<EntryPoint>]
let main arguments =
    let selected, rounds =
        match arguments with
        | [||] -> Some Scenarios.all, 10
        | [| name |] when name = "all" -> Some Scenarios.all, 10
        | [| name |] when Seq.forall System.Char.IsDigit name -> Some Scenarios.all, int name
        | [| name |] -> Scenarios.tryFind name |> Option.map List.singleton, 10
        | [| name; count |] when name = "all" -> Some Scenarios.all, int count
        | [| name; count |] -> Scenarios.tryFind name |> Option.map List.singleton, int count
        | _ -> None, 0

    match selected with
    | None ->
        let names = Scenarios.all |> List.map _.Name |> String.concat ", "
        printfn "Usage: dotnet run -- [scenario | all] [rounds]. Scenarios: %s" names
        2
    | Some scenarios ->
        let results = [ for scenario in scenarios -> report scenario rounds ]
        if List.forall id results then 0 else 1

How Axial runs them

  • On every build, the .NET test suite runs every scenario for 10 seeds and fails if any invariant is violated.
  • The same scenario files are compiled to JavaScript with Fable and run on Node at a fifth of their size, so the JavaScript runtime keeps the same guarantees.
  • A coverage test fails when a public member of the runtime and concurrency modules is not used by any scenario. A new API in those modules ships with a scenario that exercises it.

The scenarios have found real defects: a sleep that could end early, failures that recovery combinators did not see once they had been traced, a value lost when a stream's timer fired at the moment it arrived, and, on JavaScript, fibers that lost their scope after a long synchronous stretch.

Scenarios

Scenario What it stresses
Pipeline Queues, a hub, a SubscriptionRef, and graceful fibers together, stopped by closing a scope.
Queues Producers and consumers on one queue, with takes and offers interrupted, and every strategy.
Hubs Subscribers joining and leaving throughout, and every publish style.
Semaphores Permits under contention, with waiters and holders interrupted.
Deferreds Completions racing each other, and awaiters interrupted or arriving late.
Refs Ref and SubscriptionRef updated concurrently.
STM Transfers, blocking withdrawals, and atomic audits.
Fibers Latest-wins slots, inherited context, the fiber registry, and observers.
Caches Single-flight lookups with interrupted callers, failures, and invalidation.
Schedules Retry, repeat, supervise, and timeouts.
Parallel Bounded traversals, pooled resources, fail-fast zips, and races.
Errors Every outcome from concurrent branches, recovered, combined, and converted.
Scopes Nested scopes and forked children with every kind of resource.
Layers Resources provisioned in sequence and in parallel, with failures and interruption.
Streams Every stream operator, against the List model or the property it promises.
Interop Task, ValueTask, Async, and blocking calls, interrupted at random.