Skip to main content
This page covers running offline evaluations with the Axiom AI SDK CLI. The CLI provides commands for running offline evaluations locally or in CI/CD pipelines.
Online evaluations run inline in your app code and don’t use the CLI. For more information, see Online evaluations.

​
Run offline evaluations

The simplest way to run offline evaluations is to execute all of them in your project: 
axiom eval
You can also target specific evaluations by name, file path, or glob pattern: 
# By evaluation name
axiom eval spam-classification

# By file path
axiom eval src/evals/spam-classification.eval.ts

# By glob pattern
axiom eval "**/*spam*.eval.ts"
To see which evaluations are available without running them: 
axiom eval --list

​
Common options

For quick local testing without sending traces to Axiom, use debug mode: 
axiom eval --debug
To compare results against a previous evaluation, view both runs in the Axiom Console where you can analyze differences in scores, latency, and cost.

​
Run experiments with flags

Flags let you test different configurations without changing code. Override flag values directly in the command: 
# Single flag
axiom eval --flag.ticketClassification.model=gpt-4o

# Multiple flags
axiom eval \
  --flag.ticketClassification.model=gpt-4o \
  --flag.ticketClassification.temperature=0.3
For complex experiments, load flag overrides from a JSON file: 
axiom eval --flags-config=experiments/gpt4.json

​
Understand evaluation output

When you run an evaluation, the CLI shows progress, scores, and a link to view detailed results in the Axiom Console: 
✓ spam-classification (4/4 passed)
  ✓ Test case 1: spam detection
  ✓ Test case 2: legitimate question

Scorers:
  category-match: 100% (4/4)
  high-confidence: 75% (3/4)

Results:
  Total: 4 test cases
  Passed: 4 (100%)
  Duration: 3.2s
  Cost: $0.0024

View full report:
https://app.axiom.co/your-org/ai-engineering/evaluations?runId=ABC123
Click the link to view results in the Console, compare runs, and analyze performance.

​
What’s next?

To learn how to view and analyze evaluation results, see Analyze results.