SKILL.md8.6 KBView on GitHub ---
name: read-agent-traces
description: Query Cedar AI agent traces from Axiom to debug and understand how the AI made decisions. Use when asked why a deal/conversation was assigned a certain way, what the agent decided during initial sync, what an automation did, why a field was set, how a chat response was generated, or any question about AI decision-making for a specific event, deal, thread, or conversation. Covers all traced workflows: initial-sync Step Functions spans (setup, meetings, finalize, cleanup) and Mastra agent spans (event-execution, automation, chat).
---
# Read Agent Traces from Axiom
Use the `user-axiom` MCP server (`queryApl` tool, `query` parameter) to run APL queries against Cedar's trace data.
**Datasets:** `cedar-prod` (prod) · `cedar-staging` (staging) · `cedar-local` (local dev)
There are **two separate tracing systems** — use the right one for the task:
| System | What it covers | How to identify |
|---|---|---|
| **Step Functions spans** (manual OTel) | Every Step Functions step in an initial sync run: setup, meetings-prepare, meetings-batch, meeting-domain, finalize-conversation, cleanup | `name` starts with `conversation-sync.` |
| **Mastra agent spans** (OtelBridge) | LLM calls inside event-execution, chat, automation agents | `['attributes.gen_ai.agent.name']` is set |
Initial sync AOP selection and field-update agents are **not** in Axiom — they run inside the finalize-conversation Step Functions step but are not registered with the Mastra instance.
---
## MCP usage
```
tool: queryApl
parameter: query (string, APL)
```
---
## Field paths (confirmed from live schema)
### Step Functions spans (initial sync workflow steps)
All span attributes land under `attributes.custom`:
| What | APL field |
|---|---|
| Workflow run ID | `['attributes.custom']['sync.run_id']` |
| Step name | `name` (e.g. `conversation-sync.setup`) |
| Step input (sanitized JSON) | `events[0].attributes.input` |
| Step output (sanitized JSON) | `events[1].attributes.output` |
| Span name | `name` |
| Duration | `duration` (nanoseconds — divide by 1e6 for ms) |
| Trace ID (links all steps of one run) | `trace_id` |
### Mastra agent spans (event-execution / chat / automation)
| What | APL field |
|---|---|
| Workflow type | `['attributes.custom']['mastra.metadata.workflowType']` |
| Agent name | `['attributes.gen_ai.agent.name']` |
| Model used | `['attributes.gen_ai.response.model']` |
| Input tokens | `['attributes.gen_ai.usage.input_tokens']` |
| Output tokens | `['attributes.gen_ai.usage.output_tokens']` |
| Agent final output | `['attributes.gen_ai.output.messages']` (array) |
| Mastra run ID | `['attributes.custom']['mastra.metadata.runId']` |
| User ID | `['attributes.custom']['mastra.metadata.userId']` |
| Event source | `['attributes.custom']['mastra.metadata.eventSource']` |
---
## Schema discovery (run first if unsure of field paths)
```apl
['cedar-prod']
| where _time > ago(1h)
| limit 1
```
---
## Initial sync Step Functions workflow spans
These spans cover every step of the AWS Step Functions conversation-sync workflow. All steps for one run share a single `trace_id` (distributed trace propagation via KV). Use `cedar-staging` for staging runs.
### All steps for a workflow run
```apl
['cedar-staging']
| where _time > ago(2d)
| where name startswith 'conversation-sync.'
| where ['attributes.custom']['sync.run_id'] == 'WORKFLOW_RUN_ID'
| project _time, name,
input = tostring(events[0].attributes.input),
output = tostring(events[1].attributes.output),
durationMs = duration / 1e6
| order by _time asc
```
### Finalize-conversation failures for a run (see why phase 2 failed)
```apl
['cedar-staging']
| where _time > ago(2d)
| where name == 'conversation-sync.finalize-conversation'
| where ['attributes.custom']['sync.run_id'] == 'WORKFLOW_RUN_ID'
| project _time,
output = tostring(events[1].attributes.output),
durationMs = duration / 1e6
| order by _time asc
```
### View full trace tree by trace ID (use Axiom trace explorer)
```apl
['cedar-staging']
| where _time > ago(2d)
| where trace_id == 'TRACE_ID_FROM_SETUP_SPAN'
| project _time, name, parent_span_id, span_id, durationMs = duration / 1e6
| order by _time asc
```
To get the `trace_id` for a run, query the setup span first:
```apl
['cedar-staging']
| where _time > ago(2d)
| where name == 'conversation-sync.setup'
| where ['attributes.custom']['sync.run_id'] == 'WORKFLOW_RUN_ID'
| project _time, trace_id, span_id,
input = tostring(events[0].attributes.input),
output = tostring(events[1].attributes.output)
```
### Step names reference
| Step name | What it does |
|---|---|
| `conversation-sync.setup` | Validates connection, returns userId/connectionId (root span) |
| `conversation-sync.sync-meetings-prepare` | Batches meeting IDs for phase 1 |
| `conversation-sync.sync-meetings-batch` | Processes a batch of meetings through phase 1 |
| `conversation-sync.sync-meeting-domain` | Domain identification for one meeting (Map state) |
| `conversation-sync.finalize-conversation` | Phase 2 for one conversation: AOP selection + field updates (Map state) |
| `conversation-sync.sync-cleanup` | Post-sync cleanup |
---
## Mastra agent spans — workflow types and attributes
> Note: initial-sync AOP selection/field-update agents are **not** in Axiom. Only event-execution, chat, and automation agents appear here.
| `mastra.metadata.workflowType` | Key lookup attributes |
|---|---|
| `event-execution` | `mastra.metadata.userId`, `mastra.metadata.eventSource` |
| `automation` | `mastra.metadata.userId`, `mastra.metadata.eventSource` |
| `chat` | `mastra.metadata.userId` |
---
## Event-execution traces for a user
```apl
['cedar-prod']
| where _time > ago(24h)
| where ['attributes.custom']['mastra.metadata.workflowType'] == "event-execution"
| where ['attributes.custom']['mastra.metadata.userId'] == "USER_ID"
| project _time, name, duration,
agentName = ['attributes.gen_ai.agent.name'],
eventSource = ['attributes.custom']['mastra.metadata.eventSource'],
output = tostring(['attributes.gen_ai.output.messages'])
| order by _time desc
| limit 20
```
---
## Automation traces
```apl
['cedar-prod']
| where _time > ago(24h)
| where ['attributes.custom']['mastra.metadata.workflowType'] == "automation"
| where ['attributes.custom']['mastra.metadata.userId'] == "USER_ID"
| project _time, name, duration,
eventSource = ['attributes.custom']['mastra.metadata.eventSource'],
output = tostring(['attributes.gen_ai.output.messages'])
| order by _time desc
| limit 20
```
---
## Find all Mastra agent calls for a user (any workflow type)
```apl
['cedar-prod']
| where _time > ago(24h)
| where isnotnull(['attributes.gen_ai.response.model'])
| where ['attributes.gen_ai.agent.name'] != ""
| project _time, name,
agentName = ['attributes.gen_ai.agent.name'],
workflowType = ['attributes.custom']['mastra.metadata.workflowType'],
runId = ['attributes.custom']['mastra.metadata.runId'],
inputTokens = ['attributes.gen_ai.usage.input_tokens'],
outputTokens = ['attributes.gen_ai.usage.output_tokens'],
duration
| order by _time desc
| limit 20
```
---
## How the tracing works
### Step Functions spans (initial sync)
Manual OTel spans created in `apps/server/src/runtime/worker-entry.ts` via `initTracing().startActiveSpan()`. The setup step saves its span context to KV (`initial_sync_state`, key `trace-context:{workflowRunId}`). All subsequent steps load that context and attach as child spans — forming one linked trace tree per run. Span attributes go under `attributes.custom` in Axiom. Input/output captured as span events.
### Mastra agent spans
Custom attributes are set via `RequestContext` keys before each `agent.generate()` call. Mastra's `requestContextKeys` config in `mastra/index.ts` extracts them and attaches them to every span. They land under `attributes.custom` with a `mastra.metadata.*` prefix:
```
mastra.metadata.workflowType → 'event-execution' | 'automation' | 'chat'
mastra.metadata.userId
mastra.metadata.runId
mastra.metadata.eventSource
```
---
## Notes
- `duration` is in nanoseconds — divide by 1e6 for milliseconds.
- For Step Functions spans, `trace_id` links all steps of one workflow run (set up via KV trace context propagation).
- For Mastra spans, `trace_id` groups one `agent.generate()` call (LLM call + tool calls).
- Initial sync AOP selection / field-update agents run inside `finalize-conversation` but are not registered with the Mastra instance — no Axiom spans exist for them. Use CloudWatch logs (`initial_sync_phase2_*` structured log entries) to debug those.
- `scope.name == "@mastra/otel-bridge"` identifies Mastra-generated spans; Step Functions spans have `scope.name == "cedar-api"`.