Why my alert triage workflow needed a CLI
I used AI to turn my notes into a draft and to edit the result. The experiences and ideas are my own.
I wanted to hand an agent an alert from Slack and have it help me figure out what was wrong.
We used SigNoz for observability. I set up a read-only API key in the environment so the agent could make authenticated requests to the SigNoz API while it investigated.
My first attempt was pretty straightforward: copy a Slack dump into a new agent session and let it run.
It worked! It was also very slow and error-prone.
The agent wrote a lot of Python. It made incorrect assumptions about request bodies and response shapes, then spent time working through the resulting errors. I wanted help investigating an alert, but a substantial part of the session went toward figuring out how to talk to the API.
python3 - <<'PY' >/tmp/signoz_trace_payload.json
import json,time
trace='<TRACE_ID>'
now=int(time.time()*1000); start=now-2*3600*1000
fields=['timestamp','trace_id','span_id','parent_span_id','name','duration_nano','service.name','db.system','db.statement','db.operation','url.full','url.path','http.request.method','http.response.status_code','error','exception.type','exception.message']
def field(name):
if name in ['timestamp','trace_id','span_id','parent_span_id','name','duration_nano']:
return {'name':name,'fieldContext':'span','fieldDataType':'string' if name not in ['duration_nano','timestamp'] else '', 'signal':'traces'}
if name=='service.name':
return {'name':name,'fieldContext':'resource','fieldDataType':'string','signal':'traces'}
return {'name':name,'fieldContext':'attribute','fieldDataType':'string','signal':'traces'}
payload={'schemaVersion':'v1','start':start,'end':now,'requestType':'raw','compositeQuery':{'queries':[{'type':'builder_query','spec':{'name':'A','signal':'traces','aggregations':[],'filter':{'expression':f"trace_id = '{trace}'"},'groupBy':[],'selectFields':[field(f) for f in fields],'having':{'expression':''},'limit':200,'order':[],'disabled':False,'stepInterval':60}}]},'variables':{}}
print(json.dumps(payload))
PY
curl -sS -m 30 -o /tmp/signoz_trace -X POST -H "SIGNOZ-API-KEY: $SIGNOZ_API_KEY" -H 'Content-Type: application/json' --data @/tmp/signoz_trace_payload.json "$SIGNOZ_BASE_URL/api/v5/query_range"; python3 - <<'PY'
import json
j=json.load(open('/tmp/signoz_trace'))
if j.get('status')!='success': print(json.dumps(j.get('error'),indent=2)); raise SystemExit
rows=[r['data'] for r in j['data']['data']['results'][0]['rows']]
for d in sorted(rows,key=lambda x:x.get('timestamp','')):
print(f"{d.get('timestamp')} {d.get('duration_nano',0)/1e9:8.3f}s svc={d.get('service.name')} name={d.get('name')} span={d.get('span_id')} parent={d.get('parent_span_id')} db={d.get('db.system')} status={d.get('http.response.status_code')} path={d.get('url.path')} err={d.get('error') or d.get('exception.type')}")
stmt=d.get('db.statement') or d.get('exception.message')
if stmt: print(' ',str(stmt)[:220].replace('\n',' '))
PY
So I went looking for an OpenAPI spec. Fortunately, SigNoz published one.
Verifiable evidence
SigNoz also had an MCP server, which I considered. I use Pi as my agent harness, and using the server in my setup would have required an adapter. That was one reason to look elsewhere, but I also wanted a way to verify the data behind the agent's conclusions.
If it found something interesting, I wanted a command I could run myself to see the same information directly from SigNoz. I didn't want to reconstruct an API request or dig through a pile of generated Python to check its work.
A CLI seemed like a good fit. The agent could use it from a shell, and I could copy a command into my own terminal.
I generated an Effect client from the OpenAPI spec and wrapped it in a CLI. That gave the agent an interface built around the published API instead of having it improvise requests throughout each investigation.
Then I made an agent skill for Pi to use during triage. The main instruction was simple: use the new CLI to investigate the root cause, and when making an assertion, include the commands that produced the evidence supporting it.
The reports started with a prose section describing the problem, what caused it, and the supporting evidence. They ended with a numbered list of references. I could read the explanation, copy a referenced command, and run it myself.
## Problem
An access alert reported more than 80,000 denied RPC requests in five
minutes, far above its threshold of 500.
## Root cause
The alert summed samples from a cumulative counter instead of measuring
how much the counter increased. About 84 denials occurred in the window,
not 80,000.
## Evidence
1. SigNoz described `access_denied_total` as a cumulative metric.[1]
2. Replaying the alert's query reproduced values above 80,000.[2]
3. Querying the counter's five-minute increase returned about 84.[3]
This rules out an 80,000-event denial spike.
## Uncertainty
The corrected count proves that the alert was a false positive. It does not
prove that every one of the real denials was benign.
## References
[1] `signoz metrics describe access_denied_total --from "2 hours"`
Result: metric temporality was `cumulative`.
[2] `signoz query run --file /tmp/access-alert-query.json`
Result: the alert's exact query returned values above 80,000.
[3] `signoz metrics promql 'sum(increase(access_denied_total[5m]))'`
Result: the latest five-minute increase was about 84.
Before I added that requirement, triage reports often mixed useful data from SigNoz with guesses about what it meant. In one report, the agent saw a slow login span and speculated that our database connection pool was full, leaving requests waiting for a connection. The real cause turned out to be Better Auth's Sentinel plugin doing proof of work. It was slow on purpose.
Once every assertion had to be explicitly justified, I noticed that the agent hallucinated much less, anecdotally at least. If it couldn't justify something, it didn't claim it.
That was really nice. I still had to decide whether the evidence supported the explanation, but I could actually inspect that evidence. The report gave me somewhere concrete to start.
Two remaining annoyances
Once I started using it on actual alerts, two problems became obvious.
First, you can reproduce the exact query, but that isn't especially helpful when the result is a massive JSON blob. I had made the data accessible without making it pleasant to read.
Second, the agent would spend 15 or more tool calls just getting oriented. It would run every help command, then query random services, traces and metrics to see what was available. Each new alert came with another round of setup.
I made three changes.
Output I could read
I added output formats, including human-readable tables, and updated the triage skill to request the human-readable format. Now, when I copied a command from the report, I got something I could actually read.
[
{
"name": "ci.job.command",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "ci.job.exit_status",
"fieldContext": "attribute",
"fieldDataType": "number"
},
{
"name": "ci.job.state",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "ci.job.wait_time_ms",
"fieldContext": "attribute",
"fieldDataType": "number"
}
]
name fieldContext fieldDataType
ci.job.command attribute string
ci.job.exit_status attribute number
ci.job.state attribute string
ci.job.wait_time_ms attribute number
It was a small change, but it made following the references much less tedious. The verification step needed to be convenient enough that I would actually do it.
Instructions in one place
I added signoz agent instructions. It brought together help for the commands and subcommands, along with guidance on when to use them.
Instead of making the agent walk through the CLI one help invocation at a time, I could give it the guide in one command.
# See what's firing right now, then assemble the rule's triage evidence
signoz alerts list --state firing
signoz alerts triage <rule_id> --from "1 hour"
signoz alerts evaluate <rule_id> --from "2 hours"
# What's slow in a service, then drill into the relevant spans and trace
signoz services operations api --from "1 hour" --limit 10
signoz traces search --service api --error --min-duration 500ms --from "1 hour" --limit 10
signoz traces list --filter 'resource.service.name = "api"' --order-by duration --limit 10
signoz traces get <trace_id>
Context from the actual instance
Knowing how to use the tool wasn't enough. The agent also needed to know what it could query in our SigNoz instance.
I added signoz agent context. It ran discovery queries in parallel and combined the results into a single overview of what was available. It also grouped related entries, including by prefix, to make the overview more compact.
# SigNoz instance overview
- Instance: https://example.signoz.cloud
- Window: 7 days → now
## Services
_764 services → 108 families_
- api — pr-* (184), dev, production (186)
- worker — pr-* (183), dev, production (185)
- web — pr-* (96), dev, production (98)
- + 59 other services
## Metrics
_132 logical metrics (from 500 series)_
- aws (116): aws_EC2_CPUUtilization · aws_EC2_NetworkIn · …
- http (8): http_request_duration · http_response_size · …
- runtime (8): process_cpu_time · process_memory_usage · …
## Trace fields
- resource (38): deployment.environment · service.name · service.version · …
- attribute (458): db.operation.name · http.request.method ·
http.request.header.* (99) · http.response.status_code · …
- span (23): duration_nano · has_error · name · trace_id · …
The two commands handled different parts of the setup: how to use the CLI, and what data was there to investigate.
Finally, the workflow I wanted
At that point, I was getting consistently useful root-cause reports in my own triage work. The agent spent less time figuring out its tools, and I got explanations with references I could follow and output I could read.
I could give it an alert, read the investigation, and check the commands behind its findings. That was what I'd wanted from the beginning. For about three weeks, it was glorious.
And then we switched to Grafana.