Why

Exa backs both web-search and web-fetch, so an exhausted credit balance takes every research path down at once , meeting prep company research, socials and e-commerce detection, brand/news lookups, outbound trigger research. It fails silently from the user's side: the agent just reports it found nothing.

That is exactly what happened in August. web-search/web-fetch went to zero successes from Aug 28 17:00 UTC and stayed there for three days (165 calls, 0 ok) before anyone noticed , and it was found while testing something unrelated, not by any alarm.

Three separate defects made a three-day silent outage possible.

What changed

1. Nothing was watching for it. NO_MORE_CREDITS on either Exa endpoint now raises one internal Slack alert. Rate-limited on the endpoint alone , credit exhaustion is an account-wide condition, so keying it per-user or per-call would have produced hundreds of messages. Fire-and-forget, so alerting can never turn a degraded lookup into a thrown request.

2. The web-search fallback had never once worked. It asks runCodeExecutorTool for bare JSON; the executor returns it wrapped in a ```json fence, as models reliably do; JSON.parse died on the leading backtick and threw away results it had already fetched. Confirmed in CloudWatch across the whole outage window:

[webSearchTool] Fallback also failed: SyntaxError: Unexpected token '`', "```json

Now strips the fence before parsing. The resilience path this tool was built around has been dead the entire time it has existed.

3. The failure was undiagnosable from the tool result. The catch block kept neither underlying error, recording only "Web search failed (both Exa and fallback)". That is why web-fetch surfaced NO_MORE_CREDITS and web-search told us nothing, and why diagnosing this needed CloudWatch for a question the tool result should have answered. Both causes now travel with the logged result as exaError and fallbackError.

Testing

• pnpm --filter @zero/server run types , clean
• Exa /contents and /search verified returning results post-top-up
• Fence-strip verified against the exact executor output shape from the CloudWatch logs above

Risk

Low. No behaviour change on the success path. The alert is additive and fire-and-forget; the fence strip only widens what already-fetched results get accepted; the error fields are additive on a result that was already a failure.

🤖 Generated with Claude Code

Greptile Summary

This PR adds Slack notification support for exhausted Exa credits, repairs parsing of JSON-fenced fallback search results, and records both primary and fallback errors.

• Detects NO_MORE_CREDITS responses from Exa search and contents endpoints.
• Adds a typed internal Slack notification with endpoint-based throttling.
• Strips JSON markdown fences before parsing fallback results.
• Adds underlying error messages to dual-failure tool results.

Confidence Score: 4/5

The normal fallback failure path must be fixed before merging because it still reports an empty success and drops the diagnostic causes this PR intends to preserve.

runCodeExecutorTool reports ordinary failures as returned values, but webSearchTool only constructs its new dual-error result when an exception is thrown; the new alert throttle also cannot suppress duplicate notifications across server processes.

Files Needing Attention: apps/server/src/mastra/tools/research/webSearchTool.ts; apps/server/src/services/notifications/internal/slack-notifier.ts

Important Files Changed