Skip to content
Chinonso.Ani
All articles

Jun 03, 2026 ·

Serverless AI Alerting Architecture

The hard part of serverless AI alerting is not calling a model. The hard part is capturing the event, preserving state, notifying people in real time, and keeping the model stage replaceable.

Serverless AI Alerting Architecture

AI alerting systems are easy to overdescribe. A diagram puts a model in the center and skips the work that decides recovery.

This use case covers a serverless alerting scaffold built around AWS events, state, and live fan-out. Incident submission stored state in DynamoDB and published a custom EventBridge event. WebSocket handlers maintained connection state and pushed updates. A later agent stage consumed event-driven work, with Bedrock configuration available for model inference. The important point is scope: this was an event spine with AI-processing scaffolding, not a finished autonomous production system.

Project context

The system was designed for incident-style alerts. A user submits an incident, receives an incident ID, and expects progress updates while downstream processing runs. The architecture needed to preserve state, route work, support real-time updates, and allow model-assisted classification.

The constraint was reliability before intelligence. Calling a model is not difficult compared with preserving the incident, routing the event, updating status, and making failures visible. If those parts are weak, adding a model provider makes the system harder to reason about.

The useful claim is narrow. The implementation demonstrated durable intake, custom event publishing, WebSocket fan-out, and a replaceable model stage boundary. It did not prove a complete autonomous alerting agent, finished tool contracts, production infrastructure, or hardened retries.

Start with durable intake

The incident submission path did two things before any model work mattered. It wrote an incident record to DynamoDB, then published an EventBridge event with a named detail type. The caller received an incident ID and initial status.

That order was the architecture. The system captured the event as state before downstream work began. If processing failed, the incident still existed.

AWS documents PutEvents as the EventBridge API for sending custom events that rules can match. DynamoDB's PutItem creates or replaces an item by primary key. Together, those contracts support a simple line: persist the incident, publish the event, then process asynchronously.

There was an important caveat. The visible write path did not prove append-only storage or duplicate protection. A basic PutItem call can replace an item unless the design uses a conditional expression. The safe claim is that incident state was persisted before event-driven processing, not that the incident store was immutable.

Separate live updates from source of truth

The WebSocket layer owned live feedback. Connection handling stored connection IDs when viewers connected and removed them when they disconnected. The broadcast path looked up subscribers for an incident and sent JSON updates through API Gateway's management API.

That separation kept intake clean. Submitting an incident did not require knowing how many viewers were watching. Processing did not require holding open browser connections. Live delivery became a fan-out concern.

AWS's WebSocket API model supports two-way communication and backend callbacks to connected viewers. Boto3's post_to_connection sends data to a specific connection. The local pattern matched that model: post each update and remove stale connections when the platform reports that a viewer is gone.

The caveat is operational depth. Handling a gone connection is necessary, but it is not full production hardening. A production alerting system would need structured logs, metrics, retry rules, dead-letter behavior, and visibility into failed broadcasts. Missed live updates must be recoverable from persisted incident state.

Treat the model as a stage

The model belonged after the event spine, not inside the intake path. The architecture made room for Bedrock configuration, model selection, regional settings, and notification policy. The agent-shaped processing stage accepted EventBridge-shaped work and wrote state. The autonomous loop and tool behavior were still scaffolds.

That distinction prevents overclaiming. Amazon Bedrock's InvokeModel API is the runtime call for model inference, and it can fit naturally into this flow: receive an event, load incident state, call a model, write a classification, and broadcast status. The model should not own delivery guarantees.

A useful high-level flow looks like this:

  1. The user submits an incident.
  2. DynamoDB stores the incident and initial status.
  3. EventBridge routes the submitted-incident event.
  4. A processor loads the incident and relevant context.
  5. Bedrock or another model produces a bounded classification or brief.
  6. The result is written back to DynamoDB.
  7. WebSocket subscribers receive a status update.
  8. Notification tools send only policy-approved alerts.

Only part of that flow was implemented. The portfolio lesson is the boundary: once the incident ID, state table, event bus, processor, and fan-out path exist, the model stage can be replaced without redesigning the whole system.

Tool contracts before autonomy

The tool layer named future capabilities such as retrieving historical context, analyzing patterns, calculating risk, generating an action brief, routing, sending approved notifications, and logging decisions. Those functions were not complete, but they showed where contracts should attach.

Each tool needs a small, testable contract. A context tool should distinguish "no context found" from "context lookup failed." A routing tool should separate recommendation from dispatch. A notification tool should enforce required fields, authorization, destination policy, and logging.

This matters because model-assisted systems become risky when tools are vague side effects. An agent should not get a broad "send alert" capability without policy checks and audit output. It should call a constrained tool that reports whether a message was sent, skipped, rejected, or failed.

The right implementation order is conservative:

  1. Load incident context by incident ID.
  2. Write processing phase state back to DynamoDB.
  3. Broadcast phase changes through the existing fan-out path.
  4. Add model invocation with narrow prompts and structured output.
  5. Add notification tools with policy checks and audit logging.
  6. Add retry, dead-letter, and infrastructure behavior after runtime contracts are stable.

That order makes the system observable before it becomes more capable.

Infrastructure as the contract

The infrastructure layer was still a placeholder, but it named the resources the runtime needed: DynamoDB tables, Lambda functions, EventBridge rules, IAM permissions, and timing configuration. It showed deployment shape without pretending the stack was production-ready.

A deployable version would need the incident table, connection indexes, Lambda functions for submission, processing, connection handling, and broadcasting, EventBridge rules that match the emitted event shape, API Gateway WebSocket routes, scoped IAM policies, and environment variables for table names, platform endpoints, region, model settings, and notification destinations.

This is where serverless reliability lives. A model call cannot compensate for a missing EventBridge rule. A WebSocket broadcaster cannot post without the right endpoint and permissions. If infrastructure names, event detail types, table keys, and environment variables drift from runtime expectations, the architecture becomes a diagram instead of a system.

Safeguards and tradeoffs

The main safeguard was staging. Durable state came first. Event routing came second. Live fan-out was separate. The model remained replaceable. Notification tools were treated as policy-controlled operations rather than automatic side effects.

The main tradeoff was that the scaffold still needed hardening. Conditional writes would be needed where duplicate submission or replay is harmful. Status transitions should include timestamps and error state. EventBridge rules should be tested against the exact emitted event shape. Failed processors should have retry and dead-letter behavior. Model outputs should be stored with version and timestamp fields.

Those additions are not decorative. They are the difference between a demo that works when every service behaves and an alerting system operators can recover under failure.

What changed

The architecture made the model less central and the system more legible. Incident intake had a durable record. EventBridge gave downstream work a stable route. DynamoDB held recoverable state. WebSockets handled live updates without becoming the source of truth. Bedrock could be a processing stage instead of the architecture.

The evidence supports that system shape. It does not support claims that the autonomous agent, tool layer, infrastructure deployment, notification policies, or production recovery paths were complete.

Where this pattern applies

This pattern applies to incident response tools, compliance monitors, support escalation systems, operations dashboards, and any workflow where a model may summarize or classify events but should not own durability. It is especially useful when teams want real-time feedback and AI assistance without losing a clear recovery path.

For similar serverless AI alerting work, draw the model box last. First define the incident ID, state store, event bus, processor, live fan-out, and failure handling. Once those pieces are real, the model has a safe place to contribute.

Sources

Share
Email copied to clipboard