🇧🇷 Leia em português: Serverless não é à prova de falha: 7 padrões de resiliência com código
Let me start with the part that hurts. During the October 2025 outage in us-east-1, a lot of 100% serverless applications went down. Lambda, DynamoDB, SQS, the whole package. What took them down was a very common assumption: that using managed services is the same as being immune to failure.
Serverless takes the OS, the patching and the capacity planning off your plate. That shrinks your operational surface. Your risk surface is still there, it just moved: it now lives in how your code reacts when a dependency chokes. If the code wasn't designed for that, the failure shows up exactly where you had no plan.
This post is hands-on: seven serverless resilience patterns on AWS, each one with code you can paste and a “when to do it, when not to”. The examples are TypeScript with AWS SDK v3 and Powertools for AWS Lambda. It's part of the Architecture that Survives series: in the pillar post on the price of high availability I covered the strategic decision; here we go down to the code.
Why serverless went down in the October 2025 outage
The details are in the post-event summary AWS published. The trigger was a latent race condition in the system that manages DynamoDB's DNS, which left the record for the regional endpoint dynamodb.us-east-1.amazonaws.com empty. No DNS resolution, so everything that depended on that endpoint could no longer open a connection. Then came the cascade: Lambda couldn't create or update functions and fell behind on processing SQS and Kinesis, STS saw errors and latency, and even console sign-in outside us-east-1 failed for people using root credentials.
Two lessons I take from that summary.
Multi-AZ wouldn't have saved you. What failed was a regional endpoint. Multi-AZ protects you from losing one Availability Zone; when a service stumbles across the whole region, every AZ feels it at once.
Backlogs and unbounded retries keep a system on the floor after the root cause is gone. The summary shows this inside EC2 itself: when DynamoDB's DNS came back, an internal subsystem (DWFM) had so much queued-up work that it went into congestive collapse and couldn't make progress. Engineers had to throttle and restart hosts to get it moving again. If it happens to an AWS system, it can happen to yours.
Let's get to the patterns.
1. Retry with exponential backoff and jitter
Retries are necessary. Aggressive retries shoot you in the foot: if a thousand executions retry at the same moment, you create a thundering herd that keeps the dependency down even after it recovers. In the Well-Architected Reliability pillar, this is REL05-BP03, “Control and limit retry calls”.
First thing: the SDK already retries with backoff and jitter. Don't turn it off. Make what you want explicit:
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
const ddb = new DynamoDBClient({
maxAttempts: 3, // 1 attempt + up to 2 retries; don't overdo it
retryMode: "standard", // the mode AWS recommends as the default
});There's also retryMode: "adaptive", which adds a client-side rate limiter. The SDK retry behavior doc points it at one specific case (a client hitting a single resource, like one table, with frequent throttling) and warns that it can delay even the first request. For a client shared across several tables or tenants, stay on standard.
For retries in your own logic (an HTTP call to an external service, for example), use exponential backoff with full jitter, which spreads the attempts over time. Without jitter, everybody retries in the same millisecond:
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
// retry only what is transient: timeouts, throttling (429) and 5xx
const isRetryable = (err: unknown) => {
const e = err as { name?: string; status?: number };
return e.name === "TimeoutError" || e.status === 429 || (e.status ?? 0) >= 500;
};
async function withRetry<T>(
fn: () => Promise<T>,
{ retries = 3, baseMs = 100, capMs = 2000 } = {}
): Promise<T> {
for (let attempt = 0; ; attempt++) {
try {
return await fn();
} catch (err) {
if (attempt >= retries || !isRetryable(err)) throw err;
const expo = Math.min(capMs, baseMs * 2 ** attempt);
const delay = Math.random() * expo; // full jitter
await sleep(delay);
}
}
}The fuller explanation of this design is in Timeouts, retries, and backoff with jitter, from the Amazon Builders' Library.
When to do it: calls to dependencies that can throttle or time out.
When NOT to: don't stack your retry on top of the SDK's (that's 3 × 3 = 9 attempts, and it blows past your Lambda timeout). And remember a Lambda invocation is short-lived: retrying for 5 minutes inside a function with a 30s timeout gets you nowhere. Once you hit the limit, push it to a queue and reprocess later (pattern 4).
2. Idempotency (so you can retry without fear)
A retry is only safe if the operation is idempotent: running it twice gives the same result as running it once. Without that, a retry becomes a double charge, a duplicate email, a broken inventory count. And duplicates will happen: SQS standard queues deliver at-least-once, and the Lambda docs themselves warn that with asynchronous invocation (which is how EventBridge calls your function) the same event can arrive more than once. In Well-Architected terms, this is REL04-BP04, “Make mutating operations idempotent”.
The cleanest way is the Powertools idempotency utility, which stores the result of the first execution in a DynamoDB table and returns that stored result on repeats:
import { makeIdempotent, IdempotencyConfig } from "@aws-lambda-powertools/idempotency";
import { DynamoDBPersistenceLayer } from "@aws-lambda-powertools/idempotency/dynamodb";
// table with partition key "id" and TTL on the "expiration" attribute
const persistenceStore = new DynamoDBPersistenceLayer({ tableName: "idempotency" });
const config = new IdempotencyConfig({
eventKeyJmesPath: "orderId", // the idempotency key comes from orderId only
expiresAfterSeconds: 3600, // the default is also 1 hour
});
export const handler = makeIdempotent(
async (event: { orderId: string }) => {
// runs once per orderId; repeats get the stored result back
return await chargeOrder(event.orderId);
},
{ persistenceStore, config }
);Look at eventKeyJmesPath. Without it, Powertools uses the whole event as the key: if the second delivery comes with one field changed (a timestamp, a trace id), it goes through as if it were new. The key has to be the business identifier.
Without Powertools, you can do it by hand with a DynamoDB conditional write (only the first one gets through):
import { PutItemCommand, ConditionalCheckFailedException } from "@aws-sdk/client-dynamodb";
try {
await ddb.send(new PutItemCommand({
TableName: "orders",
Item: { orderId: { S: orderId }, status: { S: "charged" } },
ConditionExpression: "attribute_not_exists(orderId)", // 2nd time fails on purpose
}));
} catch (err) {
if (!(err instanceof ConditionalCheckFailedException)) throw err;
// already processed: move on without charging again
}When to do it: every operation with a side effect that can be redelivered (payments, notifications, inventory updates).
3. Short timeouts and Circuit Breaker
When a dependency is slow or down, the worst thing you can do is hang there waiting. You hold the execution, you hit the Lambda timeout, and one service's failure becomes everyone's failure.
A short, explicit timeout on every external call is the bare minimum (REL05-BP05, “Set client timeouts”). In SDK v3, with a detail from recent versions:
import { NodeHttpHandler } from "@smithy/node-http-handler";
const ddb = new DynamoDBClient({
requestHandler: new NodeHttpHandler({
connectionTimeout: 1000, // 1s to open the connection
requestTimeout: 2000, // 2s for the response
throwOnRequestTimeout: true, // without this, exceeding requestTimeout only logs a warning
}),
});Both timeouts default to 0, which disables the limit. And in current versions of @smithy/node-http-handler, going past requestTimeout only writes a warning to the log unless you turn on throwOnRequestTimeout. Without that flag, the “2s timeout” in your code doesn't interrupt anything.
The next step is the Circuit Breaker: when the error rate crosses a threshold, you “open the circuit” and stop calling the dependency for a while, answering fast with a fallback. With the opossum library it looks like this:
import CircuitBreaker from "opossum";
const breaker = new CircuitBreaker(callRecommendations, {
timeout: 2000, // fails if it takes longer than this
errorThresholdPercentage: 50, // 50% errors in the window opens the circuit
resetTimeout: 10000, // after 10s, lets one test call through
});
breaker.fallback(() => []); // no recommendations, return an empty listOne architecture caveat: in Lambda, the breaker state lives in the memory of each execution environment. A hundred concurrent instances are a hundred independent circuits, each one learning on its own that the dependency is down. To keep your Lambda from hanging, that's enough. If the goal is to protect the dependency from load, combine it with a concurrency limit or put a queue in front.
When to do it: a dependency that can slow down or fail and isn't critical. Opening the circuit stops you from holding the Lambda and spreading the slowness to everything else.
4. Dead-letter queue, partial batch response and reprocessing
Not every message will process on the first try. Instead of losing it or looping forever, send what keeps failing to a DLQ and reprocess it later, calmly.
Before the DLQ, there's a setting that saves you rework: partial batch response. By default, if your Lambda fails on an SQS batch, the whole batch goes back to the queue, including the messages that had already succeeded. With ReportBatchItemFailures, the function returns only the ids that failed. The Powertools batch utility does that bookkeeping for you:
import { BatchProcessor, EventType, processPartialResponse } from "@aws-lambda-powertools/batch";
import type { SQSHandler, SQSRecord } from "aws-lambda";
const processor = new BatchProcessor(EventType.SQS);
const recordHandler = async (record: SQSRecord): Promise<void> => {
const order = JSON.parse(record.body);
await processOrder(order); // if this throws, only THIS message goes back to the queue
};
export const handler: SQSHandler = async (event, context) =>
processPartialResponse(event, recordHandler, processor, { context });If every message in the batch fails, Powertools throws and the whole batch goes back, same as without the feature. And partial batch response only works with the flag turned on in the event source mapping. In the Serverless Framework, queue, DLQ and flag look like this:
functions:
processOrders:
handler: src/orders.handler
timeout: 30
events:
- sqs:
arn: !GetAtt OrdersQueue.Arn
batchSize: 10
functionResponseType: ReportBatchItemFailures
resources:
Resources:
OrdersQueue:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 180 # at least 6x the function timeout
RedrivePolicy:
deadLetterTargetArn: !GetAtt OrdersDLQ.Arn
maxReceiveCount: 5 # 5 receives and it goes to the DLQ
OrdersDLQ:
Type: AWS::SQS::Queue
Properties:
MessageRetentionPeriod: 1209600 # 14 days, longer than the source queueThose numbers come from the Lambda with SQS doc, which recommends a visibility timeout of at least six times the function timeout and a maxReceiveCount of at least 5. The longer retention on the DLQ is there because, in a standard queue, a message's expiration counts from when it entered the original queue; with the same retention, it can expire right after landing in the DLQ.
When Lambda is invoked asynchronously (EventBridge, SNS, S3), the asynchronous invocation config is what catches the failure. By default, Lambda retries twice more and then discards the event. To lose nothing, configure an on-failure destination:
functions:
onOrderCreated:
handler: src/notify.handler
maximumRetryAttempts: 2 # 0 to 2
maximumEventAge: 3600 # 60s to 6h
destinations:
onFailure:
type: sqs
arn: !GetAtt NotifyFailuresQueue.ArnA destination has an edge over the function's classic DLQ: besides the event, it sends the response details (the error) in the record. Remember to give the function's role permission to send to the destination.
Above all, have a plan for the DLQ. A CloudWatch alarm on the DLQ's ApproximateNumberOfMessagesVisible to warn you, and a reprocessing path. SQS has native redrive from the DLQ back to the source queue, through the console or the StartMessageMoveTask API. A dead-letter queue nobody looks at turns into a data graveyard.
If your source is a Kinesis stream instead of a queue, the conversation changes: ordering per shard is preserved and, by default, Lambda retries the failing batch until it succeeds or the records expire, holding the shard in the meantime. I went through how to choose between these options in Data Streaming on AWS: Kinesis, Firehose, Flink, or MSK?.
5. Graceful degradation instead of a blank screen
Not everything has to go down together. If recommendations are out, show the catalog without recommendations. A degraded response almost always beats a 500. In practice, you split the critical from the nice-to-have and give the nice-to-have a fallback, which is exactly Well-Architected REL05-BP01: turn hard dependencies into soft ones.
async function getProductPage(id: string) {
const product = await getProduct(id); // critical: if this fails, the page fails
let recommendations: Product[] = [];
try {
// with a fallback configured, the breaker already returns [] when the circuit opens
recommendations = await breaker.fire(id);
} catch {
recommendations = []; // safety net: degrade without taking the page down
}
return { product, recommendations };
}Deciding what's critical and what's nice-to-have is selective resilience, and I spent a section of the series pillar on it. Users would rather have a simpler site than a site that's down.
6. Know regional vs zonal (and where Multi-AZ stops)
Multi-AZ handles losing one AZ. The catch is that most managed service endpoints are regional, and DynamoDB in October 2025 showed what that means: the service stumbled at the region level, and being spread across three AZs changed nothing.
This pattern comes without a snippet. It's a question to answer honestly: which of your components are a regional SPOF? If the business needs to survive losing an entire region, that's where multi-region comes in (DynamoDB Global Tables, route failover, and so on), with the cost and complexity I break down in the pillar. And look at what the summary itself recorded: teams with Global Tables could read and write to replicas in other regions, but saw replication with us-east-1 lag significantly. Multi-region has fine print too, so it goes where your RTO/RPO justifies it.
If you're tempted to jump straight to a second cloud, run the numbers first: in Is multi-cloud more resilient? The math says no I show why, in most cases, multi-region on the same cloud solves the real problem with a fraction of the complexity.
7. Don't tie your critical control plane to us-east-1
Even teams running in other regions felt October 2025. The summary says console sign-in with root credentials failed in regions outside us-east-1, and that Redshift in every region couldn't use IAM user credentials to run queries, because of a dependency on the IAM API in us-east-1.
Part of that is out of your control. STS is in your control. In the AWS SDK for JavaScript v3, STS always uses a regional endpoint, with one catch: if no region is configured, it falls back to the us-east-1 regional endpoint. Inside Lambda, the AWS_REGION variable already takes care of it. In a container, a script or a CI pipeline, make it explicit. Here I'm using sa-east-1 (São Paulo), where I'm based; swap in yours:
import { STSClient } from "@aws-sdk/client-sts";
// SDK v3: always regional; with no region, falls back to us-east-1
const sts = new STSClient({ region: "sa-east-1" });The AWS SDK for JavaScript v2 and AWS CLI v1 still default to legacy, which sends calls to the global endpoint sts.amazonaws.com. Since April 2025, AWS serves global endpoint requests locally in regions that are enabled by default, but it still recommends the regional endpoint whenever possible. On legacy config, force it:
# AWS SDK for JavaScript v2 / AWS CLI v1 / legacy config
export AWS_STS_REGIONAL_ENDPOINTS=regionalThe full table of defaults per SDK is in the STS regionalized endpoints doc. While you're at it, map which of your dependencies go through us-east-1 without you asking for it: sign-in, federation, pipelines, DNS. The dependency that isn't on the map is the one that doesn't make it into the contingency plan.
What you take away
Managed services give you a lot of resilience for free. But the resilience that matters, the kind that keeps you standing when AWS stumbles, doesn't come from serverless. It comes from the architecture you design on top of it. As I put it in the pillar: there's no such thing as a failure-proof architecture, but there is such a thing as a surprise-proof organization. In code, that becomes lines:
- Retry with backoff and full jitter, with a limit, and without stacking your retry on the SDK's.
- Idempotency with the right key (
eventKeyJmesPath), because the duplicate is coming. - Explicit timeouts that actually throw (
throwOnRequestTimeout), and a Circuit Breaker on the dependency that isn't critical. - Partial batch response on SQS, a DLQ with an alarm and redrive, and an on-failure destination for async invocation.
- Graceful degradation for the nice-to-have, and an honest map of your regional SPOFs and the dependencies hiding in us-east-1.
Tell me in the comments which of these patterns has saved you (or burned you for not having it). And if you use Powertools, which utilities made it into your toolkit? If this post helped, drop a like and share it around. Thanks a lot!
BUILD. SCALE. REPEAT. =D
