Data Streaming on AWS: Kinesis, Firehose, Flink, or MSK?

Some data can't wait. The suspicious transaction you need to block now, not tomorrow in a report. The user click that turns into a sharp recommendation if you read it on the spot, and into a missed chance if you read it later. For a long time we treated everything the same way: collect a pile of data, store it in a database, and process it in a batch later. But some things live in the now.

The picture that helps me explain this is a river. Notice that the water doesn't stop to be used. It flows by, and whoever is on the bank takes advantage of it along the way: one turns the mill wheel, another irrigates the field, another generates power. Nobody dams it all up first just to use it afterward. Now swap the water for data and you have data streaming: information arrives in a continuous flow and you react the instant it passes, instead of stacking it all up to process down the line.

That difference is what separates reacting from only finding out later. A bank that blocks fraud the second the transaction happens, an e-commerce that recommends on the click, a factory that adjusts the machine before it breaks: in all of them, processing later is almost not processing, because by the time the midnight batch runs, the moment is gone.

The trouble is that "streaming on AWS" gets confusing, because there are four services with similar names and even good people pick the wrong one. Come with me and I'll sort them out.

Freshness note: I wrote this guide in June 2026 and revised it in July 2026 (the math already includes On-demand Advantage and an important fix about ordering). Streaming services move fast: mode, quota, and price. I'll keep the post current, but if something doesn't match the screen in front of you, check the official Kinesis docs and tell me in the comments so I can fix it.

Before the service, the concept: stream or batch

Batch is collecting a pile of data and processing it every so often, like the report that runs overnight. Streaming is processing event by event, as it arrives. They don't compete: each solves a different kind of problem. Month-end accounting close is batch and it's just fine. A fraud alert is streaming, because one minute of delay is money lost. The mistake is using batch where the business needs to react on the spot.

The map: the four services (and what each one actually does)

Before diving into each one, two warnings that save you confusion. First, AWS renamed two services: Kinesis Data Firehose became Amazon Data Firehose (Feb 2024) and Kinesis Data Analytics became Amazon Managed Service for Apache Flink (Aug 2023). If you find a tutorial with the old name, it's the same service, just a new sign on the door. And Data Streams got a new mode, On-demand Advantage, which I'll explain in a moment.

Second, for those coming from Kafka: what's a topic there is a stream here in Kinesis; what's a partition there is a shard here. The map helps, but it isn't identical: Kinesis's PartitionKey runs through a hash that picks the shard, so several different keys can share the same shard. From here on I use each service's real name, but know that the two worlds mirror each other.

Data Streaming pipeline on AWS: producers send events to Kinesis Data Streams, which feeds a Lambda consumer in real time, Amazon Data Firehose delivering to S3, and Managed Service for Apache Flink processing to a dashboard; Amazon MSK appears as an alternative entry point.

Kinesis Data Streams: the river you can re-read

It's the heart of the story. The stream (the topic, remember?) is durable: producers write, consumers read, and the data is retained for a while (up to 365 days), so you can reprocess it. It's the "log" of the river. Use it when you need a durable stream, with replay, and you'll build the consumer (a Lambda, or an app with the Kinesis Client Library, the KCL, which handles shard distribution and checkpointing for you). Capacity comes split into shards (the partitions), each with a write and read ceiling, and you pick between three capacity modes:

Provisioned: you set the number of shards and pay for them, whether they're busy or not.
On-demand: AWS manages the shards for you and you pay for the throughput you use.
On-demand Advantage (the newest): brings warm throughput, with instant capacity for spikes, up to 10 GiB/s. One detail the sales page doesn't shout: it's an ACCOUNT billing mode, with a minimum usage commitment (25 MiB/s for at least 24h, charged even if you use less). In exchange, throughput comes out 60%+ cheaper and the per-stream-hour charge goes away. It makes sense from real volume up; for a small stream, stick with the first two modes.

Amazon Data Firehose: just deliver it to a destination

This one was renamed from Kinesis Data Firehose in 2024, same thing, new name. Here you don't write a consumer, it's the role Kafka Connect plays in the Kafka world (the sink connectors). You point at a source and a destination (S3, OpenSearch, Redshift, and others) and Firehose delivers, with buffering, optional transformation, and compression along the way. No replay, it's near-real-time delivery. Use it when the goal is to take the stream and drop it somewhere, with no consumption logic of your own.

Managed Service for Apache Flink: STATEFUL processing

This one was Kinesis Data Analytics, renamed in 2023. It runs managed Apache Flink for stateful processing, what Kafka Streams or ksqlDB do in the Kafka world: windows (summing per minute), joins between streams, and exactly-once on state recovery. When the question is "what's the moving average of the last 5 minutes per user," the answer lives here. The state, checkpoint, and recovery infrastructure is the service's responsibility; end-to-end semantics (what your sources and sinks guarantee) and state compatibility across versions remain your decisions.

Amazon MSK (and MSK Serverless): managed Kafka

If your world is already Kafka (ecosystem, Kafka API, portability across clouds, a team that knows it), MSK is AWS's managed Kafka. MSK Serverless provisions and scales capacity on its own and manages the topic's partitions, without you sizing a cluster. Use it when you need real Kafka, not a native equivalent.

When to use each

You want...	AWS service	Kafka equivalent
a durable, replayable stream with your own consumer	Kinesis Data Streams	Apache Kafka (Amazon MSK)
to just deliver the stream to a destination, no code	Amazon Data Firehose	Kafka Connect (sink)
stateful processing (window, join, aggregation)	Managed Service for Apache Flink	Kafka Streams / ksqlDB

In real life, a lot of architectures combine these services in three layers: ingest, delivery, and processing. Data Streams at the entrance, Firehose delivering a raw copy to S3 for history, and Flink processing in real time for a dashboard. And the entrance doesn't have to be Data Streams: you can have MSK up front and Flink after, because Managed Flink reads from both Kinesis and MSK.

Hands-on: producer and consumer on Kinesis Data Streams

The producer writes events to the stream. Look at the PartitionKey: it runs through a hash that decides which shard the record lands in. Same key, same shard, and the shard is the territory where order can exist.

import { KinesisClient, PutRecordsCommand } from "@aws-sdk/client-kinesis";

const kinesis = new KinesisClient({});

await kinesis.send(new PutRecordsCommand({
  StreamName: "customer-events",
  Records: [
    {
      Data: Buffer.from(JSON.stringify({ userId: "u-42", event: "click", ts: Date.now() })),
      PartitionKey: "u-42", // same key = same shard (the territory of order)
    },
  ],
}));

Now the gotcha almost everyone learns late, me included: PutRecords, the batch one, does not guarantee order. Not even with the same key. It processes each record individually, accepts partial success (half the batch goes in, half fails and you resend), and the docs are explicit: if you need to read in the order you wrote, the path is PutRecord in the singular, serial, chaining each write's SequenceNumberForOrdering into the next one:

import { KinesisClient, PutRecordCommand } from "@aws-sdk/client-kinesis";

const kinesis = new KinesisClient({});

const putEvent = async (payload, previousSequence) =>
  kinesis.send(new PutRecordCommand({
    StreamName: "customer-events",
    PartitionKey: payload.userId, // same entity, same shard
    Data: Buffer.from(JSON.stringify(payload)),
    // chain the previous write: this is what guarantees the sequence
    ...(previousSequence ? { SequenceNumberForOrdering: previousSequence } : {}),
  }));

const login = await putEvent({ userId: "u-42", event: "login" });
await putEvent({ userId: "u-42", event: "purchase" }, login.SequenceNumber);

Order lives inside the shard: the PartitionKey runs through the hash and picks the shard; events with the same key stay in sequence in the same shard; across shards there is no global order. Batch PutRecords does not guarantee order, strict order needs serial chained PutRecord.

With that, the consumer always sees "login" before "purchase." The price is honest: you traded batch throughput for the sequence guarantee. If it's thousands of events per second and strict order isn't a requirement, PutRecords is the right call. The guarantee you ask for changes the bill you pay; that's what architecture is.

On the other side, the serverless consumer is a Lambda with an event source mapping on the stream. And here's the second truth that separates blog examples from production code: delivery is at-least-once, the same event can arrive twice. So the handler needs to be idempotent and know how to fail per item, without taking down the whole batch:

export const handler = async (event) => {
  const batchItemFailures = [];

  for (const record of event.Records) {
    try {
      const payload = JSON.parse(Buffer.from(record.kinesis.data, "base64").toString("utf8"));

      // In production: check and write an eventId to an idempotent store
      // BEFORE the side effect. A duplicate processed twice = a customer charged twice.
      console.log(payload.userId, payload.event);
    } catch (error) {
      // fail ONLY this item; the rest of the batch continues
      batchItemFailures.push({ itemIdentifier: record.kinesis.sequenceNumber });
    }
  }

  return { batchItemFailures };
};

Returning partial failures only takes effect with ReportBatchItemFailures enabled on the event source mapping. And for real production I'd still add persistent idempotency, Logger and Metrics from AWS Lambda Powertools, a retry limit, a maximum record age, and an on-failure destination. A poison record can't stall the shard forever.

One detail to close: order is per shard, not across the whole stream. If order matters for an entity (a user, an order), the key is that entity's. Otherwise, events spread across shards and global order becomes an illusion.

The honest trade-offs

Ordering and scale: order is per shard, not global, and strict order needs serial PutRecord (the batch one doesn't guarantee it). The partition key choice isn't a detail, it's a design decision that defines how your application behaves AND how it scales. A skewed key throws too much traffic at a single shard (the hot shard) and you lose throughput even paying for several; and each partition key handles at most 1 MiB/s, no matter the warm throughput you configured. This alone is worth a whole post, and it will get one.
Duplicates and observability: delivery is at-least-once, so replay and retry require an idempotent consumer; reprocessing an event that charges the customer twice just trades one incident for another. And consumer lag needs an alarm: streaming without observability turns into accidental batch, the event arrives now and the consumer processes it half an hour later. The Well-Architected Framework names these bills: Reliability asks for idempotency and recovery, Operational Excellence asks for metrics and alarms, Cost Optimization asks for a capacity mode that follows real traffic, not the team's hopes.
Retention and replay: Data Streams retains and lets you reprocess; Firehose doesn't, it delivers and moves on.
Cost: provisioned pays for shards even when idle; on-demand pays a per-stream-hour rate plus usage. A stream on 24/7 with predictable traffic is sometimes cheaper provisioned. Cost is an architecture decision, as Werner Vogels hammers in the Frugal Architect, and it's the same talk as availability has a price.
Latency: streaming is low, but not zero. Firehose still has buffering (seconds to minutes), so don't count on it for instant reaction.

Kinesis or MSK?

If you have no commitment to Kafka, Kinesis is simpler and native, it pairs better with Lambda and the rest of serverless. If you already live on Kafka (connectors, tooling, multi-cloud, a trained team), MSK gives you Kafka without operating the cluster by hand. The choice is about context and ecosystem; "which is best" is the wrong question.

What you take away

Data streaming is processing in the flow, in real time, to react in the moment (fraud, recommendation, IoT). Batch is for when the delay doesn't hurt.
Kinesis Data Streams is the durable, replayable stream; Amazon Data Firehose delivers to a destination; Managed Service for Apache Flink does stateful processing; MSK is managed Kafka.
Order is per shard, so choose the partition key with intent. And STRICT order needs serial PutRecord with SequenceNumberForOrdering: batch PutRecords doesn't guarantee it.
Lambda with Kinesis is at-least-once: idempotency and per-item failure (batchItemFailures) are part of the design, not a luxury.
Provisioned or on-demand is a cost decision, and real architecture combines the services instead of picking just one.

Your turn

Do you use streaming in a project? Tell me which of these four made it into your design, and whether picking the wrong one has ever burned you. Drop that like, share it with someone who still processes everything in the midnight batch, and let's talk. Thanks a lot! BUILD. SCALE. REPEAT. =D