Azure Messaging and Events: Service Bus, Event Hubs, Event Grid

The first decision when designing a distributed application is how its components will talk to each other. Defining that communication strategy is what points me at the right Azure service, because most components communicate in one of two ways: by sending messages or by publishing events. The distinction sounds pedantic until you pick the wrong one and end up rebuilding the integration layer six months later — a message carries data that someone is contracted to process, an event announces that something happened and makes no promises about who cares. This note walks through both models and the four Azure services that implement them: Azure Queue Storage and Azure Service Bus for messages, Azure Event Hubs and Azure Event Grid for events.

Messages versus events

Messages contain raw data produced by one component and consumed by another. The key property is that a message contains the data itself, not just a reference to that data, and a contract exists between publisher and consumer. The publisher sends raw data and expects the consumer to do something specific with it, perhaps create a file from that data and report back when the work is done.

Events are lighter weight and are most often used for broadcast communication. An event-driven architecture consists of event producers that generate a stream of events, event consumers that listen for them, and event channels that transfer events from producers to consumers. Receiving components generally decide which communications they're interested in and subscribe to those events. An intermediary — Event Grid or Event Hubs — manages the subscription process and routes each published event to any interested parties. This is the publish-subscribe pattern, and it's the most common shape an event-driven design takes on Azure.

Events have the following characteristics:

That last row is the one worth holding on to, because it's the fork in the road between the two event services. Microsoft's own comparison draws the same line: discrete events report an actionable state change and suit reactive. Events that form a time-ordered series report a condition that's analysable, and belong in a stream. Discrete events go to Event Grid; series events go to Event Hubs.

Message-based services

Azure offers two message-based solutions, Azure Queue Storage and Azure Service Bus. Queue Storage stores large numbers of messages in Azure Storage. Service Bus is a message broker that decouples applications and services.

Azure Queue Storage

Azure Queue Storage is a service that uses Azure Storage to store large numbers of messages. A queue can contain millions of messages, and the number and size of queues is limited only by the capacity of the storage account that owns the Queue Storage. The documented ceiling is 5 PiB, which is the single storage account capacity limit rather than a queue-specific one. Messages can be securely accessed from anywhere in the world through a simple REST-based interface over HTTP or HTTPS.

The individual limits matter more than the headline capacity. A queue message can be up to 64 KB, dropping to 48 KB of user payload if the content isn't XML-safe and has to be Base64-encoded. Where a message needs to carry more than that, the standard workaround is to put the payload in a blob and enqueue a reference to it — combining queues and blobs lets you effectively enqueue up to 200 GB for a single item, at the cost of the queue no longer being self-contained. Message time-to-live can be set to infinite, and delivery is at-least-once.

Queues generally provide increased reliability, guaranteed message delivery, and transactional support, with one important qualifier on ordering. Messages in Queue Storage are typically first-in, first-out, but they can come out of order: if a client crashes mid-processing and the message's visibility timeout expires, the message becomes visible again and gets dequeued a second time, out of its original position. Queue Storage offers no ordering guarantee, only a tendency.

Azure Service Bus

Azure Service Bus is a fully managed enterprise message broker, used to decouple applications and services from each other. It supports message queues and publish-subscribe topics, lets you load-balance work across competing workers, safely routes and transfers data and control across service and application boundaries, and coordinates transactional work that requires a high degree of reliability.

It also supports First-In, First-Out processing through the use of message sessions. When sessions are enabled, messages sharing the same SessionId are grouped and processed in order of arrival, ensuring strict FIFO behaviour within each session. This is the recommended and most reliable way to enforce ordered message handling in Azure. Architecturally, it isn't free: a session is handed to exactly one consuming node at a time, so ordering is bought by serialising everything inside that session. Design the session key to be as narrow as the ordering requirement genuinely is — per customer, per order, per device — rather than per tenant, or you've turned a parallel consumer pool into a single-threaded one.

Receiving from Service Bus is consumer-initiated, but it isn't the busy polling loop the phrase "with polling" suggests. Service Bus supports a long-polling receive operation over its TCP-based protocols, and the .NET, Java, JavaScript, and Go SDKs all expose a push-style API where you register a message handler and the SDK does the receiving for you.

Two receive modes decide the delivery guarantee:

Peek lock is the default and the right choice almost always, but at-least-once means redelivery is a normal event rather than an exception. Handlers need to be idempotent, or you need Service Bus's duplicate detection, which removes duplicates based on the message ID property.

Queues versus topics and subscriptions

Service Bus message queues are a message broker system built on a dedicated messaging infrastructure. Like Azure queues, Service Bus holds messages until the target is ready to receive them. The choice within Service Bus is between two entity types:

A subscription resembles a virtual queue that receives copies of the messages sent to the topic, and consumers read from it exactly as they would from a queue — so subscriptions inherit the same competing-consumer, temporal-decoupling, and load-levelling behaviour. Subscriptions can also carry filters: by default a subscription receives everything sent to the topic, but a SQL filter expression over system or custom message properties narrows that to a subset, with optional actions that annotate the selected messages.

Competing consumers are also where messaging meets autoscaling. KEDA scales a workload on queue depth rather than CPU, which is the mechanism that lets an event-driven consumer scale to zero. I covered that in Scaling and Extending AKS: HPA, KEDA, Dapr, and Istio.

Choosing between Queue Storage and Service Bus

The two overlap enough that the decision is usually made on one or two hard constraints rather than on general preference:

Microsoft's own selection guidance reduces to a few decisive questions. Choose Queue Storage when the queue must hold more than 80 GB of messages, when you want to track processing progress inside the message so another worker can resume where a crashed one left off, or when you need server-side logs of every transaction executed against the queue. Choose Service Bus when you need guaranteed FIFO, duplicate detection, transactional send or receive, dead-lettering, an RBAC model that separates sender and receiver rights, messages above 64 KB, or an eventual migration from point-to-point queuing to publish-subscribe.

The absence of dead-lettering on Queue Storage is the gap that catches people out. A poison message that fails repeatedly stays in the queue and keeps being redelivered unless the application checks the DequeueCount property on dequeue and moves the message to an application-defined dead-letter queue itself. Service Bus does that automatically.

Event-based services

Azure Event Hubs

Certain applications produce a massive number of events from almost as many sources — the scenarios usually filed under Big Data, which can require extensive infrastructure. Azure Event Hubs is a fully managed, real-time data streaming platform and event ingestion service that can receive and process millions of events per second with low latency. Data sent to an event hub can be transformed in real time and stored for later analysis, and Event Hubs supports real-time data ingestion and micro-batching on the same stream.

The mechanics that make it a stream rather than a queue:

• Events received are added to the end of the hub's data stream, which is an append-only distributed log ordered by the time the event is received.
• The hub is divided into partitions, each an independently consumable, ordered sequence of events. Partitions are the unit of parallelism: more partitions, more throughput.
• Consumers read by tracking their own position (offset) in each partition, and can seek along the stream by time offset. A consumer group is a logical view of the hub that lets multiple consuming applications read the same stream independently, each with its own position.
• Event Hubs uses a pull mechanism for event streaming. This is what differentiates it from a broker like Service Bus: reading a message doesn't remove it. Events stay in the log for the configured retention period and remain available for other consumers to read.
• Retention is time-based, not consumption-based: up to 7 days on Standard and up to 90 days on Premium and Dedicated. Data is deleted after the retention period, so the hub never gets too full.

Event Hubs is a multi-protocol engine supporting Apache Kafka, AMQP 1.0, and HTTPS natively, so existing Kafka workloads run against it without code changes or cluster management. The client libraries cover .NET, Java, Python, JavaScript, and Go, and the robust language and framework support makes it easy to integrate Event Hubs with other Azure and non-Azure services.

The ordering guarantee deserves the same caveat as Service Bus sessions: Event Hubs orders events per partition, not per hub. Ordering therefore depends entirely on the partition key you choose, and a key with poor cardinality — a customer ID where one customer generates most of the traffic — produces a hot partition that caps throughput no matter how many partitions the hub has.

Event Hubs Capture

Azure Event Hubs Capture is a built-in feature that automatically delivers the streaming data in an event hub to long-term storage. In data architecture we talk about hot paths and cold paths: the hot path processes real-time data as it arrives, for immediate alerts or live dashboards, while the cold path archives that same data for batch processing, historical analysis, or compliance auditing. Capture runs on the same stream as the hot path, so adding a cold path costs no extra plumbing.

Avro is a good fit for cold-path processing because it's a compact, fast binary format with the schema inlined in the file, which is exactly what the Hadoop ecosystem, Azure Stream Analytics, and Azure Data Factory want to consume. Parquet is possible, but only through the no-code editor's Stream Analytics integration, not from Capture's own configuration.

Three operational details are worth knowing before enabling it. Each partition captures independently and writes a completed block blob per interval, with the blob path encoding namespace, hub, partition, and timestamp. Capture copies data directly from internal Event Hubs storage, bypassing throughput-unit egress quotas, so it doesn't steal capacity from Stream Analytics or Spark readers. And enabling Capture on an existing hub only captures events arriving after it's switched on — it doesn't backfill what's already in the stream.

Azure Event Grid

An event-driven architecture lets you connect to a core application without modifying the existing code: when an event occurs, you react with specific code to respond to it. An event-driven application uses the send-and-forget principle — an event is sent toward the next system, which can be another service, an event hub, a stream, or a message broker.

Event Grid is a fully managed, highly scalable publish-subscribe service that exists to make it easier to build event-based and serverless applications on Azure. It aggregates all your events and provides routing from any source to any destination:

• Event Grid distributes events from sources such as Azure Blob Storage accounts.
• Events are distributed to handlers like Azure Functions and webhooks.
• The service manages routing and delivery from many sources, which minimises cost and latency by eliminating the need for polling.
• An event source such as Azure Blob Storage tags events with one or more topics and sends them to Event Grid. An event handler such as Azure Functions subscribes to the topics it's interested in. Event Grid examines the topic tags to decide which events go to which handlers, and forwards the relevant ones to subscribers.
• Event Grid reacts when an event happens, but the object that changed isn't part of the event data. A text file, video, or audio file that triggered the event is referenced by URL or identifier, not embedded — which is precisely the message-versus-event distinction made concrete.

Event Grid's delivery model is broader than the push mechanism the AZ-305 material describes. Push delivery is still the default and the reason polling disappears — you define a destination in an event subscription and Event Grid sends events to it, with a 24-hour retry mechanism using exponential backoff. But Event Grid also supports pull delivery on topics in a namespace, where subscriber applications connect to Event Grid and read events themselves. Pull is what you want when the consumer isn't always available, when it needs to release an already-read event back to the broker after a downstream failure, when it can't expose a public endpoint, or when the connection has to run over a private link — private endpoints are supported for pull delivery only. Event Grid also speaks MQTT v3.1.1 and v5.0 for IoT clients and supports the CloudEvents 1.0 specification.

Delivery is at-least-once with no ordering guarantee, so Event Grid handlers have to be idempotent as a matter of course rather than as a hardening step.

Event Hubs or Event Grid?

Think of Event Hubs as a giant data highway for massive streams of information, while Event Grid is a postal router that triggers specific actions when something happens. Microsoft's own comparison lands in the same place:

These aren't mutually exclusive choices. An e-commerce site can reasonably use Service Bus to process orders, Event Hubs to capture site telemetry, and Event Grid to respond to an item being shipped — or chain them, using Event Grid to react to events raised by the other two.

Gotchas and caveats

Queue Storage has no dead-letter queue. Poison-message handling is the application's job, via the DequeueCount property on dequeue.

Queue Storage ordering is a tendency, not a guarantee. An expired visibility timeout can put a message back into the queue out of its original position.

Service Bus FIFO costs parallelism. Sessions serialise processing within a session key, so an over-broad key turns a competing-consumer pool into a single consumer.

At-least-once is the norm everywhere. Queue Storage, Service Bus peek lock, Event Hubs, and Event Grid all deliver at least once. Idempotent handlers aren't optional.

Event Hubs orders per partition only. The partition key choice determines both ordering and whether you end up with a hot partition.

Event Hubs Capture doesn't backfill. Enabling it on an existing hub captures only events that arrive afterwards, and the Basic tier doesn't support Capture at all.

Data Lake Storage Gen1 is no longer a Capture destination. It was retired on 29 February 2024; Gen2 is the supported path.

Sources

• Choose between Azure messaging services — Microsoft Learn
• Compare Azure Storage queues and Service Bus queues — Microsoft Learn
• Azure Service Bus queues, topics, and subscriptions — Microsoft Learn
• What is Azure Event Hubs? — Microsoft Learn
• Capture streaming events with Event Hubs Capture — Microsoft Learn
• Enable event capture using the Azure portal — Microsoft Learn
• Introduction to Azure Event Grid — Microsoft Learn