In this post we are going to find out the different Enterprise Messaging solutions provided by
Azure public Cloud. The services we are looking are offered by
SAAS model. You do not need to worry about managing underlying infrastructure or scaling. You just need to write code to consume these services Azure will take care of managing hardware and scaling these services. They are highly scale-able and can have input/output of millions of messages per minute.
In this post we will not discuss how we use these services in code but we are more focused on highlighting their differences that are important when you are trying to make a decision which one should be used for your applications. So this post is about one step before you start coding or consuming one of these services.
Message vs Event
In the past I always struggled to clearly define the differences between what should be an
Event and What should be a
Message they seems very similar but they have some differences that are important.
In very simple terms Messages are like commands. You send a message from one system to another when you want something to happen. Message queues are like letter box you check everyday and if you received a new letter it will be there. So it’s basically you are polling or checking time to time to see if there is some new message in there. That’s why there will be some delay with processing Messages.
For example: In your your system you want to send email asynchronously using Send Grid so you enqueue a message on service bus queue and an azure function is checking messages on that queue and sending emails using the information provided in the message payload.
Events are generated when something is changed in one system and system is publishing the meta data about that change and if there are any listener systems whose interested in that event then can react to it.For event a listener is always listening so they process as soon as even is received that means they are near realtime in case of Event payload does not contain the data but information about the data actually.
For example: In an IOT system you have a sensor that is broadcasting events for every time a door is opened or closed. Or A New Blob is uploaded to a container inside a storage account. This will publish blob upload event and event payload will not contain the actual contents of the Blob but Metadata about the blob which may include Blob name, blob url and Timestamps about when it happened etc. Then there is an azure function who is listening to blob upload events from a given container and processing(Compression, Optimizations etc) the newly uploaded blobs
|Payload Contains the Data||Payload is information about the data|
|Is the something that happened||Say something has happened|
|Sender generally knows about who is going to receive it||Sender generally don’t know/don’t care about the receiver|
|Usually some type of polling||Near real time|
Let’s explains each of the point in the above comparison
- Payload of message contains the actual data that needs to consumed. Where as payload of an event is metadata about the data or state that is changed.
- Message is used when you want something to happen you have some intent. Where as event is just a notification of something is changes of happened in a system.
- For messages sender usually is aware about the consumers who is going to consume the message and sometimes it can be a 2 way communication also and can get a message back also or actions if message succeed or fail. In case of events the sender usually does’t know/care who is going to act on this events.
The processing of events is usually near real time but that does not qualify them as the best solution for every problem where as there are problems that are better solved with
Messages instead of events.
When you are creating a azure function that have a trigger of Blob Uploaded basically you are relying on Blob Uploaded Event that is broadcasted on Azure Event Grid and use that event as trigger point for your azure function what you receive inside Even payload is not the actual blob but the meta-data about the Blob name and other attributes like container and URL etc. ** That was the bit of introduction about what is Event and what is Message and how they are different and why these differences are significant. Let’s go back to the topic of this post Messaging solutions in Azure**
Messaging options in Azure
Usually when you have to process lot of data in your system Asynchronously you need some sort of queue to hold the data that is waiting to be processed and then data processor remove data records from queue and process them one by one. Queuing Solutions are used when you are using messaged to communicate between two systems.
We are not talking about Event Hub, IOT Hub and Event Grid in this post because they are more of event related services not messaging solutions and we highlighted the differences between events and messages. But if you want me to write about Event solution offerings please let me know in comments.
In some cases there is 1to1 one sender and one consumer but there are scenarios single record of data needs to be processed by multiple systems. (PUB-SUB architecture)
There are 2 major solutions available in azure for enterprise scale messaging. (Not Events - But Messaging)
- Azure Storage Queues
- Azure Service Bus (ESB)
Now we will look at characteristics of each of them that may help you decide which is better choice for your use case.
Storage Queues are created under Azure Storage V2 accounts where as for the Service Bus you will have to create a namespace with a name that is globally unique and then you can create Queues and Topics under that namespace. You can control access to Azure service bus at the Namespace granularity or at Queue/Topic granularity.
Now we will look at some features of each of these queueing services provided by Azure that will may help us decide which one is better for our use case.
|Azure Storage Queues||Azure Service Bus|
|REST Interfaces||Multi-Platform Support|
|Cheaper||Bit More pricey but still way cheaper than managing your own infrastructure|
|Missing lot of enterprise features||Support in order and at-most once|
|Message Size Limit 64Kb||Max Message Size 256Kb to 1Mb|
|Store messages upto 7 Days||Configurable message expiry|
|No dead letter out of the box||Dead letter queues out of the box|
|No PUB-SUB||Publish and Subscribe Features|
|Not good fit for enterprise messaging||Duplicate detection, Session and Partitioning|
Let’s Explain each of the above characteristics in more details.
REST Interfaces vs Multi-Platform Support : In oder to interface with Azure storage queues we can either use the REST api directly or use the Azure Storage SDK that under the hood uses the REST API calls to manage the information stored in storage. Sometimes the REST api interface is not good enough for your needs can be due to performance requirements so Azure service Bus is a better choice as it provides multiple ways to interact with the queue. You can use plain HTTP interface or you can some advanced message queuing protocol (AMQP) to interact with queues.
Cheaper vs Azure Service Bus vs Own Infrastructure : Even though the azure service bus is bit more expensive option than the storage queues. Considering it is managed and you don’t need to worry about managing the infrastructure and scalability. Paying $0.0185/hour for standard plan that comes with free 13 million operations per month is a very good value proposition in my opinion. Even if you decide to roll on your on premise queuing solution you cannot do much if you try to stay within the above cost boundaries.
Missing lot of Enterprise Features vs Support in Order and At Most Once : Sometimes it is important to process the messages in the same order as they are published to the queue. I understand temporal coupling should be avoided whenever possible but still in some scenarios we cannot avoid such temporal coupling. If you have such requirements in you system design and you cannot workaround it you will need to
Azure Service Busas it provides the
FIFOcapability using the
Sessionsfunctionality of azure service bus.
Message Size Limit: Most of the time 64kb message size is good enough for our requirements. But there are some scenarios where you are using some integration pipeline to sync data between 2 systems and want to send all the data updated as part of one transaction as one message so that it can be processed atomically. In that scenario you may need bigger message size. In that scenario
Azure Service Bussupport 1MB message size for premium tier. If you need even more than that i will recommend rethinking if queuing is the right solution for your problem.
Store Message Upto 7 Days vs Configurable Message Expiry : This comparison point is simple
Azure Service Busprovides far more flexibility over the retention of the messages in the queue.
No Dead Letter vs Dead Letter Queues out of box : Errors, Bugs and Request Failures happens to almost every production system in the world. No matter how elegant it is. What differentiate between a good and bad system design is how well you can monitor, log and debug those Errors or failures and make incremental improvements to reduce their count. Dead letter queues are the queues where you messages will end up if they cannot be processed either due to expiry or the message processing failures and even exceeded retry counts. You can go into dead letter queues and look at all the failed messages and try to find out the cause behind that and make required improved improvement and fixes. Other thing you can do with Dead letter queues is you can monitor for example you can create dashboards to look at the message count in dead letter queues. If this count trend start moving up suddenly we can get a proactive alarm that something went wrong instead of waiting for users of system to complain about system being not operational or not working as expected. So if you want such proactive monitoring and debug capabilities with your queueing solutions
Azure Service Buswill be the right solutions.
No PUB-SUB vs Publish And Subscribe Features : If you have the requirement where you will broadcast a message and multiple subscribers can subscribe get that message and process or you have more sophisticated requirement like Depending on the
Some Message Propertiesyou want a message to be processed by different subscribers. Then you can utilize the
Subscriptioncapability provided by the
Azure Service Bus. You can define the rules for each subscription and the rule evaluates to true then only that subscriber will receive the message. But keep in mind these rules can be evaluate only based on the metadata or properties of the message not the contents of the message. This PUB-SUB capability is not available with
Azure Storage Queues.
You cannot create subscription rules for topics inside the Azure web portal. But we can use the service bus explorer.
Message that cause filter evaluation exception move to dead letter queue. Because if it is not enabled if messages are sent that does not match to a subscription will simply be lost.
In this article we looked at the Queuing solution to send message between two system. If we have look at other different Types of enterprise eventing and messaging solutions provided by azure and their applications the table will look like
|Intent||SB Queues||SB Topic||Storage Queues||Event Grid||IOT Hub||Event Hub|
|Send a Message||✔||✔||✔|
|Subscribe and Publish Discrete Events||✔|
|Analyze Stream of Events in Realtime||✔|
|Send Message to One Subscriber||✔|
|Send Message to Multiple Subscribers||✔||✔||✔||✔|
The focus of this post was Enterprise Messaging Solutions solutions provided by Azure Public Cloud (Azure Storage Queues and Azure Service Bus). How a message is different from an event. We need messages solutions when two systems are exchanging messages between them to make something happen in other system. There are other use cases where instead of sending messages between two systems we want to send events or notifications when some information changes in the on of the system.
We looked at the some messaging solutions in details and tried to highlights the benefit of one over the other. I think talking about Event solutions like Azure Event Grid and Event Hub deserve a post of its own. If you can spare 2 minutes of your time please let me know in comments. You would like me to write
- A post about Advance features of Azure Services Bus and their Use Cases (Partitioning, Sessions and Topic Subscription Rules etc)
- A post about more in depth comparison of Event solutions provided by
Azure Public Cloud.