PartitionedCloudQueue makes it easier for Azure Storage users to deal with scalability limitations of single Azure Queue (500 msgs per second per queue). You'll no longer have to implement custom partitioning if all you need is one infinitely-scalable Azure Storage queue.
Project Details
Windows Azure Queue is a Storage abstraction to represent reliable (persistent) message queue.According to Windows Azure Storage Abstractions and their Scalability Targets article: http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx
- Partition key is the queue name
- All of the messages in a queue are accessed via a single queue partition.
- A single queue is targeted to be able to process: Up to 500 messages per second
The 500 messages per second is a target not the SLA, so its always good idea to overprovision to meet the business needs. As a result any scenario requiring single logical queue with large message throughput may require custom partitioning logic where 1 logical queue is represented by N physical queues.
PartitionedCloudQueue class is a “drop-in” replacement for stock CloudQueue which provides such an abstraction layer.
Features
- Transparent queue sharding/partitioning via simple single-class client library which extends the scalability target to 500*N messages per second. (N- number of partitions)
- 1:1 compatibility with CloudQueue (except constructor) which allows PartitionedCloudQueue to replace CloudQueue with just minor code changes.
- Round-robin partition access for even load disctribution.
- Avoidance of correlated load spikes via randomization of round-robin access order.
- Support synchronous and asynchronous operations as defined in CloudQueue.
- Auto-discovery of number of partitions.
Current limitations
- Does not inherit directly from CloudQueue.
- Number of partitions is predefined, no automatic scale up/down depending on the load.
- Unlike CloudQueue where most of the time messages are ordered as FIFO with PartitionedCloudQueue most of the time messages will not be FIFO. However in general each message will be at most N steps away of its place in case of ideal FIFO (N is number of partitions).
- With current implementation each GetMessage will only attempt to get data from 1 physical queue (may change in next versions). When rate of messages is low it will require N times more calls to GetMessage() than with CloudQueue.
Features under development
- CloudQueue methods are implemented as needed, so not all methods are implemented yet. Implementation of most of the methods is trivial.