One click to deploy a RabbitMQ cluster handling over 1 million msg/sec
Monday, June 23, 2014
If you develop scalable applications, you often want to use a messaging system, but you may be concerned that setting up such a system is time consuming and its throughput limits are not sufficient for your growing needs. But what if the whole setup required just a few minutes and benchmarks showed that the system was capable of processing over one million messages per second? This is exactly what we are announcing today with Pivotal.
When we talked about high-throughput, low-latency messaging scenarios with our customers, they were particularly interested in RabbitMQ, a popular open source messaging system that serves a variety of use cases.
We recently made it super easy for our users to deploy a dedicated RabbitMQ cluster on Google Compute Engine. All it takes is one click after providing some basic information, such as the size of the cluster and RabbitMQ username / password.
In a matter of a few minutes, you can get a RabbitMQ cluster deployed and configured. You will also get two load balancers set up for your cluster. As a next step you can securely access RabbitMQ web management console and start developing your app that uses RabbitMQ. Even if you need a very large 64-node cluster, the deployment usually takes less than 10 minutes.
In fact, the RabbitMQ cluster is so easy to deploy and configure that we’ve received feedback that for some developers the experience may be similar to having their own "Rabbit service" consisting of multiple nodes.
We also wanted to demonstrate that a RabbitMQ cluster running on Google Compute Engine is a good fit for high-throughput and low-latency scenarios. To do so, we ran a series of benchmarks in various configurations. These test clusters were able to sustain a throughput of over 1 million messages published and consumed per second (a sustained combined ingress/egress of over two million messages per second).
To put this throughput in context, one million messages per second translates to 86 billion messages per day. U.S. text messages reached 6 billion per day in 2012. Apple processes about 40 billion iMessages per day, and WhatsApp recently hit a new daily record in December when it sent 20 billion daily messages.
Google and Pivotal Benchmarks
We then invited engineers from Pivotal, the company behind RabbitMQ, to validate and endorse our results.
For our joint tests on Google Compute Engine we provisioned a cluster of 32 virtual machines with 8 vCPUs and 30GB of RAM each, and deployed RabbitMQ on Debian.
To generate load we used RabbitMQ PerfTest client tool running on a set of GCE virtual machines separate from the ones housing the RabbitMQ cluster nodes. As per RabbitMQ clustering recommendations the clients were configured to target the IP address of a GCE load balancer that was automatically created through click to deploy. This is a more flexible approach than having the clients know the IP addresses or host names of every node in the Rabbit cluster.
After the traffic generating clients warmed up, we reached a steady state as shown in RabbitMQ’s web management console: over 1.3 million messages published and consumed per second.
Under multiple hours of such load, the RabbitMQ cluster and its underlying GCE VMs remained stable, with average throughput staying at approximately the same levels as shown in the screenshot.
Conclusion
We are excited to offer our customers an easy way to deploy RabbitMQ. We are proud that both Google and Pivotal benchmarks validate that Compute Engine is an excellent choice for running RabbitMQ in a public cloud
To deploy your own dedicated RabbitMQ cluster, go to Google Developer Console and navigate to an existing project, or create a new one, and you will have an option to deploy RabbitMQ from the project dashboard:
You can also find a link at the bottom of the project landing page:
There is no extra charge for using click to deploy - you will be only billed for the underlying Compute Engine resources, so why not deploy your cluster today and tell us what you think.
-Posted by Grzegorz Gogolowicz, Cloud Solutions Architect
P.S. If you don’t want to have your own dedicated RabbitMQ cluster deployed as described in this post, and rather would like to use a managed RabbitMQ, you may want to check out CloudAMQP offering that runs on Google Compute Engine.
When we talked about high-throughput, low-latency messaging scenarios with our customers, they were particularly interested in RabbitMQ, a popular open source messaging system that serves a variety of use cases.
We recently made it super easy for our users to deploy a dedicated RabbitMQ cluster on Google Compute Engine. All it takes is one click after providing some basic information, such as the size of the cluster and RabbitMQ username / password.
In a matter of a few minutes, you can get a RabbitMQ cluster deployed and configured. You will also get two load balancers set up for your cluster. As a next step you can securely access RabbitMQ web management console and start developing your app that uses RabbitMQ. Even if you need a very large 64-node cluster, the deployment usually takes less than 10 minutes.
In fact, the RabbitMQ cluster is so easy to deploy and configure that we’ve received feedback that for some developers the experience may be similar to having their own "Rabbit service" consisting of multiple nodes.
We also wanted to demonstrate that a RabbitMQ cluster running on Google Compute Engine is a good fit for high-throughput and low-latency scenarios. To do so, we ran a series of benchmarks in various configurations. These test clusters were able to sustain a throughput of over 1 million messages published and consumed per second (a sustained combined ingress/egress of over two million messages per second).
To put this throughput in context, one million messages per second translates to 86 billion messages per day. U.S. text messages reached 6 billion per day in 2012. Apple processes about 40 billion iMessages per day, and WhatsApp recently hit a new daily record in December when it sent 20 billion daily messages.
Google and Pivotal Benchmarks
We then invited engineers from Pivotal, the company behind RabbitMQ, to validate and endorse our results.
For our joint tests on Google Compute Engine we provisioned a cluster of 32 virtual machines with 8 vCPUs and 30GB of RAM each, and deployed RabbitMQ on Debian.
To generate load we used RabbitMQ PerfTest client tool running on a set of GCE virtual machines separate from the ones housing the RabbitMQ cluster nodes. As per RabbitMQ clustering recommendations the clients were configured to target the IP address of a GCE load balancer that was automatically created through click to deploy. This is a more flexible approach than having the clients know the IP addresses or host names of every node in the Rabbit cluster.
After the traffic generating clients warmed up, we reached a steady state as shown in RabbitMQ’s web management console: over 1.3 million messages published and consumed per second.
Under multiple hours of such load, the RabbitMQ cluster and its underlying GCE VMs remained stable, with average throughput staying at approximately the same levels as shown in the screenshot.
Conclusion
We are excited to offer our customers an easy way to deploy RabbitMQ. We are proud that both Google and Pivotal benchmarks validate that Compute Engine is an excellent choice for running RabbitMQ in a public cloud
To deploy your own dedicated RabbitMQ cluster, go to Google Developer Console and navigate to an existing project, or create a new one, and you will have an option to deploy RabbitMQ from the project dashboard:
You can also find a link at the bottom of the project landing page:
There is no extra charge for using click to deploy - you will be only billed for the underlying Compute Engine resources, so why not deploy your cluster today and tell us what you think.
-Posted by Grzegorz Gogolowicz, Cloud Solutions Architect
P.S. If you don’t want to have your own dedicated RabbitMQ cluster deployed as described in this post, and rather would like to use a managed RabbitMQ, you may want to check out CloudAMQP offering that runs on Google Compute Engine.
Follow