Message middleware is a middleware technology composed of a message transmission mechanism or message queue mode, which uses an efficient and reliable messaging mechanism for platform-independent data exchange, and integrates distributed systems based on data communication. At present, there are many MQ products in the industry, such as RabbitMQ, ActiveMQ, ZeroMQ, etc., which are excellent message middleware, but which one should we choose in the project? This paper evaluates and compares the following message queue products: RabbitMQ, ZeroMQ, ActiveMQ, MSMQ, Redis, and memcacheQ
Digression: Here we can first think about a small question: "Why do we need message queue services in web applications?" ”
For example, a large number of insert, update, and other requests arrive at MySQL at the same time, directly leading to countless row locks and table locks, and even too many requests in the end, thus triggering too many connections errors. By using message queues, we can process requests asynchronously, relieving the strain on the system.
RabbitMQ
It is an open source message queue written in Erlang, which supports many protocols: AMQP, XMPP, SMTP, STOMP, which makes it very heavyweight and more suitable for enterprise-level development. It is a leading implementation of the AMQP protocol, which implements a broker architecture, which means that messages can be queued on a central node before being sent to the client. There is good support for routing, load balancing or data persistence. This feature makes RabbitMQ easy to use and deploy, suitable for many scenarios such as routing, load balancing, or message persistence, and can be done with just a few lines of code with message queues. However, this makes it less scalable and slower because the central node increases latency and is larger after message encapsulation. To configure RabbitMQ, you need to install the Erlang environment on the target machine.
Click to view this image in a new window
? MQ(ZeroMQ)
It is known as the fastest message queuing system, especially for high-throughput demand scenarios. It is a very lightweight messaging system developed specifically for high throughput/low latency scenarios and can often be found in applications in the financial world. Compared to RabbitMQ, ZeroMQ supports many advanced message scenarios, but you have to implement individual blocks in the ZeroMQ framework (such as sockets or devices, etc.).
? MQ (ZeroMQ) can implement advanced/complex queues that RabbitMQ is not good at, but developers need to combine multiple technical frameworks by themselves, and the technical complexity is a challenge to the successful application of this MQ. ZeroMQ has a unique non-middleware model where you don't need to install and run a message server or middleware because your application will play this service role. All you need to do is simply reference the ZeroMQ library, which can be installed using NuGet, and you can happily send messages between applications. However, ZeroMQ only provides non-persistent queues, which means that if the machine goes down, the data will be lost. Among them, Twitter's Storm uses ZeroMQ as the transmission of data streams. ZeroMQ is very flexible, but you have to learn its 80-page manual (if you're writing about a distributed system, be sure to read it).
ZeroMQ has no middleware architecture and does not require any service processes and runs. In fact, your application endpoint plays this service role. This makes it very simple to deploy, but the concern is that you have nowhere to watch if something goes wrong with it. As far as we know, ZeroMQ only offers non-persistent queues. You can implement your own audit and data recovery capabilities where you need them.
Click to view this image in a new window
MSMQ
This is the only thing in Microsoft's product that is considered valuable. If MSMQ can prove that it can handle this kind of task, they will choose to use it. The point is that this thing is not complicated, nothing but receiving and sending; It has some hard limitations, such as the maximum message size of 4MB. However, it can solve these problems by connecting to some software like MassTransit or NServiceBus.
Click to view this image in a new window
Jafka/Kafka
Kafka (which distributes messages across different nodes) is a distributed MQ system developed and open-sourced by LinkedIn in December 2010, and is now an incubation project of Apache, a high-performance cross-language distributed Publish/Subscribe message queuing system, and Jafka is incubated on top of Kafka, which is an upgraded version of Kafka. It has the following characteristics: fast persistence, which can persist messages under the system overhead of O(1); High throughput, which can reach a throughput rate of 10W/s on an ordinary server; Completely distributed system, Broker, Producer, and Consumer all natively support distributed and automatically achieve complex equilibrium. Supports parallel loading of Hadoop data, which is a viable solution for log data and offline analysis systems like Hadoop, but with the limitations of real-time processing. Kafka unifies online and offline message processing through Hadoop's parallel loading mechanism, which is also important for the system studied in this topic. Apache Kafka is a very lightweight messaging system relative to ActiveMQ, and in addition to very good performance, it is also a distributed system that works well.
Click to view this image in a new window
Apache ActiveMQ
ActiveMQ sits somewhere between the two (RabbitMQ & ZeroMQ), similar to ZemoMQ, and can be deployed in both proxy and P2P modes. Similar to RabbitMQ, it is easy to implement advanced scenarios and requires low consumption.
ActiveMQ is known as the backbone of the Java world. It has a long history and is widely used. It's also cross-platform, providing a natural integration access point for products that aren't on Microsoft's platform. However, it is only possible to be considered if it has run past MSMQ. To configure ActiveMQ, you need to install the Java environment on the target machine.
Click to view this image in a new window
It is important to note that ActiveMQ's next-generation product is Apollo, which is based on the ActiveMQ prototype and is a faster, more reliable, and easier to maintain message broker tool. Apache calls Apollo the fastest and most robust STOMP (Streaming Text Orientated Message Protocol) server.
The features of Apollo are as follows:
Stomp 1.0 and Stomp 1.1 protocols are supported
Topics and queues
Queue Browser
Theme persistent subscriptions
Mirror queue
Reliable messaging
Message expiration and exchange
Message picker
JAAS verified
ACL-based authorization
Support SSL/TLS and certificate validation
REST Management API
Click to view this image in a new window
Redis
It is a Key-Value NoSQL database, which is actively developed and maintained, although it is a Key-Value database storage system, but it supports MQ functions, so it can be used as a lightweight queue service. For the onboarding and out-queue operations of RabbitMQ and Redis, 1 million times each, and the execution time is recorded every 100,000 times. The test data is divided into four different sizes: 128Bytes, 512Bytes, 1K, and 10K. Experiments show that when joining the team, the performance of Redis is higher than that of RabbitMQ when the data comparison is small, and if the data size exceeds 10K, Redis is unbearably slow. When going out of the team, Redis showed very good performance regardless of the size of the data, while RabbitMQ's performance was much lower than Redis'.
MemcacheQ
Persistent message queue Memcacheq (MCQ for short) is a lightweight message queue, MemcacheQ features:
1 Simple and easy to use
2 Fast processing
3 Multiple queues
4 Good concurrency performance
5 Compatible with Memcache protocol. This means that just install the memcache extension, no additional plugins are required.
6 It is also convenient to use in the zend framework.
Ultimately, these products:
1. Both have their own client APIs or support multiple programming languages;
2. There is a lot of documentation;
3. Positive support was provided.
4. ActiveMQ, RabbitMQ, MSMQ, Redis all need to start service processes, which can be monitored and configured, and the others are problematic
5. They all provide relatively good reliability (consistency), scalability and load balancing, and of course, performance
I won't talk nonsense here, attached below is a set of test results intercepted from the Internet. The number of messages sent and received per second is displayed. The whole process generated a total of 1 million 1K messages. The test was performed on a Windows Vista stand-alone machine.
As you can see, ZeroMQ is not a level like anything else. Its performance is surprisingly high. Despite this, this product does not provide message persistence, cannot easily store and monitor intermediate processes, and requires self-auditing and data recovery, so it is not satisfactory in terms of ease of use and HA. The conclusion is clear: if you want an app to send messages as quickly as possible, you choose ZeroMQ. It is more valuable when you don't care too much about losing certain messages by chance.
The blogger in this article hopes (and does not hope very much) to use Rabbit, Rabbitmq has built-in ha, if you form a cluster, there is no need to worry about issues such as load balancing, and you can set up a queue mirror. But this kind of thing is that there should be more testing, and you end up with a favorite, and all the things I've heard and read about Rabbit make me feel like it should be the best choice.
|
Posted on 12/15/2016 10:52:40 AM
|
|
|
|
