In this article, we will look at asynchronous communication across services in a Service Oriented Architecture through Message Oriented Middleware (brokered and brokerless) using the Producer-Consumer pattern.
Advantages of Message Oriented Middleware
- Asynchronicity/Loose coupling between components: MOM technologies use asynchronous message passing as opposed to request-response architectures. In asynchronous systems, message queues provide temporary storage when the destination program is busy or not connected. In addition, most asynchronous MOM systems provide persistent storage to back up the message queue. This means that the sender and receiver do not need to connect to the network at the same time (asynchronous delivery), and problems with intermittent connectivity are solved. It also means that should the receiver application fail for any reason, the senders can continue unaffected, as the messages they send will simply accumulate in the message queue for later processing when the receiver restarts.
- Routing: Many message-oriented middleware implementations depend on a message queue system. Some implementations permit routing logic to be provided by the messaging layer itself, while others depend on client applications to provide routing information or allow for a mix of both paradigms.
Disadvantages of Message Oriented Middleware
The primary disadvantage of many broker based message oriented middleware systems is that they require an extra component in the architecture, the message transfer agent (message broker). As with any system, adding another component can lead to reductions in performance and reliability, and can also make the system as a whole more difficult and expensive to maintain.
Criteria for evaluating Message Queues
- Durability: Whether messages can be written to disk, or even committed to aDBMS for reliability in case of server crashes.
- Point-Point vs Publish-Subscribe: Whether the message queue provider supports point-point or publish-subscribe mode of delivery
- Push vs Pull Model : Whether the consumers have to pull the messages or the broker pushes the messages to the consumers.
- Message Acknowledgements: Whether messages acknowledgement modes are supported between consumers, brokers and producers.
- Delivery Policies: Whether a message should be delivered at least once, or no more than once.
- Purging Policies: Whether Queues or Messages support a “time to live”.
- Message Size/Format: What message sizes and types are supported.
- Message filtering/routing policies: Whether messages can be filtered so that a consumer can see only messages matching some criteria of interest and messages can be routed to consumers based on defined policies. An example of why routing is necessary can be understood from the scenario described below.
Lets consider an enterprise social application architected using SOA as different services like
ItemService – Manages items like documents/events/pages etc.
UploadService – Handles item uploads.
FeedService – Manages a users feed.
SearchService – Handles search in the entire application.
ConversionService – Generates intermediary artifacts like thumbnails for image document items.
User uploads a new file using the UploadService which has to trigger the ItemService for creating a new item once the upload is successful. ItemService in turn triggers
FeedService to update the users feed,
SearchService to index the item so that its searchable,
ConversionService so that any intermediary artifacts for the item can be generated.
ConversionService in turn might use other services within the company or outside. ConversionService should also notify FeedService about status of items it submitted for conversion as shown in Figure0.
The dependencies can go to many levels depending on the nature of the application and the dependency between services.
- Message Ordering: Whether messages are delivered to consumers in a FIFO ordering or not.
- Batching Policies: Should messages be delivered immediately, or should the system wait a bit and try to deliver many messages at once.
- Scalability: What order of throughput, latency and volume of queues, messages is supported.
- Availability: Does the queuing system support high availability in case of failures of servers.
- Operational and financial cost: Whether the solution incurs any financial or operational burden
- Cross platform interoperability: Whether the solution works across disparate services/systems either of different enterprises/companies or within the enterprise to easily exchange messages between each other regardless of the message broker vendor and platform.
- Using Database as a Message Queue
The current architecture used in A360 for communication across services is outlined below in Figure1
When ServiceA has to communicate with ServiceB, it uses ServiceB’s HttpClient which inturn goes through the Custom Load Balancer (which knows information about all the instances of ServiceB through data structures populated from the config DB) to distribute requests to the available instances of ServiceB in a round-robin fashion.
ServiceBHttpClient constructs a Message with all the required inputs for the call to ServiceB and makes a http call through the Custom LB. ServiceB has a MessageHandler component which receives these http calls with the message in the body of the request, delegates the call to the appropriate business component which pushes the message to a queue (which is a table in the database) and returns a response immediately.
Every instance of ServiceB is bundled with a MessageProcessingFramework which has a pool of threads continuously polling (in a certain interval) for messages in the queue by locking the table to ensure a message is picked up only by one instance for processing. Each instance picks up a batch of messages to process and invokes the appropriate business component.
- If new instances are added or an existing instance comes down, the config has to be reloaded.
- The load balancing strategy has to be intelligent to distribute requests to instances that are less loaded. A simple round-robin strategy may not be optimal.
- A short interval for polling may be hammering the database with constant queries. Alternatively, a long interval may result in processing delays.
- The solution does not scale well. Polling adds a significant load to the database even at a medium volume. A database table is typically configured to be fast at adding data, updating data or querying data, but almost never all three on the same table. As the volume scales up, load increases on the database and performance decreases.
- Inserting, updating and querying on the database table by a lot of concurrent producers and consumers may result in race conditions.
- Does not support a callback mechanism to notify a Producer of any intermediary events when a consumer is either processing or has finished processing a message. For example, ServiceA submits a document to ServiceB which generates multiple intermediate derivatives (thumbnails for images etc) each taking a certain amount of time to be generated. Whenever an artifact is available, ServiceA has to be notified. In such cases a callback mechanism (for ex, invoking a callback url on the producer end) needs to be supported.
Evaluation of Solution
Durability: Messages are saved to the database always.
Point-Point vs Publish-Subscribe: Supports only Point-Point model.
Delivery Policies: Messages are guaranteed to be delivered as the Message producer gets an acknowledgement. Messages are consumed by only one consumer by using a status field in the table to co-ordinate between multiple consumers.
Message Acknowledgements: Messages are acknowledged as soon as they are stored in the persistent store
Purging Policies: Messages can be retained and can be purged manually or through a database trigger based on some purging policies.
Message Size/Format: Can store binary/text messages of any size limited by the underlying disk space and datatype used for the column. Even though there is no limitation on message size, its not ideal to have messages of very large size.
Message filtering/routing Policies: Message filtering has to be implemented in the MessageHandler component of the consumer service. Does not support routing of message to multiple services implicitly. Each message will have to be sent to every other Service that has to consume the message as shown in Figure2 below.
Message Ordering: Consumers pick up messages from the table based on the timestamp of message. This does not result in strict FIFO due to multiple consumers.
Batching Policies: Messages can be processed in batches based on batching policies defined.
Scalability: Not very scalable. Would hit performance issues due to lock contention on database read/write operations and too much disk I/O.
Availability: Using a dynamic service discovery solution to detect service instances being added or going down, would achieve high availability.
Operational and Financial Cost: Does not incur any financial cost except the initial effort to design the solution; would incur minimal operational burden.
Cross Platform interoperability:
- Using Amazon SQS as the broker
Amazon Simple Queue Service (SQS) is a fast, reliable, scalable, fully managed message queuing service. Figure3 illustrates how to use SQS to design an asynchronous messaging solution in SOA.
ServiceA pushes messages to ServiceB’s SQS queue. MessageHandler component in each instance of ServiceB polls for messages from the queue (in batches) and process the messages. SQS takes care of locking on concurrent read requests on the queue to ensure the same messages are not returned to multiple consumers.
- Consumers have to poll the queue in order to receive messages. If messages are available, they are delivered; otherwise consumers must idle for a while, and then poll again increasing the latency of processing messages and also the cost of using SQS as each request to SQS incurs a minor cost which is multiplied when we consider the number of consumers making requests and the number of queues. When there is a chain of services using queues for asynchronous communication, the latency effect is multiplied. SQS supports long polling (with a configurable wait time) to overcome this; but threads polling for messages would remain idle until messages are available.
- Consumers are responsible for deleting messages in the queue after processing as the messages are not deleted by SQS once delivered to the consumer. SQS generates a receipt handle for each successful delivery of messages to the consumer. The consumer needs to pass in the receipt handle to delete all the consumed messages from the queue once it finishes processing all the received messages.
- There is still a locking overhead when multiple consumers request for receiving/deleting messages but that overhead is handled by SQS.
- If a Producer has to be notified of any intermediary events when a consumer is processing the Message, either a callback URL has to be made available in the message or the consumer can send a message to the Producer’s SQS queue as shown in Figure4.
Evaluation of Solution
Durability: Messages are durable though only for a limited period of time. (default being 4 days and max being 14 days)
Point-Point vs Publish-Subscribe: Supports only Point-Point model.
Delivery Policies: Messages are guaranteed to be delivered at least once since they are retained in the queue as long as a consumer instructs SQS to delete messages. The possible scenarios for more than once delivery are
- Messages not getting deleted from SQS before visibility timeout due to consumer crashing or taking longer than visibility timeout to process the message. The visibility timeout should be set to a value longer than the consumer workers normally take to finish processing, plus a bit extra for padding.
- Concurrent consumers polling visible messages simultaneously.
- In case of SQS server crashing and coming back up, if the messages that the server going down holds are already processed, they would be deleted from other servers but not from the server that went down resulting in re-delivery once the server comes up again.
As a result of at-least-once delivery, consuming services need to make sure that re-processing of messages does not result in an inconsistent state of the system by handling the messages in an idempotent manner.
Message Acknowledgements: Does not support message acknowledgement; a trade-off that SQS makes for providing high throughput.
Purging Policies: Messages are retained in SQS for a default period of 4 days and a max period of 14 days after which they are automatically deleted from the queue.
Message Size/Format: Can store text messages upto 256kb.
Message filtering/routing Policies: Does not support filtering/routing of messages. Message filtering has to be implemented in the MessageHandler component of the consumer service. Does not support routing of message to multiple consuming services. Each message will have to be sent to the SQS queue for every other Service that has to consume the message as shown in Figure5
Message Ordering: Messages are not guaranteed to be delivered in FIFO order. When a client polls for messages, SQS picks up messages from a random sample subset of servers due to which message ordering cannot be guaranteed.
Batching Policies: Messages can be requested in batches with a max batch size of 10 messages.
Scalability: Highly Scalable in terms of throughput. Latency is in the order of a few to several seconds as the message will have to be replicated before it is available for consumption and message consumption by the consumers is not real time due to polling interval. Supports unlimited volume of queues/messages.
Availability: Supports High availability in case of server failures through replication of the queue across a cluster of servers. Chooses Availability over Consistency in the event of a network partition.
Operational and Financial Cost: Zero operational cost but every request to SQS would incur some cost (1$ for 1million requests).
- Using JMS Message Brokers
Specific JMS broker implementations are not evaluated in detail in this section. JMS has evolved over 10 years into a robust and mature specification. We will specifically look at cross platform interoperability with JMS and the underlying issues. There are many JMS brokers like ActiveMQ, HornetQ etc which are quite popular and have been used extensively.
With JMS, it becomes easy to replace any JMS-compliant message broker with another one with minor configuration changes provided the protocol used for communication is supported by both brokers. JMS provides a High level API for Producers and Consumers which abstracts out the underlying wire protocols used for communication between the clients and JMS brokers. Messaging Interoperability between Java Platform and other platforms/languages and between non-Java platforms is proprietary and dependent on broker implementations.
Communication between Producers->Broker and Broker->Consumers can happen through various wire-level protocols. Before the advent of AMQP, there were other protocols like OpenWire, STOMP, XMPP etc. REST over HTTP ( to provide technology agnostic and language neutral web based API to messaging) is another communication option. Different vendors supported different protocols. For example,
ActiveMQ has support for OpenWire, STOMP, XMPP, MQTT, WSIF protocols and REST (over HTTP). After AMQP was standardized, ActiveMQ also added support for AMQP.
HornetQ has support for STOMP, REST and also AMQP after it was standardized.
Apache Apollo has support for STOMP, MQTT, OpenWire, AMQP, WebSockets.
There are many client libraries that support one or more protocols for each language platform. Some of the libraries and the supported protocols is given below. This is not an extensive list and includes only a few libraries and protocols.
- ActionScript3 (as3-stomp – STOMP, as3-amqp – AMQP)
- WebSockets (Stomple, stomp-websocket – STOMP; Eclipse Paho – XMPP)
- C (openwire-c – OpenWire; libstomp – STOMP; rabbitmq-c – AMQP)
- C++ (Apache CMS – OpenWire, STOMP, AMQP-CPP – AMQP)
- C# and .Net (Apache NMS – OpenWire, STOMP, AMQP, MSMQ)
- Java (ActiveMQ Java Client – OpenWire; Gozirra, Stampy – STOMP; rabbitmq-java – AMQP)
- Perl (AnyEvent::Stomp, Net::Stomp – STOMP; AnyEvent::RabbitMQ, Net::AMQP – AMQP)
- PHP (simplisticstompclient, stomp – STOMP; php-amqplib – AMQP)
- Python (pyactivemq, stompest – STOMP; pika – AMQP)
- Ruby (activemessaging, onstomp – STOMP; bunny, March Hare – AMQP)
- Erlang (stomp.erl – STOMP; bunny_farm – AMQP)
With these semantics of communication in JMS, we will look into messaging models before the advent of AMQP.
- In the simplest case where the Producer and Consumer are both written in Java, the resulting messaging model would be as shown in the Figure below
In this particular example, the clients are using the activemq client which uses OpenWire protocol to communicate with broker; however we can replace the client with any other client which uses OpenWire protocol for communicating with broker. Also, any JMS broker which supports OpenWire protocol like ActiveMQ or Apollo can be used as the middleware. OpenWire is the default native protocol used by ActiveMQ. In this example, the broker does not have to use a message bridge to transform the message structure and protocol into another format since both clients are talking to the broker through the same protocol.
- Now lets consider a case where the Producer is written in Java and the Consumer is written in Python or vice-versa. In this case the messaging model would be as shown in the Figure below.
In Model 1, the Java Producer uses the ActiveMQ client to communicate to the broker using OpenWire and the Python Consumer uses the pyactivemq client to talk to the broker in STOMP. The most popular protocol choice for Python is STOMP and there are no clients available that use other protocols for communication. So, in this model, the broker has to provide a message bridge which can transform the protocol and message structure from OpenWire to Stomp. It will not be possible to replace one broker with any other as the new broker should also support the transformations that the existing broker was supporting. Simlarly in Model2, the message bridge should be able to transform from Stomp to OpenWire.
In Model3, there is no transformation required as both the Producer and Consumer clients are using the same protocol to communicate. Although this scenario seems to solve the cross-platform interoperability problem, there are some inherent problems.
- Both protocols may not support the same message body types.
- Applications would be locked into one specific vendor solution (or in some cases only a few vendor choices) due to the built-in message bridge.
- Both protocols may not support same datatypes, custom properties and header properties between the messaging clients.
These scenarios illustrate that JMS isnt’t the right solution for cross-platform interoperability. While vendors such as ActiveMQ, HornetQ, Apollo, SonicMQ etc provide some level of cross platform interoperability, it is usually through proprietary protocils, API’s and client libraries. Without some sort of messaging standard, it is difficult to know what messaging system another company or system is using and how to interface with it.
This is exactly why AMQP was created; to standardize cross-platform messaging interoperability. Whereas JMS provides a standard messaging API for the Java Platform, AMQP provides a standard messaging protocol across all platforms. AMQP does not provide a specification for an industry standard API. Rather, it provides a specification for an industry standard wire-level binary protocol to describe how the message should be structured and sent across the network.
With AMQP, we can use whatever AMQP-compliant client library we want and any AMQP-compliant broker. As a result, messaging clients using AMQP are completely agnostic to which AMQP client API or AMQP message broker we are using. With AMQP, the messaging model for cross platform messaging can be seen in Figure 5c below.
The advantages of having an industry-wide messaging standard become very clear when looking at the messaging model with AMQP in Figure5c. In the JMS cross platform messaging model, we have seen why we had to be concerned about protocols and what message broker vendors supported both protocols and transformations between protocols. A messaging system using AMQP as its messaging standard (regardless of the client API or message broker) can send messages back and forth to other messaging systems that use AMQP.
NOTE: References to AMQP in this article are to the 0.9.1 specification. With AMQP 1.0, some of the statements might not hold true as there are some drastic change in the specification.
- Using AMQP Message Brokers
AMQP (Advanced Message Queuing Protocol) is a standard binary wire level protocol that enables conforming client applications to communicate with conforming messaging middleware brokers. AMQP allows cross platform services/systems between different enterprises or within the enterprise to easily exchange messages between each other regardless of the message broker vendor and platform. AMQP model has the notion of 3 entities Exchanges, Bindings and Queues
The AMQP Model at a high level works like this: messages are published to exchanges, which are often compared to post offices or mailboxes. Exchanges then distribute message copies to queues using rules called bindings. Then AMQP brokers either deliver messages to consumers subscribed to queues, or consumers fetch/pull messages from queues on demand. There are various brokers that have implemented the AMQP protocol like RabbitMQ, StormMQ, Apache Qpid, Apache ActiveMQ, Apache Apollo etc. We will specifically evaluate RabbitMQ in this section
ServiceA publishes messages to the exchange which routes the message to ServiceB’s AMQP queue. In the push model, the broker delivers the messages in the queue to the subscribing consumers (consumer subscribes using basic.consume amqp method) in a round-robin fashion as shown in Figure6.
In the pull model, each consumer fetches messages using amqp basic.get method (requests to basic.get are synchronized by the broker) as shown in Figure7.
The broker maintains information about the state of the queue and messages in Erlang’s distributed database Mnesia.
- Consumers have to establish a connection to the broker and all communication happens on a channel over amqp. Each connection can be multiplexed between multiple channels.
- Consumer can send message to the same exchange to notify producer of status if required as shown in Figure8. In case of an External Service, the external service can notify the producer through a callback url.
Evaluation of Solution
Durability: Durability can be configured at exchange/queue/message level. Messages will be persisted until they are acknowledged (either auto or explicit acknowledgement).
Point-Point vs Publish-Subscribe: Supports Point-Point model as well as publish-subscribe through different exchange types (direct, fanout, topic etc).
Delivery Policies: Supports different message delivery policies
- At-most once Delivery: When using auto-acknowledgement messages are delivered at most once as a message is deleted as soon as it is delivered to the consumer
2. At-least once Delivery: When using explicit acknowledgement messages may be delivered more than once if the broker does not receive an acknowledgement from the consumer either due to network failure or consumer/broker crash. In such scenarios, messages will re re-queued resulting in delivery again with a delivered flag set to true. Consumers can also reject or nack messages which will result in re-queuing of messages for re-delivery.
Message Acknowledgements: Supports message acknowledgements between broker->producer as well as consumer->broker.
Purging Policies: Messages are removed from the queue once they are acknowledged. Messages are stored in memory (also on disk for persistent messages) and retaining for long periods of time would exhaust memory resulting in performance issues. Rejected messages and negative acknowledged messages can be requeued or forwarded to dead letter exchange which will route to appropriate queue for handling such messages.
Message Size/Format: Supports binary messages and message size is limited by the amount of memory/disk space on the broker. When a queue gets too large exhausting memory, messages published to the queue will be written to disk. Even though there is no limitation on message size, its not ideal to have messages of very large size.
Message filtering/routing Policies: Supports a variety of filtering/routing techniques through the different types of exchanges (direct, fanout, topic, headers etc). No other broker solutions offer such a wide range of routing capabilities. Figure9 demonstrates a sample model where a message is routed to two different queues from the exchange. This is the simplest of bindings; AMQP supports more complex bindings depending on the requirement of applications.
Message Ordering: Messages are delivered in FIFO order but message consumption may not be in FIFO order due to multiple consumers; unless each queue is associated with an exclusive consumer.
Batching Policies: This applies for both the push/pull model. Messages can be pushed/fetched in batches by setting a Qos prefetch count either on the channel level or consumer level. This value should be optimized based on the average round trip time (time taken for broker to send a message and receive ack) and the processing time of the message on the consumer so that the consumer can consume messages at the maximum possible rate.
Scalability: Highly Scalable in terms of throughput(million messages/sec in some cases; varies with message size) and latency is in the order of tens or hundreds of millisecs. The number of queues that can be supported would also be in the order of millions in a clustered deployment limited only by the number of concurrent socket connections that the broker can support.
Availability: Supports high availability by replicating entities(Exchanges and Bindings) to all brokers in the cluster to recover from unexpected failures. Queues have to be mirrored to a subset of brokers for making them highly available. Chooses Availability over Consistency in the event of a network partition (AP model)
Operational and Financial Cost: Involves operational and financial cost to manage servers and monitor deployments and health of the system.
- Using Kafka as Message Broker
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, with a unique design. In comparison to most messaging systems Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.
ServiceA publishes the message to a topic using the Kafka producer client library which balances the messages across the available partitions using a Partitioner. The broker to which the producer connects to takes care of sending the message to the broker which is the leader of that partition using the partition owner information in zookeeper. Instances of ServiceB use Kafka’s High-level consumer library (which handles broker leader changes, managing offset info in zookeeper and figuring out partition owner info etc implicitly) as shown in Figure10 to consume messages from partitions in streams; each stream may be mapped to a few partitions depending on how the consumer chooses to create the message streams.
For example, if there are 10 partitions for a topic and 3 consumer instances (C1,C2,C3 started in that order) all belonging to the same Consumer Group, we can have different consumption models that allow read parallelism as below
- Each consumer uses a single stream. In this model, when C1 starts all 10 partitions of the topic are mapped to the same stream and C1 starts consuming from that stream. When C2 starts, Kafka rebalances the partitions between the two streams. So, each stream will be assigned to 5 partitions(depending on the rebalance algorithm it might also be 4 vs 6) and each consumer consumes from its stream. Similarly, when C3 starts, the partitions are again rebalanced between the 3 streams. Note that in this model, when consuming from a stream assigned to more than one partition, the order of messages will be jumbled between partitions.
- Each consumer uses more than one stream (say C1 uses 3, C2 uses 3 and C3 uses 4). In this model, when C1 starts, all the 10 partitions are assigned to the 3 streams and C1 can consume from the 3 streams concurrently using multiple threads. When C2 starts, the partitions are rebalanced between the 6 streams and similarly when C3 starts, the partitions are rebalanced between the 10 streams. Each consumer can consume concurrently from multiple streams. Note that the number of streams and partitions here are equal. In case the number of streams exceed the partitions, some streams will not get any messages as they will not be assigned any partitions. Consumers can send message to another topic in to notify producer of status if required as shown in Figure11 below.
Evaluation of Solution:
Durability: Messages are always persisted to disk and the retention time can be configured. Note that in order to improve performance, Kafka buffers messages upto a certain limit or a certain time before flushing the page cache to disk. In the event of a broker crash during this period, messages may not be persisted to disk.
Point-Point vs Publish-Subscribe: Supports Point-Point model (many consumers per group) as well as publish-subscribe (one consumer per group) through a single abstraction Consumer Group.
Delivery Policies: Supports different message delivery policies
- a) At-most once Delivery: This can happen when a consumer receives a message and updates its offset (in zookeeper) before processing the message and the consumer crashes before processing the message.
- At-least once Delivery: This can happen when a consumer receives a message, processes the message but crashes before updating its offset. As a result the same messages are consumed by a different consumer belonging to the same consumer group.
Message Acknowledgements: Message acknowledgements between broker->producer are supported by setting a configuration property;
No acknowledgements between broker->consumer. The guarantee that kafka offers is all messages before a given offset for a partition will have been already consumed.
Purging Policies: Messages are written to log files on disk and the retention period for the log files can be configured using a property on the broker.
Message Size/Format: Supports binary/text messages and message size can be limited by configuring a property on the broker which limits the message size that can be sent by the producer. On the consumer side, there is another property that controls the number of bytes of messages to attempt to fetch in one request to the broker. These two properties have to be in sync so that a producer cannot send a large message which cannot be fetched by consumer. Even though its possible to set the message size to a very high value, its not ideal to use send and receive large messages. Most message broker solutions are not designed specifically to handle large messages.
Message filtering/routing Policies: Does not support filtering/routing policies. Filtering has to be done in the MessageHandler component in the consuming Service. If a message has to be routed to multiple consuming services, the producer has to explicitly send it to the corresponding topics for those services as shown below.
Message Ordering: Messages are delivered in FIFO order within the partition; however the overall order across the partitions is not FIFO. If there is only one partition per topic, then Kafka guarantees FIFO order.
Batching Policies: Messages can be batched by the producer(until a certain threshold or until a certain interval ) while sending messages to the broker through configuration properties. Kafka has an AsyncProducer that offers the capability not only to do the sends on a separate thread, but to batch multiple messages together into a single send. Both characteristics are generally desirable–isolating network I/O from the threads doing computation and reducing the number of network messages into a smaller number of larger sends. Consumers also can fetch messages in batches configured by a property.
Scalability: Highly Scalable in terms of throughput(million writes/sec, order of hundreds of MB/sec depending on message size) and latency is in the order of tens or hundreds of millisecs. The number of topics that can be supported will be in the order of a few thousands. This is limited due to the fact that the configuration information of brokers, topic-partitions, consumer offsets, partition-owners is stored in zookeeper znodes and zookeeper being a in-memory non-sharded data store, we will eventually exhaust the memory.
The possible limitation on kafka can occur due to the number of open file handles the native OS on the broker supports. For each broker, there will be a open file descriptor per partition for the segment file (since each kafka partition is maintained as a directory with several segment files) and for each socket connection (multiple producers and one consumer for each partition in the broker)
Availability: Supports high availability by replicating partitions to other brokers in the cluster based on a replication factor, rebalancing the partitions to other brokers in case of broker crashes and by maintain ISR’s for each partition. Chooses Availability over Consistency in the event of a network partition (AP model) in the latest stable release; with future releases supporting a configuration for choosing consistency over availability or availability over consistency.
Operational and Financial Cost: Involves operational and financial cost to manage servers and monitor deployments and health of the system.
6. Using ZeroMQ Library
To be updated.
I have covered only a few message queue solutions in this article. There are a lot of other solutions like
9. Celery etc