Broker Configs
| Property | Default | Description | | broker.id | | Each broker can be identified with a unique non-negative integer ID; This id can be used as the "name" of the broker, and its existence allows the broker to migrate to a different host/port without confusing consumers. You can choose any number you like as the ID, as long as the ID is unique. | | log.dirs | /tmp/kafka-logs | The path where kafka stores data. This path is not unique, it can be multiple, and the paths only need to be separated by commas; Whenever a new partition is created, it is chosen to do so under the path that contains the least number of partitions. | | port | 6667 | server accepts the port to which the client connects | | zookeeper.connect | null | The format of the ZooKeeper connection string is: hostname:port, where hostname and port are the host and port of a node in the ZooKeeper cluster, respectively. In order to connect to other ZooKeeper nodes when a host goes down, you can create multiple hosts as follows:
hostname1:port1, hostname2:port2, hostname3:port3.
ZooKeeper allows you to add a "chroot" path to store all kafka data in the cluster under a specific path. When multiple Kafka clusters or other applications use the same ZooKeeper cluster, you can use this method to set the data storage path. This can be implemented by formatting the connection string like this:
hostname1：port1，hostname2：port2，hostname3：port3/chroot/path
This setup stores all kafka cluster data under the /chroot/path path. Note that before you start the broker, you must create this path, and consumers must use the same connection format. | | message.max.bytes | 1000000 | The maximum size of messages that a server can receive. It is important that the settings of the consumer and producer regarding this property must be synchronized, otherwise the message published by the producer is too large for the consumer. | | num.network.threads | 3 | The number of network threads used by the server to process network requests; You generally don't need to change this property. | | num.io.threads | 8 | The number of I/O threads used by the server to process requests; The number of threads should be at least equal to the number of hard disks. | | background.threads | 4 | The number of threads used for background processing, such as file deletion; You don't need to change this property. | | queued.max.requests | 500 | The maximum number of requests that can be queued for an I/O thread to process before a network thread stops reading new requests. | | host.name | null | broker's hostname; If the hostname is already set, the broker will only be bound to this address; If there is no setting, it will bind to all interfaces and publish one copy to the ZK | | advertised.host.name | null | If set, it is sent to producers, consumers, and other brokers as the hostname of the broker | | advertised.port | null | This port is given to producers, consumers, and other brokers, and is used to establish a connection; It only needs to be set if the actual port and the port that the server needs to bind are different. | | socket.send.buffer.bytes | 100 * 1024 | SO_SNDBUFF Cache size, which the server uses to make socket connections | | socket.receive.buffer.bytes | 100 * 1024 | SO_RCVBUFF cache size, which is used by the server to connect to sockets | | socket.request.max.bytes | 100 * 1024 * 1024 | The maximum request size allowed by the server; This will avoid server overflows, which should be smaller than the Java heap size | | num.partitions | 1 | If the number of partitions is not given when creating a topic, this number will be the default number of partitions under the topic. | | log.segment.bytes | 1014*1024*1024 | The logs of the topic partition are stored in many files in a certain directory, which divide the logs of the partition into segments; This attribute is the maximum size of each file; When the dimensions reach this value, a new file is created. This setting can be overridden by the basis of each topic.
View The hyperlink login is visible. | | log.roll.hours | 24 * 7 | Even if the file does not reach log.segment.bytes, a new file is created whenever the file creation time reaches this property. This setting can also be overridden by topic-level settings;
ViewThe hyperlink login is visible. | | log.cleanup.policy | delete | | | log.retention.minutes and log.retention.hours | 7 days | The time each log file was saved before it was deleted. The default data saving time is the same for all topics.
log.retention.minutes and log.retention.bytes are both used to set the deletion of log files, regardless of which property has been overflowed.
This property setting can be overridden when the topic is basically set.
ViewThe hyperlink login is visible. | | log.retention.bytes | -1 | The total amount of data saved by each partition under each topic; Note that this is the upper limit per partition, so this number multiplied by the number of partitions is the total amount of data stored per topic. Also note: If both log.retention.hours and log.retention.bytes are set, exceeding either limit will cause a segment file to be deleted.
Note that this setting can be overridden by each topic.
ViewThe hyperlink login is visible. | | log.retention.check.interval.ms | 5 minutes | Check the interval between log segmented files to determine if the file attributes meet the deletion requirements. | | log.cleaner.enable | false | When this property is set to false, it will be deleted once the log is stored for a maximum time or size. If set to true, it will happen when the save attribute reaches the upper limitThe hyperlink login is visible.。 | | log.cleaner.threads | 1 | The number of threads performing log compression | | log.cleaner.io.max.bytes.per.second | None | The maximum number of I/Os that a log cleaner can have when performing a log compaction. This setting limits the cleaner to avoid interfering with active request services. | | log.cleaner.io.buffer.size | 500*1024*1024 | Log Cleaner indexes logs during the cleanup process and reduces the size of the cache used. It is best to set it large to provide ample memory. | | log.cleaner.io.buffer.load.factor | 512*1024 | The size of the I/O chunk required for log cleaning. You don't need to change this setting. | | log.cleaner.io.buffer.load.factor | 0.9 | load factor of the hash table used in log cleaning; You don't need to change this option. | | log.cleaner.backoff.ms | 15000 | The time interval at which the log is cleaned up is performed | | log.cleaner.min.cleanable.ratio | 0.5 | This configuration controls how often the log compactor tries to clean up the logs (assumedThe hyperlink login is visible.is open). By default, avoid cleaning more than 50% of the logs. This ratio is tied to the maximum space consumed by the backup log (50% of the logs are compressed at 50%). A higher rate means less waste and more space can be cleared more efficiently. This setting can be overridden in each topic setting.
ViewThe hyperlink login is visible.。 | | log.cleaner.delete.retention.ms | 1day | storage time; The maximum time to keep compressed logs; It is also the maximum time for the client to consume messages, and the difference between log.retention.minutes is that one controls the uncompressed data and the other controls the compressed data, which will be overwritten by the specified time when the topic is created. | | log.index.size.max.bytes | 10*1024*1024 | The maximum size per log segment. Note that if the log size reaches this value, a new log segment needs to be generated even if the size does not exceed the log.segment.bytes limit. | | log.index.interval.bytes | 4096 | When performing a fetch, you need to scan the nearest offset with a certain amount of space, the larger the setting, the better, generally use the default value | | log.flush.interval.messages | Long.MaxValue | log file "sync" to disk before accumulating messages. Because disk IO operations are a slow operation, but also a necessary means of "data reliability", check whether the time interval to cure to the hard disk is required. If this value is too large, it will lead to too long a "sync" time (IO blocking), if this value is too small, it will lead to a long time of "fsync" (IO blocking), if this value is too small, it will lead to a large number of "sync" times, which means that the overall client request has a certain delay, and the failure of the physical server will lead to the loss of messages without fsync. | | log.flush.scheduler.interval.ms | Long.MaxValue | Check if fsync intervals are required | | log.flush.interval.ms | Long.MaxValue | This number is used to control the time interval of "fsync", if the number of messages never reaches the number of messages solidified to the disk, but the time interval from the last disk synchronization reaches the threshold, disk synchronization will also be triggered. | | log.delete.delay.ms | 60000 | The retention time after the file is cleared in the index generally does not need to be modified | | auto.create.topics.enable | true | Whether to allow automatic creation of topics. If true, it will automatically create a topic that does not exist when produce or fetch does not exist. Otherwise, you need to use the command line to create a topic | | controller.socket.timeout.ms | 30000 | The timeout time of the socket when the partition management controller performs a backup. | | controller.message.queue.size | Int.MaxValue | controller-to-broker-channles | | default.replication.factor | 1 | The default number of backup copies refers only to automatically created topics | | replica.lag.time.max.ms | 10000 | If a follower does not send a fetch request within this time, the leader will reremove the follower from the ISR and consider the follower to be hung | | replica.lag.max.messages | 4000 | If a replica has more than this number of unbacked up, the leader will remove the follower and consider the follower to be hung | | replica.socket.timeout.ms | 30*1000 | leader timeout time for socket network requests when backing up data | | replica.socket.receive.buffer.bytes | 64*1024 | Socket receive buffer when sending a network request to the leader during backup | | replica.fetch.max.bytes | 1024*1024 | The maximum value of each fetch at the time of backup | | replica.fetch.min.bytes | 500 | The maximum wait time for data to reach the leader when the leader makes a backup request | | replica.fetch.min.bytes | 1 | The smallest size of the response after each fetch when backing up | | num.replica.fetchers | 1 | The number of threads that back up data from the leader | | replica.high.watermark.checkpoint.interval.ms | 5000 | Each replica checks how often the highest water level is cured | | fetch.purgatory.purge.interval.requests | 1000 | fetch request the purge interval | | producer.purgatory.purge.interval.requests | 1000 | producer requests a purge interval | | zookeeper.session.timeout.ms | 6000 | Zookeeper session timeout. | | zookeeper.connection.timeout.ms | 6000 | The maximum amount of time the client waits to establish a connection with zookeeper | | zookeeper.sync.time.ms | 2000 | zk follower lags behind zk leader for the longest time | | controlled.shutdown.enable | true | Whether it is possible to control the closure of the broker. If possible, the broker will be able to move all leaders to other brokers before closing. This reduces unavailability during the shutdown process. | | controlled.shutdown.max.retries | 3 | The number of commands that can successfully execute a shutdown before performing an incomplete shutdown. | | controlled.shutdown.retry.backoff.ms | 5000 | backoff time between shutdowns | | auto.leader.rebalance.enable | true | If this is true, the controller will automatically balance the brokers' leadership over partitions | | leader.imbalance.per.broker.percentage | 10 | The maximum imbalance ratio allowed by each broker | | leader.imbalance.check.interval.seconds | 300 | Check the frequency of leader imbalance | | offset.metadata.max.bytes | 4096 | Allows clients to save the maximum number of their offsets | | max.connections.per.ip | Int.MaxValue | The maximum number of connections per broker can be made to each IP address | | max.connections.per.ip.overrides | | The maximum coverage of the default connection per IP or hostname | | connections.max.idle.ms | 600000 | Timeout limit for empty connections | | log.roll.jitter. {ms,hours} | 0 | The maximum number of jitters abstracted from logRollTimeMillis | | num.recovery.threads.per.data.dir | 1 | The number of threads that each data directory uses to log recovery | | unclean.leader.election.enable | true | Indicates whether it is possible to use the non-replicas setting in ISR as a leader | | delete.topic.enable | false | Able to delete topics | | offsets.topic.num.partitions | 50 | The number of partitions for the offset commit topic. Since changing this after deployment is currently unsupported, we recommend using a higher setting for production (e.g., 100-200). | | offsets.topic.retention.minutes | 1440 | Offsets that have existed longer than this time limit will be marked as pending deletion | | offsets.retention.check.interval.ms | 600000 | The offset manager checks the frequency of stale offsets | | offsets.topic.replication.factor | 3 | The number of backup copies of the topic's offset. It is recommended to set higher numbers to guarantee higher availability | | offset.topic.segment.bytes | 104857600 | offsets topic. | | offsets.load.buffer.size | 5242880 | This setting is related to the batch size and is used when reading from the offsets segment. | | offsets.commit.required.acks | -1 | The number of confirmations needs to be set before the offset commit is acceptable, and generally does not need to be changed |
| Property | Default | Server Default Property | Description | | cleanup.policy | delete | log.cleanup.policy | Either "delete" or "compact"; This string indicates how to utilize the old log portion; The default method ("delete") will discard the old part when their recycling time or size limit is reached. "compact" will compress the logs | | delete.retention.ms | 86400000 (24 hours) | log.cleaner.delete.retention.ms | The difference between log.retention.minutes is that one controls uncompressed data and the other controls compressed data. This configuration can be overridden by the pinning parameters when the topic is created | | flush.messages | None | log.flush.interval.messages | This configuration specifies a time interval to force fsync logs. For example, if this option is set to 1, then fsync is required after each message, and if set to 5, fsync is required for every 5 messages. In general, it is recommended that you do not set this value. The setting of this parameter requires the necessary trade-off between "data reliability" and "performance". If this value is too large, it will cause a long time to "fsync" each time (IO blocking), and if this value is too small, it will lead to a large number of "fsync", which also means that there is a certain delay in the overall client request. Physical server failure will cause messages to be lost without fsync. | | flush.ms | None | log.flush.interval.ms | This configuration is used to pin the time interval between forcing fsync logs to disk; For example, if set to 1000, then fsync is required every 1000ms. This option is generally not recommended | | index.interval.bytes | 4096 | log.index.interval.bytes | The default setting ensures that we add an index to the message every 4096 bytes, and more indexes make the read message closer, but the index size will be increased. This option is generally not required | | max.message.bytes | 1000000 | max.message.bytes | The maximum size of the kafka append message. Note that if you increase this size, you must also increase the fetch size of your consumer so that the consumer can fetch messages to those maximum sizes. | | min.cleanable.dirty.ratio | 0.5 | min.cleanable.dirty.ratio | This configuration controls how often the log compressor attempts to purge the logs. By default, logs with a compression rate of more than 50% will be avoided. This ratio avoids the greatest waste of space | | min.insync.replicas | 1 | min.insync.replicas | When the producer is set to request.required.acks to -1, min.insync.replicas specifies the minimum number of replicas (each repica write must be successful), and if this number is not reached, the producer will produce an exception. | | retention.bytes | None | log.retention.bytes | If you use the "delete" retention policy, this configuration refers to the maximum size that the log can achieve before it is deleted. By default, there is no size limit but only a time limit | | retention.ms | 7 days | log.retention.minutes | If you use the "delete" retention policy, this configuration refers to the time when the log was saved before the deletion log. | | segment.bytes | 1GB | log.segment.bytes | In Kafka, log logs are stored in chunks, and this configuration refers to the size of log logs divided into chunks | | segment.index.bytes | 10MB | log.index.size.max.bytes | This configuration is about the size of the index file mapped between offsets and file locations; This configuration generally does not need to be modified | | segment.ms | 7 days | log.roll.hours | Even if the log chunk file does not reach the size that needs to be deleted or compressed, once the log time reaches this upper limit, a new log chunk file will be forced | | segment.jitter.ms | 0 | log.roll.jitter. {ms,hours} | The maximum jitter to subtract from logRollTimeMillis. |
Consumer Configs
| Property | Default | Description | | group.id | | A string that uniquely identifies the group in which the consumer process is located, and if the same group ID is set, it means that these processes belong to the same consumer group | | zookeeper.connect | | Specify the string of the Zookeeper connection, the format is hostname:port, here the host and port are both the host and port of the Zookeeper Server, in order to avoid losing contact after a Zookeeper machine goes down, you can specify multiple hostname:ports, using commas as separation:
hostname1：port1，hostname2：port2，hostname3：port3
You can add the ZooKeeper's chroot path to the ZooKeeper connection string, which is used to store its own data, in a way:
hostname1：port1，hostname2：port2，hostname3：port3/chroot/path | | consumer.id | null | No setup is required, and is generally generated automatically | | socket.timeout.ms | 30*100 | Timeout limits for network requests. The true timeout limit is max.fetch.wait+socket.timeout.ms | | socket.receive.buffer.bytes | 64*1024 | socket is used to receive the cache size of network requests | | fetch.message.max.bytes | 1024*1024 | The maximum number of bytes per fetch message per fetch request. These bytes will be supervised in the memory used for each partition, so this setting will control the amount of memory used by the consumer. The fetch request size must be at least equal to the maximum message size allowed by the server, otherwise the size of the message that the producer may send is larger than the size that the consumer can consume. | | num.consumer.fetchers | 1 | The number of fetcher threads used for fetch data | | auto.commit.enable | true | If true, the offset of the message fetched by the consumer will be automatically synchronized to the zookeeper. This commit offset will be used by the new consumer when the process hangs up | | auto.commit.interval.ms | 60*1000 | The frequency at which the consumer submits offset to the zookeeper is in seconds | | queued.max.message.chunks | 2 | The maximum number of messages used to cache for consumption. Each chunk must be the same as fetch.message.max.bytes | | rebalance.max.retries | 4 | When a new consumer is added to a consumer group, the collection of consumers attempts to rebalance the number of partitions allocated to each consumer. If the consumers collection changes, this rebalancing fails and reentrifies when the allocation is being executed | | fetch.min.bytes | 1 | The minimum number of bytes that the server should return with each fetch request. If not enough data is returned, the request waits until enough data is returned. | | fetch.wait.max.ms | 100 | If there is not enough data to satisfy fetch.min.bytes, this configuration refers to the maximum amount of time the server will block before responding to a fetch request. | | rebalance.backoff.ms | 2000 | backoff time before retrying reblance | | refresh.leader.backoff.ms | 200 | There is a backoff time to wait before trying to determine whether a partition's leader has lost its leadership | | auto.offset.reset | largest | If there is no initialized offset in zookeeper, if offset is a response to the following value:
smallest: Auto reset offset to smallest offset
largest: Auto reset offset to the offset of largest
anything else: throws an exception to the consumer | | consumer.timeout.ms | -1 | If no message is available, even after waiting for a specific amount of time, a timeout exception is thrown | | exclude.internal.topics | true | Whether to expose messages from internal topics to consumers | | paritition.assignment.strategy | range | Select the policy for assigning partitions to the consumer flow, optionally range, roundrobin | | client.id | group id value | is a user-specific string that helps track calls in each request. It should logically confirm the application that generated the request | | zookeeper.session.timeout.ms | 6000 | Timeout limits for zookeeper sessions. If the consumer does not send a heartbeat message to the zookeeper during this time, it is considered to be hung up and a reblance will be generated | | zookeeper.connection.timeout.ms | 6000 | The maximum wait time for a client to establish a Zookeeper connection | | zookeeper.sync.time.ms | 2000 | ZK followers can lag behind the ZK leader for a maximum time | | offsets.storage | zookeeper | Places used to store offsets: zookeeper or kafka | | offset.channel.backoff.ms | 1000 | The backoff time of reconnecting to the offsets channel or retrying the fetch/commit request of the failed offset | | offsets.channel.socket.timeout.ms | 10000 | The socket timeout limit for the response to the fetch/commit request response when reading offset. This timeout limit is used by the consumerMetadata request to request offset management | | offsets.commit.max.retries | 5 | The number of times the offset commit was retried. This retry is only applied to offset commits between shut-down. him | | dual.commit.enabled | true | If you use "kafka" as offsets.storage, you can commit offset to zookeeper twice (and once to kafka). This is a must when migrating from zookeeper-based offset storage to kafka-based offset storage. For any given consumer group, it is a safe recommendation to turn off this option when the migration is complete | | partition.assignment.strategy | range | Choose between the "range" and "roundrobin" policies as the policy for assigning partitions to consumer dataflows; The circular partition allocator allocates all available partitions as well as all available consumer threads. It will assign the partition loop to the consumer thread. If all consumer instances are subscribed to a determined, the partitions are divided into deterministic distributions. The round-robin allocation strategy is only possible if the following conditions are met: (1) Each topic has the same number of data flows per consumer strength. (2) The collection of subscribed topics is determined for each consumer instance in the consumer group. |
Producer Configs
| Property | Default | Description | | metadata.broker.list | | Serve bootstrapping. producer is only used to get metadata (topics, partitions, replicas). The socket connection to send the actual data will be established based on the returned metadata data. The format is:
host1：port1，host2：port2
This list can be a sublist of brokers or a VIP pointing to brokers | | request.required.acks | 0 | This configuration is an acknowledgment value that indicates when a produce request is considered complete. In particular, how many other brokers must have submitted data to their logs and confirmed this information to their leader. Typical values include:
0: Indicates that the producer never waits for confirmation from the broker (same behavior as 0.7). This option provides the least latency but at the same time the greatest risk (because data is lost when the server goes down).
1: Indicates that the leader replica has received the data confirmation. This option has a low latency and ensures that the server confirms that it is received.
-1: The producer gets confirmation that all synchronized replicas have received data. The latency is the largest, however, this method does not completely eliminate the risk of lost messages, because the number of synchronized replicas may be 1. If you want to ensure that some replicas receive data, then you should set the option min.insync.replicas in the topic-level settings. Read the design documentation for a more in-depth discussion. | | request.timeout.ms | 10000 | The broker tries its best to implement the request.required.acks requirement, otherwise an error will be sent to the client | | producer.type | sync | This option pins whether the message is sent asynchronously in a background thread. Correct values:
(1) async: Asynchronous sending
(2) sync: Synchronized sending
By setting the producer to asynchronous, we can process requests in batches (which is good for higher throughput), but this also creates the possibility that the client machine will lose unsent data | | serializer.class | kafka.serializer.DefaultEncoder | The serialization category of the message. The default encoder enters one byte[] and returns the same byte[] | | key.serializer.class | | Serialization class for keywords. If this is not given, the default is to match the message | | partitioner.class | kafka.producer.DefaultPartitioner | partitioner class to divide messages between subtopics. The default partitioner is based on the key's hash table | | compression.codec | none | This parameter can set the codec for compressing data, which can be selected as "none", "gzip", "snappy". | | compressed.topics | null | This parameter can be used to set whether certain topics are compressed. If the compressed codec is a codec other than NoCompressCodec, these codecs are applied to the specified topics data. If the list of compressed topics is empty, apply the specific compressed codec to all topics. If the compressed codec is NoCompressionCodec, the compression is not available for all topics. | | message.send.max.retries | 3 | This parameter will cause the producer to automatically retry failed send requests. This parameter pins the number of retries. Note: Setting a non-0 value will cause certain network errors to be repeated: cause a send and cause a loss of acknowledgment | | retry.backoff.ms | 100 | Before each retry, the producer updates the metadata of the relevant topic to see if the new leader is assigned. Because the leader selection takes a little time, this option specifies how long the producer needs to wait before updating the metadata. | | topic.metadata.refresh.interval.ms | 600*1000 | The producer generally updates the topic's metadata in some failure scenarios (partition missing, leader unavailable, etc.). He will go through a regular cycle. If you set it to a negative value, the metadata will only be updated if it fails. If set to 0, the metadata is updated after each message is sent (this option is not recommended, the system consumes too much). Important: Updates occur only after the message is sent, so if the producer never sends the message, the metadata is never updated. | | queue.buffering.max.ms | 5000 | The maximum time interval at which the user caches data when async mode is applied. For example, if the message is set to 100, messages within 100ms will be processed in batches. This will improve throughput, but increase latency due to caching. | | queue.buffering.max.messages | 10000 | When using async mode, the maximum number of unsent messages that can be cached to the queue before the producer must be blocked or data must be lost | | batch.num.messages | 200 | When using async mode, you can batch process the maximum number of messages. Or the number of messages has reached this online or queue.buffer.max.ms has arrived, and the producer will process it | | send.buffer.bytes | 100*1024 | socket write cache size | | client.id | “” | This client id is a user-specific string that is included in each request to track the call, and he should logically be able to confirm that the application made the request. |
Producer Configs
| Name | Type | Default | Importance | Description | | boostrap.servers | list | | high | Host/port group to establish a connection to the kafka cluster. Data will be loaded evenly across all servers, regardless of which server is designated for bootstrapping. This list only affects the initialized hosts (which is used to discover all servers). This list format:
host1:port1,host2:port2,...
Since these servers are only used to initialize connections to discover all the memberships of the cluster (which can change dynamically), this list does not need to contain all servers (you may want more than one server, although in this case, one server may be down). If no server appears in this list, sending data will fail until the list is available. | | acks | string | 1 | high | The producer needs a signal from the server to acknowledge receipt after receiving the data, and this configuration refers to how many such acknowledgment signals the procuder needs. This configuration actually represents the availability of data backups. The following settings are common options:
(1) acks=0: Set to 0 means that the producer does not need to wait for any confirmation of the information received. The replica will be immediately added to the socket buffer and considered sent. There is no guarantee that the server has successfully received the data in this case, and the retry of the configuration will not work (because the client does not know if it failed) and the offset of the feedback will always be set to -1.
(2) acks=1: This means that at least wait for the leader to successfully write the data to the local log, but not for all the followers to write successfully. In this case, if the follower does not successfully back up the data and the leader hangs up again, the message will be lost.
(3) acks=all: This means that the leader needs to wait for all backups to successfully write logs, and this strategy will ensure that data will not be lost as long as one backup survives. This is the strongest guarantee.
(4) Other settings, such as acks=2, are also possible, which will require a given number of acks, but this strategy is generally rarely used. | | buffer.memory | long | 33554432 | high | The producer can be used to cache the memory size of the data. If the data is generated faster than it is sent to the broker, the producer will block or throw an exception, indicated by "block.on.buffer.full".
This setting will be related to the total memory that the producer can use, but it is not a hard limit, as not all memory used by the producer is used for caching. Some additional memory is used for compression (if compression is introduced), and some is used for maintenance requests. | | compression.type | string | none | high | producer is the type of compression used to compress data. The default is uncompressed. The correct option values are none, gzip, snappy.
Compression is best used for batch processing, the more messages are processed in batches, the better the compression performance. | | retries | int | 0 | high | Setting a value greater than 0 will cause the client to resend any data once that data fails. Note that these retries are no different from those when the client receives a send error. Allows retries to potentially change the order of the data, if both message records are sent to the same partition, the first message fails, the second message appears earlier than the first message. | | batch.size | int | 16384 | medium | The producer will attempt to batch the message records to reduce the number of requests. This will improve performance between client and server. This configuration controls the default number of bytes of batch processing messages.
No attempts are made to process message bytes greater than this byte count.
Requests sent to brokers will contain multiple batches, which will contain one request for each partition.
Smaller batching values are less used and may reduce throughput (0 will only use batch processing). Larger batch values waste more memory space, so you need to allocate memory for specific batch values. | | client.id | string | | medium | This string is sent to the server when a request is made. The purpose is to be able to trace the source of requests to allow some applications outside the IP/Port allowlist to send information. This app can set any string because it has no functional purpose other than recording and tracking | | linger.ms | long | 0 | medium | The producer group will aggregate any messages that arrive between the request and the send, recording a separate batch of requests. Typically, this only happens when the record is generated faster than the sending rate. However, under certain conditions, the client will want to reduce the number of requests, or even to a moderate load. This setup will be done by adding a small delay - i.e., instead of sending a record immediately, the producer will wait for a given delay time to allow other message records to be sent, which can be batched. This can be considered a similar algorithm to TCP Nagle. This setting sets a higher latency boundary for batching: once we get the batch.size of a partition, it will send it immediately regardless of this setting, but if we get a message with a much smaller byte count than this setting, we need to "linger" a specific time to get more messages. This setting defaults to 0, i.e. no delay. Setting linger.ms=5, for example, will reduce the number of requests, but at the same time increase the delay by 5ms. | | max.request.size | int | 1028576 | medium | The maximum number of bytes requested. This is also an effective coverage for the maximum recorded size. Note: The server has its own override of message record sizes, which are different from this setting. This setting limits the number of requests that producers can send in bulk at a time to prevent a large number of requests. | | receive.buffer.bytes | int | 32768 | medium | TCP receivecache size, which is used when reading data | | send.buffer.bytes | int | 131072 | medium | TCP send cache size, which is used when sending data | | timeout.ms | int | 30000 | medium | This configuration option controls the maximum amount of time the server waits for confirmation from followers. If the number of confirmed requests is not fulfilled within this time, an error is returned. This timeout limit is measured on the server side and has no network latency including requests | | block.on.buffer.full | boolean | true | low | When our memory cache runs out, we must stop receiving new message records or throw errors. By default, this is set to true, however some blocking may not be worth expecting, so it's better to throw an error right away. This is the case when set to false: the producer throws an exception error: BufferExhaustedException if the record has been sent and the cache is full | | metadata.fetch.timeout.ms | long | 60000 | low | It refers to the first time data of some elements that we have obtained. Elements include: topic, host, partitions. This configuration refers to the time required for the element to complete successfully according to fetch, otherwise an exception will be sent to the client. | | metadata.max.age.ms | long | 300000 | low | Time in microseconds is the interval at which we force the metadata to be updated. Even if we don't see any partition leadership changes. | | metric.reporters | list | [] | low | A list of classes used to measure metrics. Implementing the MetricReporter interface will allow for the addition of classes that change as new metrics are generated. JmxReporter will always include a way to register JMX statistics | | metrics.num.samples | int | 2 | low | The number of samples used to maintain metrics | | metrics.sample.window.ms | long | 30000 | low | The Metrics system maintains a configurable number of samples in a correctable window size. This configuration configures the window size, for example. We may maintain two samples for the duration of 30s. When a window is rolled out, we erase and rewrite the oldest window | | recoonect.backoff.ms | long | 10 | low | When the connection fails, the wait time when we reconnect. This avoids repeated client reconnections | | retry.backoff.ms | long | 100 | low | The wait time before trying to retry a failed produce request. Avoid getting stuck in a send-fail dead loop.
|
|
Posted on 6/24/2021 11:47:12 AM
|
|
|
Landlord