This article is a mirror article of machine translation, please click here to jump to the original article.

Architect_Programmer_Code Agriculture Network»Architect › Programming › Java › Spring Boot Kafka restarts to discard old data
View: 21274|Reply: 1

[Source] Spring Boot Kafka restarts to discard old data

[Copy link]
Posted on 2021-11-14 21:42:26 | | |
Requirements: When the spring boot application is launched, some configuration information will be read from the database or cache, and Kafka will be used to subscribe to configuration information changes. Since the app starts with the latest configuration information, messages about historical Kafka configuration updates will be meaningless, and if you use historical messages, you will have problems.

After the Spring Boot app restarts, there is a question about how to discard old data on a topicKafka messages for the history of that topic are not receivedThis? In fact, this problem can be solved by setting the starting offset, which allows the subscriber to set the offset of the topic to the last position before subscribing to the message.

Even if spring.kafka.consumer.auto-offset-reset is set to latest, the consumer will read the message from the flag bit where it was last acknowledged to have received the message, which will cause the consumer to read a large number of messages after restarting if the producer sends a large number of messages when the app is closed.

The consumer looks for a specific offset during initialization, with the following code:

(End)





Previous:Spring Boot dynamically sets the Topic or GroupId of Kafka
Next:Linux looks at the log output of the Systemd service

Related Posts
 Landlord| Posted on 2021-11-15 10:37:05 |
Spring Boot dynamically sets the Topic or GroupId of Kafka
https://www.itsvse.com/thread-10152-1-1.html

Under CentOS 7, Kafka often hangs up solutions
https://www.itsvse.com/thread-10090-1-1.html

Kafka's auto.offset.reset value is explained in detail
https://www.itsvse.com/thread-10017-1-1.html

The Kafka Windows system will hang up after running for a period of time
https://www.itsvse.com/thread-9984-1-1.html

Windows Kafka ERROR Failed to clean up log for __consumer_offsets
https://www.itsvse.com/thread-9980-1-1.html

Kafka configures JMX remote monitoring
https://www.itsvse.com/thread-9968-1-1.html

Detailed explanation of Kafka configuration parameters
https://www.itsvse.com/thread-9965-1-1.html

Kafka manually sets the offset offset
https://www.itsvse.com/thread-9641-1-1.html

Kafka operations for .NET/C# message queues [with source code]
https://www.itsvse.com/thread-9619-1-1.html
Disclaimer:
All software, programming materials or articles published by Code Farmer Network are only for learning and research purposes; The above content shall not be used for commercial or illegal purposes, otherwise, users shall bear all consequences. The information on this site comes from the Internet, and copyright disputes have nothing to do with this site. You must completely delete the above content from your computer within 24 hours of downloading. If you like the program, please support genuine software, purchase registration, and get better genuine services. If there is any infringement, please contact us by email.
Mail To:help@itsvse.com