This article is a mirror article of machine translation, please click here to jump to the original article.

Architect_Programmer_Code Agriculture Network»Architect › Programming › Technical chat › Under CentOS 7, Kafka often hangs up the solution ...
View: 10663|Reply: 2

Under CentOS 7, Kafka often hangs up solutions

[Copy link]
Posted on 9/22/2021 5:31:38 PM | | | |
Review:

The Kafka Windows system will hang up after running for a period of time
https://www.itsvse.com/thread-9984-1-1.html

Windows Kafka ERROR Failed to clean up log for __consumer_offsets
https://www.itsvse.com/thread-9980-1-1.html

Use the following command to check the Kafka running status. As follows:

kafka.service
   Loaded: loaded (/usr/lib/systemd/system/kafka.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2021-09-22 14:43:11 CST; 1h 43min ago
  Process: 7363 ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties (code=exited, status=1/FAILURE)
Main PID: 7363 (code=exited, status=1/FAILURE)

Sep 22 14:43:11 devops02 kafka-server-start.sh[7363]: [2021-09-22 14:43:11,295] WARN [ReplicaManager broker=1] Stopping serving replicas in dir /tmp/kafka-logs ( kafka.server.ReplicaManager)
Sep 22 14:43:11 devops02 kafka-server-start.sh[7363]: [2021-09-22 14:43:11,298] WARN [GroupCoordinator 1]: Failed to write empty metadata for group KqBatchAna: This is not the correct coordinator. (kafka.co... upCoordinator)
Sep 22 14:43:11 devops02 kafka-server-start.sh[7363]: [2021-09-22 14:43:11,303] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions HashSet(__consumer_offsets-22, __ consumer_offsets-30, ...-8, __consumer
Sep 22 14:43:11 devops02 kafka-server-start.sh[7363]: [2021-09-22 14:43:11,304] INFO [ReplicaAlterLogDirsManager on broker 1] Removed fetcher for partitions HashSet(__consumer_ offsets-22, __consumer_offsets... fsets-8, __con
Sep 22 14:43:11 devops02 kafka-server-start.sh[7363]: [2021-09-22 14:43:11,378] WARN [ReplicaManager broker=1] Broker 1 stopped fetcher for partitions __consumer_offsets-22,__ consumer_offsets-30,__consumer_... fsets-21,__con
Sep 22 14:43:11 devops02 kafka-server-start.sh[7363]: [2021-09-22 14:43:11,379] WARN Stopping serving logs in dir /tmp/kafka-logs (kafka.log.LogManager)
Sep 22 14:43:11 devops02 kafka-server-start.sh[7363]: [2021-09-22 14:43:11,386] ERROR Shutdown broker because all log dirs in /tmp/kafka-logs have failed (kafka.log.LogManager)
Sep 22 14:43:11 devops02 systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE
Sep 22 14:43:11 devops02 systemd[1]: Unit kafka.service entered failed state.
Sep 22 14:43:11 devops02 systemd[1]: kafka.service failed.
Hint: Some lines were ellipsized, use -l to show in full.



Go to the Kafka log directory /usr/local/kafka/logs to view the server.log log files, as follows:

[2021-09-22 14:43:11,286] ERROR Error while rolling log segment for __consumer_offsets-8 in dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)
java.io.FileNotFoundException: /tmp/kafka-logs/__consumer_offsets-8/00000000000000000000.index (No such file or directory)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile. <init>(RandomAccessFile.java:243)
        at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:182)
        at kafka.log.AbstractIndex.resize(AbstractIndex.scala:175)
        at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:241)
        at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:241)
        at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:507)
        at kafka.log.Log.$anonfun$roll$8(Log.scala:2037)
        at kafka.log.Log.$anonfun$roll$8$adapted(Log.scala:2037)
        at scala. Option.foreach(Option.scala:437)
        at kafka.log.Log.$anonfun$roll$2(Log.scala:2037)
        at kafka.log.Log.roll(Log.scala:2453)
        at kafka.log.Log.maybeRoll(Log.scala:1988)
        at kafka.log.Log.append(Log.scala:1263)
        at kafka.log.Log.appendAsLeader(Log.scala:1112)
        at kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1069)
        at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1057)
        at kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:958)
        at scala.collection.Iterator$$anon$9.next(Iterator.scala:575)
        at scala.collection.mutable.Growable.addAll(Growable.scala:62)
        at scala.collection.mutable.Growable.addAll$(Growable.scala:57)
        at scala.collection.immutable.MapBuilderImpl.addAll(Map.scala:692)
        at scala.collection.immutable.Map$.from(Map.scala:643)
        at scala.collection.immutable.Map$.from(Map.scala:173)
        at scala.collection.MapOps.map(Map.scala:266)
        at scala.collection.MapOps.map$(Map.scala:266)
        at scala.collection.AbstractMap.map(Map.scala:372)
        at kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:946)
        at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:616)
        at kafka.coordinator.group.GroupMetadataManager.storeGroup(GroupMetadataManager.scala:325)
        at kafka.coordinator.group.GroupCoordinator.$anonfun$onCompleteJoin$1(GroupCoordinator.scala:1206)
        at kafka.coordinator.group.GroupMetadata.inLock(GroupMetadata.scala:227)
        at kafka.coordinator.group.GroupCoordinator.onCompleteJoin(GroupCoordinator.scala:1178)
        at kafka.coordinator.group.DelayedJoin.onComplete(DelayedJoin.scala:43)
        at kafka.server.DelayedOperation.forceComplete(DelayedOperation.scala:72)
        at kafka.coordinator.group.DelayedJoin.$anonfun$tryComplete$1(DelayedJoin.scala:38)
        at kafka.coordinator.group.GroupCoordinator.$anonfun$tryCompleteJoin$1(GroupCoordinator.scala:1172)
        at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.scala:17)
        at kafka.coordinator.group.GroupMetadata.inLock(GroupMetadata.scala:227)
        at kafka.coordinator.group.GroupCoordinator.tryCompleteJoin(GroupCoordinator.scala:1171)
        at kafka.coordinator.group.DelayedJoin.tryComplete(DelayedJoin.scala:38)
        at kafka.server.DelayedOperation.safeTryCompleteOrElse(DelayedOperation.scala:110)
        at kafka.server.DelayedOperationPurgatory.tryCompleteElseWatch(DelayedOperation.scala:234)
        at kafka.coordinator.group.GroupCoordinator.prepareRebalance(GroupCoordinator.scala:1144)
        at kafka.coordinator.group.GroupCoordinator.$anonfun$maybePrepareRebalance$1(GroupCoordinator.scala:1118)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at kafka.coordinator.group.GroupMetadata.inLock(GroupMetadata.scala:227)
        at kafka.coordinator.group.GroupCoordinator.maybePrepareRebalance(GroupCoordinator.scala:1117)
        at kafka.coordinator.group.GroupCoordinator.removeMemberAndUpdateGroup(GroupCoordinator.scala:1156)
        at kafka.coordinator.group.GroupCoordinator.$anonfun$handleLeaveGroup$3(GroupCoordinator.scala:498)
        at scala.collection.immutable.List.map(List.scala:246)
        at kafka.coordinator.group.GroupCoordinator.$anonfun$handleLeaveGroup$2(GroupCoordinator.scala:470)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at kafka.coordinator.group.GroupMetadata.inLock(GroupMetadata.scala:227)
        at kafka.coordinator.group.GroupCoordinator.handleLeaveGroup(GroupCoordinator.scala:467)
        at kafka.server.KafkaApis.handleLeaveGroupRequest(KafkaApis.scala:1659)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:180)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:74)
        at java.lang.Thread.run(Thread.java:748)

Error causes:Linux regularly cleans files in the /tmp directory, the kafka file directory is stored by default/tmp/kafka-logsdirectory, causing it to be cleaned up regularly, resulting in abnormal program operation.

Under CentOS 7, there are 3 system services related to cleanup:

systemd-tmpfiles-setup.service  ：Create Volatile Files and Directories
systemd-tmpfiles-setup-dev.service：Create static device nodes in /dev
systemd-tmpfiles-clean.service ：Cleanup of Temporary Directories

There are also 3 related profiles, as follows:

/etc/tmpfiles.d/*.conf
/run/tmpfiles.d/*.conf
/usr/lib/tmpfiles.d/*.conf

Use the following command to view the logs related to tmpfiles:



tmp directory in/usr/lib/tmpfiles.d/tmp.confThe file configuration is as shown in the following figure:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

# Clear tmp directories separately, to make them easier to override
v /tmp 1777 root root 10d
v /var/tmp 1777 root root 30d

# Exclude namespace mountpoints created with PrivateTmp=yes
x /tmp/systemd-private-%b-*
X /tmp/systemd-private-%b-*/tmp
x /var/tmp/systemd-private-%b-*
X /var/tmp/systemd-private-%b-*/tmp



Solution 1

Modify Kafka's configuration file /config/server.properties to modify the log.dirs configuration, for example:



Solution 2

Add an exclusion directory and edit the file: /usr/lib/tmpfiles.d/tmp.conf


(End)




Previous:CentOS 7 installs the runtime environment with ASP.NET Core 3.1
Next:nslookup command briefly introduced

Related Posts
Posted on 9/22/2021 7:51:17 PM |
Come and learn again。。。。。
 Landlord| Posted on 2/7/2022 2:31:46 PM |
Check out the cleanup log command:


Feb 02 18:18:09 centos7-itsvse systemd[1]: Starting Cleanup of Temporary Directories...
Feb 02 18:18:09 centos7-itsvse systemd[1]: Started Cleanup of Temporary Directories.
Feb 03 18:19:09 centos7-itsvse systemd[1]: Starting Cleanup of Temporary Directories...
Feb 03 18:19:09 centos7-itsvse systemd[1]: Started Cleanup of Temporary Directories.
Feb 04 18:20:09 centos7-itsvse systemd[1]: Starting Cleanup of Temporary Directories...
Feb 04 18:20:09 centos7-itsvse systemd[1]: Started Cleanup of Temporary Directories.
Feb 05 18:21:09 centos7-itsvse systemd[1]: Starting Cleanup of Temporary Directories...
Feb 05 18:21:09 centos7-itsvse systemd[1]: Started Cleanup of Temporary Directories.
Feb 06 18:22:09 centos7-itsvse systemd[1]: Starting Cleanup of Temporary Directories...
Feb 06 18:22:09 centos7-itsvse systemd[1]: Started Cleanup of Temporary Directories.
Disclaimer:
All software, programming materials or articles published by Code Farmer Network are only for learning and research purposes; The above content shall not be used for commercial or illegal purposes, otherwise, users shall bear all consequences. The information on this site comes from the Internet, and copyright disputes have nothing to do with this site. You must completely delete the above content from your computer within 24 hours of downloading. If you like the program, please support genuine software, purchase registration, and get better genuine services. If there is any infringement, please contact us by email.
Mail To:help@itsvse.com