Using the elasticsearch-head panel to connect to Elasticsearch(ES) I found the following error:
Cluster Health: red (24 of 31) 
To view status information, visit:The hyperlink login is visible.
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 24,
"active_shards" : 24,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 7,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 77.41935483870968
}
The head plugin will be displayed in different colors
1) Green - the healthiest state, which means that all primary and replica shards are available;
2) Yellow - all primary shards are available, but some replica shards are not available;
3) Red - Some primary shards are not available. (At this time, some data can still be found when executing the query, so it is better to solve it quickly.) ）
To view the index status, access:The hyperlink login is visible.
.monitoring-es-6-2021.01.22 0 p UNASSIGNED ALLOCATION_FAILED
Explanation in one sentence: Unallocated shards.
When starting ES, you will find that the cluster shards will appear purple, gray, and finally green after constantly refreshing through the Head plug-in.
Why does unassigned sharding occur?
If you cannot allocate shards, for example, you have overallocated the number of replica shards for the number of nodes in the cluster, the shards remain UNASSIGNED.
The error code is: ALLOCATION_FAILED.
Possible causes of unassigned sharding issues?
1) INDEX_CREATED: Not allocated due to the API that created the index.
2) CLUSTER_RECOVERED: Not allocated due to full cluster recovery.
3) INDEX_REOPENED: Not allocated due to opening open or closing an index.
4) DANGLING_INDEX_IMPORTED: Not allocated due to the result of importing the dangling index.
5) NEW_INDEX_RESTORED: Not allocated due to reverting to a new index.
6) EXISTING_INDEX_RESTORED: Not allocated due to reverting to a closed index.
7) REPLICA_ADDED: Not allocated due to explicitly adding replica shards.
8) ALLOCATION_FAILED: Not allocated due to shard allocation failure.
9) NODE_LEFT: The node carrying the shard leaves the cluster and is not allocated.
10) REINITIALIZED: Due to unallocated shards when moving from start to initialization (e.g. sharding with shadow shadow copies).
11) REROUTE_CANCELLED: Deallocate as a result of an explicit unresail command.
12) REALLOCATED_REPLICA: Determine that a better replica location is calibrated for use, resulting in the cancellation of existing copy allocation and unallocation.
Log in to the server and view the Elasticsearch (ES) logs as follows:
Elasticsearch outputs a lot of logs, all in the ES_HOME/logs directory. The default logging level is INFO. It provides moderate information, but is designed not to overwhelm your logs.
There are a large number of errors such as:
[2021-01-21T03:33:26,435] [WARN ] [o.e.x.m.e.l.LocalExporter] [A_OefhJ] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_222]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_222]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_222]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_222]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_222]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:1.8.0_222]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:1.8.0_222]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_222]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) ~[?:1.8.0_222]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:129) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:111) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:607) ~[elasticsearch-6.5.2.jar :6.5.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:414) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:409) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:901) ~[elasticsearch-6.5.2.jar :6.5.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockException(TransportReplicationAction.java:824) ~[ elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:812) ~[ elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:710) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:169) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:97) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:126) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:384) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:496) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction.executeIngestAndBulk(TransportBulkAction.java:243) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:169) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:549) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.ingest.IngestService$4.doRun(IngestService.java:417) [elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.2.jar:6.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:183) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:810) ~[ elasticsearch-6.5.2.jar:6.5.2]
... 21 more
After analysis, it was found that it was becauseThe disk space of the server where the ES is located is too low, once any index with one or more shards is assigned to a node in a disk that stores more than 95% of the data, the index will be forced into read-only mode.
View all index settings information to visit:The hyperlink login is visible., as shown in the figure below:
"knowledge" : {
"settings" : {
"index" : {
"number_of_shards" : "3",
"blocks" : {
"read_only_allow_delete" : "true"
},
"provided_name" : "knowledge",
"creation_date" : "1576206251443",
"number_of_replicas" : "0",
"uuid" : "yeSIP_otQo6JQ8ooRdr8hA",
"version" : {
"created" : "6050299"
}
}
}
}
solution
After solving the disk problem, expand or delete junk files, follow the tutorial below.
Set read_only_allow_delete to false
Get to send a put request using elasticsearch-head:The hyperlink login is visible., as shown in the figure below:
This still can't get rid of Red's health status,Delete the ".monitoring-es-6-*" data directly, the state becomes healthy, as shown in the figure below:
(End)
|
Posted on 2021-1-27 23:08:07
|
|
|
|
Landlord|
Posted on 2021-6-15 11:22:58
|
