Error message here!

Hide Error message here!

忘记密码?

Error message here!

请输入正确邮箱

Hide Error message here!

密码丢失?请输入您的电子邮件地址。您将收到一个重设密码链接。

Error message here!

返回登录

Close

ElasticSearch--性能优化相关的配置

IT利刃出鞘 2022-01-18 22:25:15 阅读数:137 评论数:0 点赞数:0 收藏数:0

原文网址:

简介

本文介绍ElasticSearch中性能优化有关的配置。

相关的配置有:

  1. 内存大小
  2. 批量写的数量
  3. 索引数量

内存大小

说明

ElasticSearch是个Java进程,JVM默认的最大内存是1G。一般需要增大这个配置。

配置方法

配置文件:config/jvm.options

原来是:
##-Xms4g
##-Xmx4g

改为:
-Xms4g
-Xmx4g

报错信息

如果这个值不够大,会报的错如下:

主要错误信息:

Caused by: org.springframework.data.elasticsearch.UncategorizedElasticsearchException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [991614656/945.6mb], which is larger than the limit of [986061209/940.3mb], real usage: [991390584/945.4mb], new bytes reserved: [224072/218.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=224072/218.8kb, model_inference=0/0b, accounting=11298560/10.7mb]]; nested exception is ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [991614656/945.6mb], which is larger than the limit of [986061209/940.3mb], real usage: [991390584/945.4mb], new bytes reserved: [224072/218.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=224072/218.8kb, model_inference=0/0b, accounting=11298560/10.7mb]]]

详细错误信息:

Caused by: org.springframework.data.elasticsearch.UncategorizedElasticsearchException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [991614656/945.6mb], which is larger than the limit of [986061209/940.3mb], real usage: [991390584/945.4mb], new bytes reserved: [224072/218.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=224072/218.8kb, model_inference=0/0b, accounting=11298560/10.7mb]]; nested exception is ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [991614656/945.6mb], which is larger than the limit of [986061209/940.3mb], real usage: [991390584/945.4mb], new bytes reserved: [224072/218.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=224072/218.8kb, model_inference=0/0b, accounting=11298560/10.7mb]]]
at org.springframework.data.elasticsearch.core.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:67)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.translateException(ElasticsearchRestTemplate.java:370)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:353)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doBulkOperation(ElasticsearchRestTemplate.java:238)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.bulkIndex(ElasticsearchRestTemplate.java:191)
at org.springframework.data.elasticsearch.core.DocumentOperations.bulkIndex(DocumentOperations.java:152)
at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.save(AbstractElasticsearchTemplate.java:175)
at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.save(AbstractElasticsearchTemplate.java:159)

错误的解析

Data too large, data for [] would be [991614656/945.6mb]    //A
which is larger than the limit of [986061209/940.3mb]       //B
real usage: [991390584/945.4mb]                             //C
new bytes reserved: [224072/218.8kb]                        //D

这里有4个数值:

  • A:本次将会消耗的总内存
  • B:上限,超过这个就报错。(缺省值是ES最大内存的95%)
  • C:本机上ES进程已使用的内存大小
  • D:你本次操作(或者说执行当前的任务)所需要的内存

本次操作中,C + D = A > B,所以报错了。

批量写的数量

说明

提高ES的批量写入数据的性能。

配置方法

配置文件:config/elasticsearch.yml

添加下边一行:

thread_pool.write.queue_size: 1000

报错信息

如果这个值不够大,会报429错误:

主要错误信息:

{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[elasticsearch-0][10.16.9.130:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@19a5cf02 on EsThreadPoolExecutor[name = elasticsearch-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@389297ad[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 147611]]"},"status":429}

详细错误信息:

Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:9200], URI [/service_instance_inventory/type/6_tcc-app-gateway-77b98ff6ff-crblx.cards_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[elasticsearch-0][10.16.9.130:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@19a5cf02 on EsThreadPoolExecutor[name = elasticsearch-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@389297ad[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 147611]]"},"status":429}
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch

索引数量

详见:ElasticSearch--解决this action would add [5] total shards, but this cluster currently has [1000]/[1000_IT利刃出鞘的博客-CSDN博客

说明

默认最大允许的索引数是1000个,如果业务需要大于这个数,就需要修改配置。

配置方法

法1:head工具

无需重启ES。

PUT /_cluster/settings
{
"persistent": {
"cluster": {
"max_shards_per_node":10000
}
}
}

persistent:永久生效,transient:临时生效。  

法2:CURL命令

无需重启ES。

curl --location --request PUT 'http://127.0.0.1:9200/_cluster/settings' \
--header 'Content-Type: application/json' \
--data '{"persistent":{"cluster":{"max_shards_per_node":10000}}}'

persistent:永久生效,transient:临时生效。  

报错信息

Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]

版权声明
本文为[IT利刃出鞘]所创,转载请带上原文链接,感谢
https://knife.blog.csdn.net/article/details/122404271

支付宝红包,每日可领(支付宝免费1-2元支付红包)