Error message here!

Hide Error message here!

忘记密码?

Error message here!

请输入正确邮箱

Hide Error message here!

密码丢失?请输入您的电子邮件地址。您将收到一个重设密码链接。

Error message here!

返回登录

Close

Elasticsearch cluster upgrade guidelines

Secretary Xiangzi 2021-04-08 22:38:33 阅读数:2 评论数:0 点赞数:0 收藏数:0

Catalog

  • background
  • The first part Version upgrade guidelines
  • The second part Upgrade methods and specific steps
  • summary
  • References and materials

background

Elasticsearch The version upgrade of a cluster is an important work of cluster maintenance . This article refers to the official documents , Details will be given .

The first part Version upgrade guidelines

1.1 Synchronous upgrade Elastic Stack Components

about Elasticsearch We need to upgrade our ecosystem components at the same time , The specific supporting version can refer to the upgrade guide provided by the official .

https://www.elastic.co/cn/products/upgrade_guide

1.2 Index compatibility

Elasticsearch For the old version of the index (index) Compatibility is as follows :

  • Elasticsearch 6.x compatible Elasticsearch 5.x Index created in , But not compatible Elasticsearch 2.x Or an older version of the index .
  • Elasticsearch 5.x compatible Elasticsearch 2.x Index created in , But not compatible Elasticsearch1.x Or an older version of the index .

If index incompatibility is encountered during upgrade , After upgrading, the cluster will not start normally .

1.3 Version upgrade route

Elasticsearch The specific route of version upgrade is summarized as follows :

Serial number Original version Upgrade the target version Supported upgrade types
1 5.x 5.y Rolling upgrade ( among y > x
2 5.6 6.x Rolling upgrade
3 5.0-5.5 6.x Cluster restart
4 <5.x 6.x reindex upgrade
5 6.x 6.y Rolling upgrade ( among y > x)
6 1.x 5.x reindex upgrade
7 2.x 2.y Rolling upgrade ( among y > x
8 2.x 5.x Cluster restart
9 5.0.0 pre GA 5.x Cluster restart
10 5.x 5.y Rolling upgrade ( among y > x

About Elasticsearch The version sequence of needs special explanation .Elasticsearch The version sequence is not continuously increasing , from 2.4.x After the release, jump straight to 5.0.x. So for 5.x edition , If the numbers are incremented in strict order , Should be 3.x. There is no serial number , Mainly to keep ELK(Elasticsearch 、 Logstash 、 Kibana) The unity of the whole version .

Among them the first 4 In this case , Less than 5.x In fact, that is 2.x and 1.x. because 6.x Incompatible with lower version indexes , So we need to implement the index in the original cluster reindex. The schemes are as follows :

1.3.1 2.x Upgrade to 6.x

According to the above upgrading route, there are two upgrading schemes :

  • programme 1: First of all 2.x Upgrade to 5.6 edition (reindex Upgrade index version ), Then from 5.6 Upgrade to 6.x( Rolling upgrade );
  • programme 2: Create a new 6.x colony , Then the index data in the old cluster is remotely reindex Into a new cluster ;

1.3.2 1.x Upgrade to 6.x

There are also two options :

  • programme 1: First of all 1.x Upgrade to 2.4.x edition (reindex Upgrade index version ), Finally, follow the above 2.x Upgrade to 6.x The implementation of the plan ;
  • programme 2: Create a new 6.x colony , Then the index in the old cluster reindex Into a new cluster ;

The second part Upgrade methods and specific steps

Cluster upgrade route , Upgrade between different versions , There are three upgrade options : Rolling upgrade 、 Cluster restart 、reindex. The following will be introduced separately .

2.1 Rolling upgrade

The so-called rolling upgrade means that the nodes in the cluster upgrade the version to the target one by one ( high ) edition , During the period of service interruption, the cluster will not be upgraded . This kind of upgrade scheme is aimed at the upgrade within the same large version , namely x.y Upgrade to x.z(z>y). Special ,5.6 Upgrade to 6.x It also supports rolling upgrade .

https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html

Generally, the steps of rolling upgrade are as follows :

The first 1 Step Disable copy fragmentation (shards) Distribute

Before the next upgrade node is down , It is necessary to prohibit the distribution of replica fragmentation in advance .

After the node goes down , The replica allocation process will wait index.unassigned.node_left.delayed_timeout( By default 1 minute ), Then start copying the shards on that node to other nodes in the cluster , This can lead to a lot of I/O. Because the node will restart soon , So there's no need to redistribute .

API The order is as follows :

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "primaries"
}
}

The first 2 Step Perform a synchronous refresh

When restarting the cluster, if translog Too big , Log playback takes a long time to recover data , It is recommended to refresh manually , Reduce translog.

Be careful : The process is slow .

POST _flush/synced

The first 3 Step Stop machine learning homework

If a machine learning task is running in the cluster , The task needs to be stopped .

Reference resources :https://www.elastic.co/guide/en/elastic-stack-overview/6.8/stopping-ml.html

The first 4 Ministry Down the node to be upgraded and install the main version and plug-ins

Implement the next down of the upgrade node , Start the file system upgrade .

The first 5 Step Start node

Start node , And use the following API Check whether the node joins the cluster .

GET _cat/nodes

The first 6 Step Restart partition allocation

After the nodes join the cluster , Set enable partition allocation to start using this node .

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}

Before upgrading to the next node , Wait for cluster fragmentation to complete . You can go through the API Check cluster status :

GET _cat/health?v

The state of the waiting cluster is determined by red become yellow, Until then green. It shows that the cluster completes the allocation of all primary and secondary partitions .

The first 7 Step Repeatedly upgrade other nodes

Repeatedly scroll to upgrade other nodes in the cluster .

The first 8 Step Restart the machine learning task

If there are machine learning tasks in the cluster , It needs to be rebooted .

2.2 Overall cluster restart

The overall restart of the cluster refers to the shutdown of all nodes of the cluster before upgrading , The cluster stops serving the outside world , After all nodes are upgraded , Start the cluster as a whole , Restore external services . for example :5.6 The previous version was upgraded to 6.x You need to restart the cluster to upgrade .

https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html

The steps of cluster restart and upgrade are similar to that of scrolling , The main steps are as follows :

The first 1 Step Disable copy fragmentation (shards) Distribute

Before upgrading the node, you need to , Prohibit the distribution of replica fragmentation in advance .( Refer to rolling upgrade )

The first 2 Step Stop unnecessary indexes and perform a synchronous refresh

Refer to rolling upgrade .

The first 3 Step Stop machine learning homework

Refer to rolling upgrade

The first 4 Ministry Take down all nodes and install the main version and plug-ins

Implement the down load for all nodes in the cluster , Start file system version upgrade .

The first 5 Step Start the node and wait for the cluster state to be yellow

Start all nodes , And use the following API Check whether all nodes join the cluster .

GET _cat/nodes

The first 6 Step Restart partition allocation

After the nodes join the cluster , Set enable partition allocation to start using this node .

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}

Before upgrading to the next node , Wait for cluster fragmentation to complete . You can go through the API Check cluster status :

GET _cat/health?v

The state of the waiting cluster is determined by yellow Turn into green. It shows that the cluster completes the allocation of all primary and secondary partitions .

The first 7 Step Restart the machine learning task

Refer to rolling upgrade

2.3 reindex

Elasticsearch Adjacent versions of index With compatibility , But larger versions are no longer downward compatible . Above (1.2 Index compatibility ) It has been introduced in . And in the ElasticSearch in , Indexed field Settings cannot be modified , If you want to modify one field, Then we should follow the new mapping, Build a index, Then query the data in batches , Reuse bulk api Write new index in .

When querying in batch , The proposal USES scroll api, And adopt the way of multithreading concurrency reindex data , Every time scroll Query a piece of data on a specified date , Just give it to a thread .

The first 1 Step Build a new version cluster

Apply for server resources , Build a new version of ElasticSearch colony . Direct all external services to the new cluster .

The first 2 Step Put the data in the old cluster reindex To the new cluster

Use... On old clusters reindex API Put the old cluster index Historical data are gradually migrated to new clusters .

If the cluster has a large amount of data , Migration is a slow process .

API Case study ( Here's a simple configuration ):

POST _reindex
{
"source": {
"remote": {
"host": "http://otherhost:9200",
"username": "user",
"password": "pass"
},
"index": "source",
"query": {
"match": {
"test": "data"
}
}
},
"dest": {
"index": "dest"
}
}
//host For remote clustering ( New clusters ) The address of .
//username and password Key verification for secure clusters .
//"index": "source" For the old cluster index name ,dest Corresponding to the new cluster target index name .

After migration , You can clean up the data in the old cluster . After cleaning up, according to the needs of the situation , Old nodes can upgrade the file system offline , Finally, join the new cluster as a new node .

If the historical data in the old cluster is not important , After deleting the data , Build a new cluster .

2.4 Step by step upgrade

For larger version upgrades , If we don't build a new cluster, we can implement it again reindex The way , Then you need to upgrade step by step . for example A、B、C There are three versions in turn , Version level A<B<C, among index data B compatible A,C compatible B, however C Are not compatible A. This situation needs to be upgraded step by step :

  • A Upgrade to B, Use the method of rolling upgrade or overall cluster restart .
  • about B Version of cluster , take A All versions of the data reindex To B edition . This process is time-consuming .
  • Wait until all the history in the cluster index( New index Naturally B edition ) Are all B After version , Upgrade the cluster version to C edition .

If index Data is time series data , Can not be implemented reindex, Wait until the end of the historical data lifecycle ( There are no more in the cluster A Version of index data ), Again from B Version upgrade to C edition .

summary

(1) commonly Elasticsearch The upgrade between large versions needs to restart the whole cluster .

(2) part ElasticSearch Between big versions index Not compatible , You need to re index the data (reindex).

(3) Small version upgrade in big version , Usually, you just need to scroll to restart .

Reference material

1、Elasticsearch Official website link :https://www.elastic.co/cn/

More attention :

Copyright statement
In this paper,the author:[Secretary Xiangzi],Reprint please bring the original link, thank you

编程之旅,人生之路,不止于编程,还有诗和远方。
阅代码原理,看框架知识,学企业实践;
赏诗词,读日记,踏人生之路,观世界之行;