How to Speed Up Snapshot Restore in Elasticsearch on GCP Instances

Discover effective methods to increase the speed of snapshot restoration in your Elasticsearch cluster hosted on Google Cloud Platform with practical tips based on recent updates and troubleshooting.
---
This video is based on the question https://stackoverflow.com/q/68421183/ asked by the user 'Amit' ( https://stackoverflow.com/u/8552269/ ) and on the answer https://stackoverflow.com/a/68464616/ provided by the user 'Amit' ( https://stackoverflow.com/u/8552269/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: (ES v7.5.1) Speed up snapshot restore from GCS Snapshot Repository

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Speeding Up Snapshot Restore in Elasticsearch Hosted on GCP

When working with Elasticsearch (ES) clusters, particularly on Google Cloud Platform (GCP), optimizing performance is crucial. A common challenge many users face is the slow restoration of snapshots, particularly when the infrastructure has the capability to handle much higher speeds. In this post, we’ll explore the issue of slow snapshot restoration speeds and present a successful solution that led to a significant performance boost.

The Problem: Slow Snapshot Restoration Speeds

In a recent case, an Elasticsearch cluster hosted on GCP was experiencing suboptimal snapshot restoration speeds:

Current Performance: 56 MBps per data node

Infrastructure Capability: Up to 500 MBps

Total Restoration Speed: 4.5 Gbps across the whole cluster, equating to only 10% of the available disk write throughput.

The configuration details of the infrastructure were as follows:

3 Master Nodes

10 Data Nodes (each with 8 cores and 16GB memory)

Each data instance capable of 15,000 disk IOPS with a 500 MBps throughput

Despite the robust infrastructure, the restore speed remained notably low, prompting further investigation and troubleshooting.

Initial Solutions Attempted

Multiple configurations to boost the snapshot restoration speed were tested before arriving at an effective resolution:

Setting max_restore_bytes_per_sec and indices.recovery.max_bytes_per_sec to 0

Increasing the number of data nodes from 10 to 20 to assess any throttling effects

Observations

After modifying node configurations, the restoration speed did not improve. Instead, the individual node speeds decreased by half when increasing the node count. This indicated that the limitation was not just from the individual data nodes but possibly from other systemic factors.

The Solution: Upgrade Elasticsearch Version

After multiple unsuccessful attempts to diagnose the problem, the team decided to upgrade the Elasticsearch version from 7.5.1 to 7.10.2. This decision stemmed from insights suggesting potential improvements in performance capabilities with newer releases.

Results of the Upgrade

New Snapshot Restore Speed: 33 Gbps

The restored cluster maintained similar configurations to the previous version, indicating that the upgrade itself was responsible for the performance boost.

Key Takeaways

Upgrading your Elasticsearch version can sometimes yield significant improvements in performance, especially in case of persistent issues. Here are some key takeaways from this experience:

Always Check Version Updates: Regular updates can bring enhancements and optimizations that resolve ongoing issues.

Test Different Speeds: Utilize tools like gsutil -m to establish baseline performance metrics on different nodes during troubleshooting.

Analyze Configuration Changes: If performance remains an issue, systematically change configurations to determine the best settings for your specific environment.

Work with Infrastructure Limitations: Understand the specifications and limitations of your hosted infrastructure to better tailor your configurations.

In conclusion, if you're facing similar challenges regarding snapshot restoration speeds in your Elasticsearch cluster, consider evaluating and upgrading to a more recent Elasticsearch version. The impactful results can far outweigh the initial time spent on troubleshooting prior configurations.