Pagination of results in Elasticsearch | from & size, scroll, search after [ES7 for Beginners #4.4]

In this elastic search tutorial, we discuss about Paginating the search results or search result Pagination. This is part of Query DSL (Domain Specific Language)

Sections:
Simple Pagination using from and size (0:00)
Scaling problem with from and size pagination (3:00)
Elasticsearch Pagination Scroll Request (5:05)
Scaling problem with scroll pagination request (9:00)
Search after pagination in Elasticsearch (9:55)
Things to note about Elasticsearch search after pagination API (12:32)

References: https://www.elastic.co/guide/en/elast...

Playlist Link:    • What is Elastic Search  and ELK Stack? | E...  

Hashtags:
#coding #theory #computerscience #elasticsearch #clusters #distributedSystems #tutorial #logstash #kibana #beats #aws #dataScience #pagination #queryDSL

Places where these ideas can be used:
To paginate the search results using ElasticSearch
To build infinite scroll using ElasticSearch

Some Theory:

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

To paginate through a larger set of results, you can use the search API’s size and from parameters. The size parameter is the number of matching documents to return. The from parameter is a zero-indexed offset from the beginning of the complete result set that indicates the document you want to start with.

While a search request returns a single “page” of results, the scroll API can be used to retrieve large numbers of results (or even all results) from a single search request, in much the same way as you would use a cursor on a traditional database.

Pagination of results can be done by using the from and size but the cost becomes prohibitive when the deep pagination is reached. The index.max_result_window which defaults to 10,000 is a safeguard, search requests take heap memory and time proportional to from + size. The scroll API is recommended for efficient deep scrolling but scroll contexts are costly and it is not recommended to use it for real time user requests. The search_after parameter circumvents this problem by providing a live cursor. The idea is to use the results from the previous page to help the retrieval of the next page.

(This tutorial is part of a series of tutorials on Elasticsearch, logstash and Kibana. It uses docker for purpose of installation, and may even use aws in the future.)