What is Elasticsearch?

• Open-source Search Engine: Elasticsearch is a free and open-source tool designed for storing, searching, and analyzing various types of data.
• Handles Diverse Data Types: It works with textual, numerical, geospatial, structured, and unstructured data.
• NoSQL Database: Although it's a NoSQL database, it specializes in search capabilities and advanced data analysis rather than just data storage.
• JSON-based Storage: Data in Elasticsearch is stored in an unstructured format using JSON, making it flexible and adaptable.
• Built with Java: Developed in the Java programming language, Elasticsearch uses the powerful Lucene library under the hood.
• RESTful APIs: It comes with built-in RESTful APIs to easily send and receive requests for data operations.

Why use Elasticsearch?

• Efficient Full-Text Search: Elasticsearch is excellent for performing full-text searches on large datasets. Unlike relational or non-relational databases, it provides a faster and more efficient way to search and retrieve results based on relevance.
• What is Full-Text Search? Full-text search looks for matches not only in titles but also in the content of documents. It scores each result based on relevance. For example:
  • If you search for "science," it will return documents that contain the word "science," similar words, or closely related terms.
• High Performance and Scalability: Elasticsearch is designed with a distributed and scalable architecture, enabling quick searches across massive datasets.
• Support for Multiple Data Types: It can handle various types of data, such as:
  • Textual data (e.g., articles, descriptions).
  • Numerical data (e.g., metrics, statistics).
  • Geospatial data (e.g., locations, maps).
  • Structured and unstructured data.
• Aggregation Requests: Elasticsearch provides features to analyze data, spot trends, and uncover patterns, making it great for business insights.
• Handles Typos: It accounts for small errors or typos in search queries, ensuring relevant results are still returned.

Elasticsearch Architecture Explained with Example

Elasticsearch is a distributed system designed for storing, searching, and analyzing large volumes of data efficiently.

1. Cluster
• A cluster is a collection of one or more nodes that work together as a single system.
• Nodes in each cluster are assigned to one or various roles, e.g., holding data.

---

2. Node
• A node is an individual server in a cluster
• Each node has its unique ID and name
• Each node has a specific role
• node belongs to only one cluster.

---

3. Index
• An index is a collection of documents that have similar characteristics.
• Each index is identified by a name.
Dish 1

            {
              "_id": "1",
              "name": "Pizza",
              "price": 12.99,
              "category": "Entree",
              "ingredients": [
                "dough",
                "sauce",
                "cheese"
              ]
            }
                

Dish 2

            {
              "_id": "2",
              "name": "Spaghetti",
              "price": 10.99,
              "category": "Entree",
              "ingredients": [
                "eggs",
                "bacon",
                "parmesan cheese",
                "pepper"
              ]
            }
                

Actor 1

            {
              "_id": "1",
              "name": "leonardo dicaprio",
              "age": 48,
              "nationality": "American",
              "Occupations": [
                "actor",
                "film producer"
              ],
              "height": 1.83
            }
                

Actor 2

            {
              "_id": "2",
              "name": "John Smith",
              "age": 54,
              "nationality": "American",
              "Occupations": [
                "actor",
                "rapper",
                "film producer"
              ],
              "height": 1.88
            }
                

---

4. Document
The document is a JSON format object stored in the Elasticsearch index under a unique ID.
Example : Pizza Document

{
  "_id": "1",
  "name": "Pizza",
  "price": 12.99,
  "category": "Emtree",
  "ingredients": [
    "dough",
    "sauce",
    "cheese",
    "toppings"
  ]
}
            

---

5. Shards
• Sharding is the process of dividing an index into smaller parts (shards) and distributing them across nodes in the cluster.
• This enables Elasticsearch to scale horizontally by adding more nodes to the cluster and allows for parallel processing of search requests across multiple shards.
• Each shard contains a subset of the index’s documents and is stored on a single node in the cluster.
• When we create an index, it is created with one shard by default, but we can configure it to have multiple shards that are distributed across different nodes.
• Improved performance: By distributing data across multiple shards, search and indexing operations can be parallelized, resulting in improved performance.
• Increased capacity: Sharding allows Elasticsearch to scale horizontally, allowing it to handle larger volumes of data.
• Improved flexibility: Sharding allows Elasticsearch to distribute data across different hardware or infrastructure, providing flexibility in terms of resource allocation.

---

6. Replicas
Replicas are copies of shards, ensuring high availability and fault tolerance.
• Replicas ensure data is accessible even if a node fails.
• They also improve search performance by balancing requests across replicas.

---

7. Query and Analysis Layer

Elasticsearch provides a powerful query language for searching and analyzing data in real time.
Example: A librarian wants to find all books published in 2023 with "Elasticsearch" in the title. Elasticsearch quickly retrieves and ranks the results based on relevance.

Creating (indexing) a document

The create operation in Elasticsearch is commonly known as indexing a document, and it refers to adding a new document to an index.
Index a document in Elasticsearch:
• Indexing a document with a specified ID
• Indexing a document with an autogenerated ID
Indexing a Document with a specified ID
Using python script

          from elasticsearch import Elasticsearch

          # Connect to Elasticsearch
          es = Elasticsearch("http://localhost:9200")
          
          # Movie index name
          index_name = "movies"
          
          # Movie document to index
          movie_doc = {
              "title": "Inception",
              "director": "Christopher Nolan",
              "year": 2010,
              "genre": ["Sci-Fi", "Thriller"],
              "rating": 8.8,
              "cast": [
                  {"name": "Leonardo DiCaprio", "role": "Dom Cobb"},
                  {"name": "Joseph Gordon-Levitt", "role": "Arthur"},
                  {"name": "Ellen Page", "role": "Ariadne"}
              ]
          }
          
          # Index the document
          response = es.index(index=index_name, id=1, document=movie_doc)
          
          # Print response
          print(response)
          
        

Elasticsearch Console : Using HTTP request

          PUT /movies/_doc/1
          {
            "title": "Inception",
            "director": "Christopher Nolan",
            "year": 2010,
            "genre": ["Sci-Fi", "Thriller"],
            "rating": 8.8,
            "cast": [
              {
                "name": "Leonardo DiCaprio",
                "role": "Dom Cobb"
              },
              {
                "name": "Joseph Gordon-Levitt",
                "role": "Arthur"
              },
              {
                "name": "Ellen Page",
                "role": "Ariadne"
              }
            ]
          }
          
        

Indexing a document with an autogenerated ID
When no id is provided, Elasticsearch generates a unique _id for the document
Using python script

          from elasticsearch import Elasticsearch

          # Connect to Elasticsearch
          es = Elasticsearch("http://localhost:9200")
          
          # Movie index name
          index_name = "movies"
          
          # Movie document to index
          movie_doc = {
              "title": "Interstellar",
              "director": "Christopher Nolan",
              "year": 2014,
              "genre": ["Sci-Fi", "Adventure"],
              "rating": 8.6,
              "cast": [
                  {"name": "Matthew McConaughey", "role": "Cooper"},
                  {"name": "Anne Hathaway", "role": "Brand"},
                  {"name": "Jessica Chastain", "role": "Murph"}
              ]
          }
          
          # Index the document without providing an ID
          response = es.index(index=index_name, document=movie_doc)
          
          # Print the response
          print("Document Indexed:", response)
                  
        

Elasticsearch Console : Using HTTP request

          POST /movies/_doc
          {
            "title": "Interstellar",
            "director": "Christopher Nolan",
            "year": 2014,
            "genre": ["Sci-Fi", "Adventure"],
            "rating": 8.6,
            "cast": [
              {
                "name": "Matthew McConaughey",
                "role": "Cooper"
              },
              {
                "name": "Anne Hathaway",
                "role": "Brand"
              },
              {
                "name": "Jessica Chastain",
                "role": "Murph"
              }
            ]
          }
                  
        

Reading a document

You can retrieve the document using Elasticsearch's REST API with the GET method:
Using python script

          from elasticsearch import Elasticsearch

          # Connect to Elasticsearch
          es = Elasticsearch("http://localhost:9200")
          
          # Index name
          index_name = "movies"
          
          # Document ID
          document_id = "Ht5qkoUBceh9VYVhszY2"  # Replace with the actual ID of your document
          
          # Retrieve the document
          response = es.get(index=index_name, id=document_id)
          
          # Print the retrieved document
          print("Document Retrieved:", response["_source"])                
        

Elasticsearch Console : Using HTTP request

          GET /movies/_doc/Ht5qkoUBceh9VYVhszY2  # Replace with the actual ID of your document
        

Updating the document

To update documents, Elasticsearch provides the _update endpoint that allows us to modify specific fields in a document without reindexing the entire document.
Using python script

          from elasticsearch import Elasticsearch

          # Connect to Elasticsearch
          es = Elasticsearch("http://localhost:9200")
          
          # Index name
          index_name = "movies"
          
          # Document ID
          document_id = "Ht5qkoUBceh9VYVhszY2"
          
          # Update data (fields to modify or add)
          update_body = {
              "doc": {
                  "rating": 9.0,  # Updating the rating
                  "genre": ["Sci-Fi", "Adventure", "Drama"]  # Adding a new genre
              }
          }
          
          # Update the document
          response = es.update(index=index_name, id=document_id, body=update_body)
          
          # Print the response
          print("Update Response:", response)                  
        

Elasticsearch Console : Using HTTP request

          POST /movies/_update/Ht5qkoUBceh9VYVhszY2
          {
            "doc": {
              "rating": 9.0,  // Updating the rating
              "genre": ["Sci-Fi", "Adventure", "Drama"]  // Adding a new genre
            }
          }        
        

Deleting the document

To delete a document in Elasticsearch, we can send a DELETE request method and specify the document ID.
Using python script

          from elasticsearch import Elasticsearch

          # Connect to Elasticsearch
          es = Elasticsearch("http://localhost:9200")
          
          # Index name
          index_name = "movies"
          
          # Document ID
          document_id = "Ht5qkoUBceh9VYVhszY2"
          
          # Delete the document
          response = es.delete(index=index_name, id=document_id)
          
          # Print the response
          print("Delete Response:", response)                             
        

Elasticsearch Console : Using HTTP request

          DELETE /movies/_doc/Ht5qkoUBceh9VYVhszY2
        

Bulk CRUD operation

The bulk data format requires one action metadata line followed by the document data for create, index, or update operations. For delete, only the action metadata line is required.
Using python script

          from elasticsearch import Elasticsearch, helpers

          # Connect to Elasticsearch
          es = Elasticsearch("http://localhost:9200")
          
          # Index name
          index_name = "movies"
          
          # Bulk operations
          actions = [
              # Create operation
              {"create": {"_index": index_name, "_id": "1"}},
              {"title": "Inception", "director": "Christopher Nolan", "year": 2010, "genre": "Sci-Fi"},
              
              # Index (insert or replace) operation
              {"index": {"_index": index_name, "_id": "2"}},
              {"title": "The Matrix", "director": "The Wachowskis", "year": 1999, "genre": "Sci-Fi"},
              
              # Update operation
              {"update": {"_index": index_name, "_id": "1"}},
              {"doc": {"genre": "Science Fiction"}},  # Only update specified fields
              
              # Delete operation
              {"delete": {"_index": index_name, "_id": "2"}}
          ]
          
          # Perform the bulk operation
          response = helpers.bulk(es, actions)
          
          # Print the response
          print("Bulk operation response:", response)                                    
        

Elasticsearch Console : Using HTTP request

          POST /_bulk
  Content-Type: application/json

  { "create": { "_index": "movies", "_id": "1" } }
  { "title": "Inception", "director": "Christopher Nolan", "year": 2010, "genre": "Sci-Fi" }
  { "index": { "_index": "movies", "_id": "2" } }
  { "title": "The Matrix", "director": "The Wachowskis", "year": 1999, "genre": "Sci-Fi" }
  { "update": { "_index": "movies", "_id": "1" } }
  { "doc": { "genre": "Science Fiction" } }
  { "delete": { "_index": "movies", "_id": "2" } }
        

Dynamic Mapping

• Dynamic mapping automatically detects and indexes the structure and data types of documents.
• When you index a document without explicitly defining a mapping, Elasticsearch dynamically creates the mapping based on the document's fields and their data types.
• Dynamic mapping is helpful when you are working with unknown or rapidly changing schemas, as it allows you to index new data without prior setup.

---

How Dynamic Mapping Works
• Detects New Fields: When a new field is encountered during indexing, Elasticsearch assigns a type to the field based on the data provided.
• Adds the Field to the Mapping: The new field and its detected type are added to the index mapping.
• Field Type Guessing: The data type is inferred from the field’s content (e.g., a string becomes a text field, a number becomes an integer or float, etc.).

---

Default Data Type Mapping

Data in Document	Inferred Data Type
"text" (string)	text and keyword
123 (integer)	long or integer
12.34 (float)	float or double
true or false	boolean
2025-01-22T10:00:00Z	date
{ "key": "value" }	object
[1, 2, 3] (array)	array of numbers

---

Dynamic Mapping in Action
• Step 1: Index a Document Without Explicit Mapping

When you index a document in a new index without pre-defining a mapping, Elasticsearch applies dynamic mapping.

PUT /my_dynamic_index/_doc/1
{
  "name": "John Doe",
  "age": 30,
  "address": {
    "city": "New York",
    "zip": 10001
  },
  "tags": ["developer", "writer"],
  "created_at": "2025-01-22T10:00:00Z"
}

        

• Step 2: Elasticsearch Creates the Mapping Automatically

You can view the generated mapping by querying:

GET /my_dynamic_index/_mapping
        

Response of dynamically created mapping

          {
            "my_dynamic_index": {
              "mappings": {
                "properties": {
                  "name": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword"
                      }
                    }
                  },
                  "age": {
                    "type": "integer"
                  },
                  "address": {
                    "properties": {
                      "city": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword"
                          }
                        }
                      },
                      "zip": {
                        "type": "long"
                      }
                    }
                  },
                  "tags": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword"
                      }
                    }
                  },
                  "created_at": {
                    "type": "date"
                  }
                }
              }
            }
          }
          
        

---

Note:Automatically adding too many fields or incorrect types can impact search performance. Prevent Elasticsearch from dynamically mapping fields you don’t want to index.

Explicit Mapping

• Explicit mapping is where you define the structure, field types, and other attributes of the fields in your index before indexing documents.
• This gives you more control over how Elasticsearch handles your data.

---

Example of Explicit Mapping for a "Movies" Index

Field Name	Type	Description
title	text	Stores the movie title; supports full-text search.
genre	keyword	Stores the genre; exact match searches only.
release_date	date	Stores the release date of the movie in a standard date format.
rating	float	Stores the rating (e.g., IMDb score).
cast	nested	Stores information about the cast members (e.g., name, role).




---

Define the Mapping in Elasticsearch

      PUT /movies
      {
        "mappings": {
          "properties": {
            "title": {
              "type": "text"
            },
            "genre": {
              "type": "keyword"
            },
            "release_date": {
              "type": "date"
            },
            "rating": {
              "type": "float"
            },
            "cast": {
              "type": "nested",
              "properties": {
                "name": {
                  "type": "text"
                },
                "role": {
                  "type": "text"
                }
              }
            }
          }
        }
      }
      
    

keyword and text

In Elasticsearch, the main difference between keyword and text field types lies in how they are analyzed and used for searching. Both are used to store string data but are optimized for different use cases:

keyword Field Type:
• Purpose: Suitable for exact matches, aggregations, and sorting.
• Useful for scenarios where you need exact matching or aggregation queries like terms, term, or match.
• Searching for a tag ["sports"] or ["technology"] in a blog post.
• Stores the entire string as a single value (not tokenized).
text Field Type:
• Purpose: Suitable for full-text search..
• Good for large, unstructured data like: Blog content, Product descriptions, Comments or reviews.
• Supports search queries like match, match_phrase, and full-text relevance scoring.
• Example: Searching for "Elasticsearch tutorial" in an article's content.
Combining Both in a Single Field:

Elasticsearch allows you to use multi-fields to index a single field as both text and keyword.
This is common when you need both exact matches and full-text search on the same field

Overview

Elasticsearch provides a flexible and powerful search engine that supports a wide range of search types. These search types are commonly classified into three categories:

• Full-text queries
• Term-level queries
• Compound queries
Example: we will create a book index with the following fields:
• title: It is stored as a text type with the english analyzer.
• author: It is stored as a keyword type.
• publisher: It is stored as a keyword type.
• year: It is stored as an integer type.
Document 1

  PUT /book/_create/1
  {
    "title": "The Indexing Companion",
    "author": "Glenda Browne",
    "publisher": "Information Today, Inc.",
    "year": 2007,
    "description": "A classic novel depicting the glamorous and decadent life of the Roaring Twenties."
  }
            

Document 2

  PUT /book/_create/2
  {
    "title": "Indexing Books",
    "author": "Nancy C. Mulvany",
    "publisher": "University of Chicago Press",
    "year": 1994,
    "description": "A masterpiece of classic novel fiction, it explores themes of love, societal norms, and personal growth."
  }
          

Document 3

  PUT /book/_create/3
  {
    "title": "The Catcher in the Rye",
    "author": "J.D. Salinger",
    "publisher": "Brown and Company",
    "year": 1951,
    "description": "Explore a collection of classic tales and novels from various literary periods."
  }
            

Document 4

  PUT /book/_create/4
  {
    "title": "The Complete Book of Indexing",
    "author": "Nancy C. Mulvany",
    "publisher": "Information Today, Inc.",
    "year": 1991,
    "description": "Embark on an extraordinary journey through the pages of this book."
  }
            

1. Full-Text Queries

Designed for searching unstructured or large text fields. They analyze the input and field data before searching.

• Match Query: This is the standard query for performing full-text searches and supporting fuzzy matching, phrase, and proximity queries.
  It searches for documents that contain the specified term or terms.
  
  Syntax

  GET my_index_name/_search
  {
    "query": {
      "match": {
        "my_field_name": "my_search_query"
      }
    }
  }
          

  Query

  GET book/_search
  {
    "query": {
      "match": {
        "title": "Indexing books"
      }
    }
  }
            

  Response

  {
    "took": 19,
    "timed_out": false,
    "_shards": {
      "total": 1,
      "successful": 1,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": {
        "value": 3,
        "relation": "eq"
      },
      "max_score": 1.0998136,
      "hits": [
        {
          "_index": "book",
          "_id": "2",
          "_score": 1.0998136,
          "_source": {
            "title": "Indexing Books",
            "author": "Nancy C. Mulvany",
            "publisher": "University of Chicago Press",
            "year": 1994
          }
        },
        {
          "_index": "book",
          "_id": "4",
          "_score": 0.9238435,
          "_source": {
            "title": "The Complete Book of Indexing",
            "author": "Nancy C. Mulvany",
            "publisher": "Information Today, Inc.",
            "year": 1991
          }
        },
        {
          "_index": "book",
          "_id": "1",
          "_score": 0.37365946,
          "_source": {
            "title": "The Indexing Companion",
            "author": "Glenda Browne",
            "publisher": "Information Today, Inc.",
            "year": 2007
          }
        }
      ]
    }
  }
        

• Match Phrase Query: This query specifically looks for an exact phrase within a field, preserving the order of the terms.

  GET book/_search
  {
    "query": {
      "match_phrase": {
        "description": "classic novel"
      }
    }
  }
        

• Multi-Match Query: Searches across multiple fields.
  Example : Retrieving products with the term "novel" present in either their title or description fields

  GET book/_search
  {
    "query": {
      "multi_match": {
        "query": "novel",
        "fields": ["title", "description"]
      }
    }
  }
        

• Common Terms Query: Excludes high-frequency terms for efficiency.
  Example: { "common": { "content": { "query": "common terms in text" } } }

2. Term-Level Queries

Used for structured data or exact matches. They don’t analyze input or field data.

• Term Query: Searches for an exact value in a field.

  GET product/_search
  {
     "query": {
      "term": {
        "publisher": "Information Today, Inc."
      }
    }
  }
        

  
  Example: { "term": { "status": "active" } }
• Terms Query: Matches any value in a list.
  Example: { "terms": { "tags": ["sports", "technology"] } }
• Range Query: Finds documents within a range.
  Example: { "range": { "price": { "gte": 10, "lte": 100 } } }
• Exists Query: Finds documents where a field is present.
  Example: { "exists": { "field": "email" } }
• Prefix Query: Matches documents with fields starting with a prefix.
  Example: { "prefix": { "name": "ela" } }

3. Compound Queries

Combine multiple queries or apply logic to queries (e.g., AND, OR).

• Bool Query: Combines queries using must, should, must_not, and filter.
  Example:

  {
    "bool": {
      "must": { "match": { "content": "search" } },
      "filter": { "term": { "status": "published" } }
    }
  }

• Constant Score Query: Wraps another query and assigns a constant score to all matching documents.
  Example: { "constant_score": { "filter": { "term": { "status": "active" } } } }
• DisMax Query: Combines subqueries, returning the highest score from any subquery.
  Example:

  {
    "dis_max": {
      "queries": [
        { "match": { "title": "quick" } },
        { "match": { "description": "quick" } }
      ]
    }
  }

• Function Score Query: Modifies scores of matching documents using functions (e.g., decay, weight).
  Example:

  {
    "function_score": {
      "query": { "match": { "content": "Elasticsearch" } },
      "functions": [
        { "weight": 2, "field_value_factor": { "field": "popularity" } }
      ]
    }
  }

Quick Revision Notes

• Full-Text Queries: Use for unstructured text. Analyze input before searching (e.g., match, multi_match, common).
• Term-Level Queries: Use for exact matches or structured data (e.g., term, range, prefix, exists).
• Compound Queries: Combine queries or modify scoring (e.g., bool, constant_score, dis_max, function_score).

Some other queries:

Fuzzy Query : Match query with fuzziness

GET /book/_search
{
  "query": {
    "match": {
      "title": {
        "query": "indexig book",
        "fuzziness": "2"
      }
    }
  }
}
      

Products with the terms "smartphone" and "android" in the description field

GET /products/_search
{
  "query": {
    "match": {
      "description": {
        "query": "smartphone android",
        "operator": "and"
      }
    }
  }
}
      

"Apple" products in the $500–$1000 price range

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "price": {
              "gte": 500,
              "lte": 1000
            }
          }
        },
        {
          "term": {
            "brand": "Apple"
          }
        }
      ]
    }
  }
}
      

Products in the "Electronics" category with a price range of $500–$1000 and a minimum rating of 4.5

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "rating": {
              "gte": 4.5
            }
          }
        },
        {
          "range": {
            "price": {
              "gte": 500,
              "lte": 1000
            }
          }
        },
        {
          "term": {
            "category": "Electronics"
          }
        }
      ]
    }
  }
}
      

Products with the description field containing the exact phrase "premium ultrabook"

GET /products/_search
{
  "query": {
    "match_phrase": {
      "description": "premium ultrabook"
    }
  }
}
      

Products with the name field containing "Samsung

GET /products/_search
{
  "query": {
    "match": {
      "name": "Samsung"
    }
  }
}    

Products containing the seller field that starts with the prefix "Apple"

GET /products/_search
{
  "query": {
    "prefix": {
      "seller": "Apple"
    }
  }
}
      

Products in the "Electronics" category with a $500–$1000 price range and an approximate match for "applle air"

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "name": {
              "query": "applle air",
              "fuzziness": "2"
            }
          }
        },
        {
          "term": {
            "category": "Electronics"
          }
        },
        {
          "range": {
            "price": {
              "gte": 500,
              "lte": 1000
            }
          }
        }
      ]
    }
  }
}
      

Aggregations in Elasticsearch are used to analyze and summarize data. Here's a detailed explanation of the key terms with examples:

1. Aggregation

A framework for performing data analysis on Elasticsearch documents.

• Example: Calculate the average price of products or group users by country.

2. Bucket

Represents a group of documents based on certain criteria.

• Example: Group users by country (e.g., bucket: { "USA": 100, "India": 200 }).
• Types of Bucket Aggregations:
  • Terms: Groups documents by unique terms in a field.
  • Date Histogram: Groups documents by date intervals.

3. Metric

Used to calculate numerical values from document fields.

• Example: Average price of products, total sales, or maximum salary.
• Types of Metric Aggregations:
  • Avg: Calculates the average value of a field.
  • Sum: Calculates the total value.
  • Min/Max: Finds the minimum or maximum value.

4. Pipeline

Aggregations that take the output of other aggregations as input.

• Example: Calculate the percentage of sales from different regions based on the total sales.
• Common Pipeline Aggregations:
  • Moving Average: Calculates trends over time.
  • Bucket Script: Perform custom calculations on bucket data.

5. Compound Aggregation

Nests multiple aggregations together.

• Example: Group by country (bucket) and then calculate the average salary (metric) for each country.

6. Global Aggregation

Performs aggregations across all documents, regardless of filters.

• Example: Calculate the total number of users, ignoring country-specific filters.

Quick Revision Notes for System Design Interview:

• Aggregations are used for real-time data analysis in Elasticsearch.
• Focus on bucket (grouping) and metric (calculations).
• Popular use cases:
  • Terms Aggregation for top N categories (e.g., top 5 products by sales).
  • Date Histogram for time-series analysis (e.g., user activity per day).
  • Metrics like avg, sum, min, max for numerical insights.
• Combining buckets and metrics (compound) is key for deeper insights.
• Use pipeline aggregations for advanced scenarios like trend analysis or percentages.