Elasticsearch

---

name: Elasticsearch description: Query and index Elasticsearch with proper mappings, analyzers, and search patterns. metadata: {"clawdbot":{"emoji":"🔍","requires":{"anyBins":["curl"]},"os":["linux","darwin","win32"]}}

---

Mapping Mistakes

Always define explicit mappings—dynamic mapping guesses wrong (first "123" makes field integer, later "abc" fails)

text for full-text search, keyword for exact match/aggregations—using text for IDs breaks filters

Can't change field type after indexing—must reindex to new index with correct mapping

Set dynamic: "strict" to reject unmapped fields—catches typos in field names

Text vs Keyword

text is analyzed (tokenized, lowercased)—"Quick Brown" matches search for "quick"

keyword is exact bytes—"Quick Brown" only matches exactly "Quick Brown"

Need both? Use multi-field: "title": { "type": "text", "fields": { "raw": { "type": "keyword" }}}

Sort/aggregate on title.raw, search on title

Query vs Filter Context

Query context calculates relevance score—expensive, use for search ranking

Filter context is yes/no—cacheable, use for exact conditions (status, date ranges)

Combine: bool.must for scoring, bool.filter for filtering without scoring

Range queries on dates/numbers almost always belong in filter, not query

Analyzers

standard analyzer lowercases and removes punctuation—fine for most text

keyword analyzer keeps exact string—use for codes, SKUs, emails

Language analyzers (english) stem words—"running" matches "run"

Test analyzer with _analyze endpoint before indexing—surprises in production hurt

Nested vs Object

Object type flattens arrays—{"tags": [{"key":"a","val":1}, {"key":"b","val":2}]} becomes tags.key: [a,b], tags.val: [1,2]

Flattened loses association—query key=a AND val=2 incorrectly matches above

Use nested type to preserve object boundaries—requires nested query wrapper

Nested is expensive—avoid for high-cardinality arrays

Pagination Traps

from + size limited to 10,000 hits—deep pagination fails

search_after for deep pagination—requires consistent sort, typically _id

Scroll API for bulk export—keeps point-in-time view, but ties up resources

Don't use scroll for user pagination—search_after is correct choice

Bulk Operations

Never index documents one-by-one—use _bulk API, 5-15MB batches

Bulk format: newline-delimited JSON, action line then document line

Check response for partial failures—bulk can succeed overall with individual doc errors

Set refresh=false during bulk loads—refresh after batch completes

Performance

_source: false with stored_fields if you don't need full document—reduces I/O

Use filter for cacheable conditions—Elasticsearch caches filter results

Avoid leading wildcards (*term)—forces full scan; use reverse field for suffix search

profile: true shows query execution breakdown—find slow clauses

Sharding

Shard size 10-50GB optimal—too small = overhead, too large = slow recovery

Number of shards fixed at creation—can't reshard without reindexing

Replicas for read throughput and availability—set based on query load

Start with 1 shard for small indices—over-sharding kills performance

Index Management

Use index templates—new indices get consistent mappings and settings

Use aliases for zero-downtime reindexing—point alias to new index after reindex

ILM (Index Lifecycle Management) for time-series—auto-rollover, delete old indices

Close unused indices to free memory—closed index uses no heap

Aggregations

terms agg needs keyword field—text fields fail or give garbage

Default size: 10 on terms agg—increase to get all buckets, or use composite

Cardinality is approximate (HyperLogLog)—exact count requires scanning all docs

Nested aggs require nested wrapper—matches nested query pattern

Common Errors

"cluster_block_exception"—disk > 85%, cluster goes read-only; clear disk, reset with _cluster/settings

"version conflict"—concurrent update; retry with retry_on_conflict or use optimistic locking

"circuit_breaker_exception"—query uses too much memory; reduce aggregation scope

Mapping explosion from dynamic fields—set index.mapping.total_fields.limit and use strict mapping