Rewrite the explanation of chunking strategies

[SKHDev195]

Summary

Rewrites and expands the "Configuring chunking" section of explore-analyze/elastic-inference/inference-api.md to make chunking strategies easier to understand and choose between.

Changes:

• Reframes why chunking happens into two clear situations set as visually distinct bullet points.
• Adds a quick-reference table comparing all strategies with "How it works", "When to use", and "When not to use" columns.
• Adds a collapsible worked example (max_chunk_size: 20) to each strategy showing exactly how sample text is split.
• Fixes the ICU4J documentation links that pointed to a page with several outdated URLs.

Generative AI disclosure

1. Did you use a generative AI (GenAI) tool to assist in creating this contribution?

• Yes
• No

2. If you answered "Yes" to the previous question, please specify the tool(s) and model(s) used.

Tool(s) and model(s) used: Claude Code (Claude Opus 5)

[timgrein]

Maybe this just looks weird in the markdown diff, but shouldn't this read from left to right instead of top to bottom and then left to right?

[timgrein]

EOL comments would help to understand what these regexes do, otherwise it's a bit hard to understand IMO

[leemthompo]

could use code callouts for this

[timgrein]

Adding two findings by Claude

[timgrein]

I checked for correctness together with Claude against the codebase and found:

Actual behavior: RecursiveChunker.mergeChunkOffsetsUpToMaxChunkSize() greedily merges adjacent chunks when their combined word count fits
  within max_chunk_size. After splitting on \n\n:
  - Chunk 2 (10 words) + Chunk 3 (3 words) = 13 words ≤ 20 → they are merged.

  Real result with max_chunk_size: 20:
  Chunk 1: Elasticsearch stores data in indices. Each index is divided into shards.  (11 words)
  Chunk 2: [separator]\nShards are distributed across nodes. [...] Replicas provide redundancy.  (13 words)

  This is confirmed by the test testChunkInputRequiresOneSplitWithMerges in RecursiveChunkerTests.java:61, where 3 sentences of 10 words each
  and max_chunk_size=20 produces only 2 chunks (first two merged).

[timgrein]

Another finding by Claude:

The PR shows 3 chunks, one per ## Heading. Actual behavior:

 The separators for MARKDOWN (SeparatorGroup.java:30-39) are \n# , \n## , etc. Splitting on \n##  produces:
 - # Elasticsearch\n → 1 word
 - \n## Storage\n\n...shards.\n\n → 1 ("Storage") + 11 = 12 words
 - \n## Distribution\n\n...scaling.\n\n → 1 + 10 = 11 words
 - \n## Redundancy\n\nReplicas provide redundancy. → 1 + 3 = 4 words

 Greedy merge with max_chunk_size: 20:
 - 1 + 12 = 13 ≤ 20 → merge into Chunk 1 (13 words): contains # Elasticsearch + ## Storage section
 - 13 + 11 = 24 > 20 → emit Chunk 1, start new
 - 11 + 4 = 15 ≤ 20 → merge into Chunk 2 (15 words): contains ## Distribution + ## Redundancy sections

 Real result:
 Chunk 1: # Elasticsearch\n\n## Storage\n\nElasticsearch stores data in indices. Each index is divided into shards.  (13 words)
 Chunk 2: \n## Distribution\n\n...scaling.\n\n## Redundancy\n\nReplicas provide redundancy.  (15 words)

[github-actions]

Elastic Docs AI PR menu

Check the box to run an AI review for this pull request.

• Review docs changes (docs-review). Status: not started.

Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team.