Skip to content

perf: avoid repeated scans in MarkdownHeaderSplitter - #13859

Open
jianyx1 wants to merge 1 commit into
deepset-ai:mainfrom
jianyx1:perf/markdown-header-splitter-large-documents
Open

jianyx1 wants to merge 1 commit into
deepset-ai:mainfrom
jianyx1:perf/markdown-header-splitter-large-documents

Conversation

@jianyx1

@jianyx1 jianyx1 commented Nov 22, 2026

Copy link
Copy Markdown

Related Issues

Standalone performance fix found through a local benchmark of the splitter.

Proposed Changes

When chunking a long technical manual for retrieval, MarkdownHeaderSplitter checks each candidate heading against every code fence and recounts page breaks from the beginning of the source for every chunk. Walk the fences in source order and index page-break ends once. Chunk contents, header metadata, and page-number rules are preserved.

How did you test it?

  • hatch run test:unit --no-cov -q test/components/preprocessors/test_markdown_header_splitter.py yielded 111 passes. The project's unit selection deselected one integration case.

  • Ruff and the applicable pre-commit checks passed.

  • Timed MarkdownHeaderSplitter().run([Document(content=text)]) on macOS with Python 3.13.15. The synthetic input repeats a heading, prose, a fenced shell example with a comment, and a form-feed page break. Each number is the median of three runs. The benchmark also checks chunk count, byte-exact reconstruction, and the last page number.

    Sections Before After
    1,000 0.054 s 0.005 s
    2,000 0.207 s 0.011 s
    4,000 0.810 s 0.022 s

Notes for the reviewer

Page-break end offsets preserve the behavior of a custom multi-character delimiter that straddles a chunk boundary. The added test covers that case.

Checklist

  • Read the contribution guidelines and used a conventional PR title.
  • Added a regression test and a release note.
  • Ran the applicable pre-commit checks.

This PR was generated and locally validated with Codex at the account owner's request.

@jianyx1
jianyx1 requested a review from a team as a code owner November 22, 2026 03:41
@jianyx1
jianyx1 requested review from davidsbatista and removed request for a team November 22, 2026 03:41
@vercel

vercel Bot commented Nov 22, 2026

Copy link
Copy Markdown
Contributor

@jianyx1 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 22, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants