Skip to content

fix: improve content chunking logic - #376

Merged
justlevine merged 1 commit into
mainfrom
fix/chunk-size
Nov 3, 2026
Merged

justlevine merged 1 commit into
mainfrom
fix/chunk-size

Conversation

@Kallyan01

@Kallyan01 Kallyan01 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

What

Fixes the handling of chunk sizing to use the full json-encoded size for measurement. If the chunk is too big, it's rechunked by the loop.

Why

Related:

How

AI Disclosure

Testing Instructions

Screenshots

Additional Info

Co-authored-by: @justlevine

Checklist

  • I have read the Contribution Guidelines.
  • I have read the Development Guidelines.
  • I have added necessary tests to cover my changes.
  • I have updated the project documentation as needed.
  • My code has detailed inline documentation.
  • My code is tested to the best of my abilities.
  • My code passes all lints, tests, and checks.
Open WordPress Playground Preview

Copilot AI balanced review requested due to automatic review settings November 3, 2026 19:48
@Kallyan01
Kallyan01 requested a review from justlevine November 3, 2026 19:49
@Kallyan01 Kallyan01 changed the title fix: improve chunking logic for Algolia records to handle oversized c… fix: improve content chunking logic Nov 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Oversized fixed metadata can trigger excessive record allocation and exhaust indexing-worker memory, while empty posts whose records exactly fit the size limit are incorrectly rejected.

2 open issues
What changed in this PR

This PR updates Algolia record chunking to measure complete JSON-encoded records and retry with smaller content chunks when records exceed the size limit.

Changes:

  • Centralizes chunk-record construction and encoded-size measurement.
  • Rechecks record sizes after filters run and reduces chunk sizes on overflow.
  • Adds integration tests for escaped markup, oversized metadata, and filter-driven rechunking.
File Description
tests/​phpunit/​Integration/​Modules/​Search/​PostRecordTest.php Adds encoded-size and filtered-record chunking tests.
inc/​Modules/​Search/​Post_Record.php Measures full encoded records and retries oversized chunks.

Review effort: Balanced

Comment thread inc/Modules/Search/Post_Record.php Resolved
Comment thread inc/Modules/Search/Post_Record.php Resolved
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.07%. Comparing base (ae07cc0) to head (663f208).

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##               main     #376      +/-   ##
============================================
+ Coverage     90.01%   90.07%   +0.05%     
- Complexity      849      851       +2     
============================================
  Files            36       36              
  Lines          2674     2680       +6     
============================================
+ Hits           2407     2414       +7     
+ Misses          267      266       -1     
Flag Coverage Δ
unit 90.07% <100.00%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
inc/Modules/Search/Post_Record.php 95.72% <100.00%> (+0.55%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@justlevine
justlevine merged commit 6b11151 into main Nov 3, 2026
21 checks passed
@justlevine
justlevine deleted the fix/chunk-size branch November 3, 2026 19:59
@justlevine

Copy link
Copy Markdown
Collaborator

(cloned from #375

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants