[Bug]: JSON text fallback raises TypeError when malformed input exceeds chunk_chars #556

Description

JsonFileChunker.chunk() has an explicit text fallback for files that raise json.JSONDecodeError. With the default chunk_chars=2000, this fallback works for a 2,000-character malformed file but raises TypeError at 2,001 characters. Valid JSON above the limit still chunks successfully.

The large-file fallback calls raw_text.split("\n", keepends=True). str.split() does not accept keepends.

Expected: syntactically malformed, decodable JSON should use the existing text fallback at either size, preserving the text and meaningful line ranges without rewriting the source file. This does not request more permissive character decoding or changes to valid JSON chunking.

For applications explicitly indexing JSON through update_index_step, a newly added affected file is omitted from the derived index. If an indexed file becomes malformed, the update fails and its previous chunks remain. Other files in the batch continue to index. Repairing the file to valid JSON allows its next update to succeed.

The default daily/digest index watcher watches Markdown only. The indexing impact above was reproduced with an explicit JSON indexing Job using real local BM25 and graph components, without model calls.

Steps to reproduce

From a checkout of the commit below, install the package with its AgentScope extra, then run:

python -m pip install -e ".[as]"
python - <<'PY'
import asyncio
import tempfile
from pathlib import Path
from reme.components.file_chunker import JsonFileChunker

async def main():
    with tempfile.TemporaryDirectory() as directory:
        path = Path(directory) / "memory.json"
        path.write_text('{"note": "' + "unfinished memory entry\n" * 100, encoding="utf-8")
        await JsonFileChunker().chunk(path)

asyncio.run(main())
PY

This creates a 2,410-character UTF-8 file with an incomplete JSON string.

Relevant configuration

backend: json
encoding: utf-8
chunk_chars: 2000

Logs or traceback

reme/components/file_chunker/json_file_chunker.py, line 140, in chunk
    lines = raw_text.split("\n", keepends=True)
TypeError: 'keepends' is an invalid keyword argument for split()

Boundary checks on the unchanged upstream commit:

empty: 0 chars, 0 chunks
malformed: 2000 chars, 1 chunk
malformed: 2001 chars, TypeError
malformed multiline: 2410 chars, TypeError
valid JSON: 2511 chars, 2 chunks

The existing JSON, JSONL, plain-text, Markdown chunker and background-step suites pass: 200 passed in 1.60s. The existing test_malformed_fallback covers only a short malformed input.

ReMe version

0.4.1.13, upstream main commit 5231f3970c550bb0bc5d1bc7fbc85534274ca530, checked on 2026-09-16 UTC.

Python version

3.12.10

Operating system

macOS 27.0, arm64

Affected area

Search, catalog, graph, or index

Possible implementation and scope

Correct the line splitting in the shared malformed-JSON fallback and add focused regression coverage for the size boundary, multiline text and JSON indexing. Keep valid JSON parsing, strict encoding, source files and the default watcher configuration unchanged. No new dependency is needed.

Data safety

• I removed credentials and private memory content from this report.