feat(code_ast): wire tree-sitter Lua grammar - #2370
Open
hardness1020 wants to merge 1 commit into
Open
Conversation
The `lua` entry carried its seven extensions but `None` for `treesitter_info`, so `.lua` files silently fell back to separator-based splitting: functions cut mid-body, chunks starting with no signature above them. - Add `tree-sitter-lua 0.5.0` and populate the existing `Option` - Leave the name and alias list untouched, so `detect_code_language()` is unchanged by construction - Assert `treesitter_info.is_some()` in the registry test - Add two splitter tests per binding, both on samples with no blank lines: one asserts every chunk is a whole function, one that a `setmetatable` table constructor is not cut at the comma between its fields. Given blank lines the separator fallback finds the same boundaries, so such a test passes unwired. All five assertions fail if the grammar is unwired - Add the Lua docs row, drop "lua" from the fallback list, and correct that list's count: 110 entries now lack a grammar, so "over 110" is off by one - Hand-add the two root Cargo.lock entries. `cargo update --workspace` there also dedupes windows-sys, 45 lines of unrelated churn. The 11 example locks are untouched output of dev/update_rust_extern_locks.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the
tree-sitter-luagrammar tococoindex_code_astand wires it into the existinglualanguage entry, so.luafiles get syntax-aware splitting inRecursiveSplitteron par with the other grammar-backed languages. Previously Lua was registered with its seven extensions butNonefortreesitter_info, so it silently fell back to separator-based splitting: functions cut mid-body, chunks starting with no signature above them.Changes
tree-sitter-lua = "0.5.0"dependency torust/code_ast, exact-pinned like the other 35 grammar cratesTreeSitterLanguageInfoto theluaentry inprog_langs.rs; name and alias list untouched, sodetect_code_language()is unchanged by constructiontest_lua_has_treesitter(prog_langs),test_split_with_lua_languageandtest_split_lua_keeps_table_constructor_whole(recursive splitter)detect_code_languagefor.lua, plus the same two splitter tests throughRecursiveSplitterops/text.mdx, drop"lua"from the separator-fallback list, correct that list's count to 110Cargo.lockplus the 11 example locks viadev/update_rust_extern_locks.shThe existing splitter tests for other languages assert only that the chunk list is non-empty, which passes identically with a grammar unwired (separator fallback also returns chunks). The new splitter tests use samples with no blank lines, so they fail if the grammar is ever unwired: one asserts every chunk is a whole
local function ... end, one asserts asetmetatable({ x = x, y = y }, Point)constructor is not cut at the comma between its fields. Verified by unwiring the grammar and re-running.Testing
cargo test -p cocoindex_code_ast -p cocoindex_ops_text: 42 + 36 passeduv run pytest python/tests/ops/test_text.py: 37 passedruff format --check .: 283 files already formatted;ruff checkclean on the touched file