add untyped_cst

report.md

@@ -0,0 +1,136 @@
+# `untyped_cst` 与 `handrolled_parser` 语法解析差异审查报告
+
+- 审查日期：2026-07-13
+- 更新日期：2026-07-15
+- 审查目标：验证以下关键不变量：
+
+  > 对任意输入源码，`untyped_cst` 生成的 CST 转换为 AST 后，应等同于该源码经 `handrolled_parser` 直接生成的 AST。
+
+## 结论
+
+关键不变量整体上仍未成立。本报告只保留当前尚未解决或尚未完整验证的事项：
+
+1. 存在双方都无诊断、CST 转 AST 成功，但语义 AST 不同的输入。
+2. 即使忽略语义节点差异，两套实现生成的 AST 源码位置仍存在多类不一致。
+3. 尚未完整复扫未纳入 parser runner 的 token fixtures，不能据当前回归集合宣称全输入等价。
+
+因此，当前仍不能将 `untyped_cst -> AST` 当作 `handrolled_parser` 的完全等价替代。
+
+## 审查与验证基线
+
+审查覆盖表达式、pattern、type、顶层声明、AST lowering、差分测试设施和源码位置。按照项目 `AGENTS.md` 的要求，没有读取由 `.mbty` 生成的大型 `.mbt` 文件；正式语法规则以原始 `.mbty` 文件为准。
+
+当前验证结果：
+
+| 检查 | 结果 |
+|---|---:|
+| `moon check` | 通过 |
+| `moon test untyped_cst/diagnostics_equiv_test.mbt -v` | 13/13 通过 |
+| `moon test untyped_cst -v` | 85/85 通过 |
+| `moon test test/sync_test/parser_test.mbt -v` | 967/967 通过 |
+| `moon test test/manual_test -v` | 11/11 通过 |
+| `moon info` | 无公开接口差异 |
+| `git diff --check` | 通过 |
+| 142 个非生成项目源码 | 双方均接受，发现 2 个 AST 差异 |
+| 230 个未纳入 runner 的 token fixtures | 尚未完整复扫 |
+
+## 一、双方接受但语义 AST 不同
+
+### 1. 单元素 tuple pattern 的尾逗号处理不同
+
+输入：
+
+```moonbit
+match x { (y,) => y }
+```
+
+- handrolled：`Pattern::Var(y)`
+- CST：`Pattern::Tuple([Var(y)])`
+
+相关实现：
+
+- [`handrolled_parser/parser.mbt`](handrolled_parser/parser.mbt)
+- [`untyped_cst/parse_pattern.mbt`](untyped_cst/parse_pattern.mbt)
+- [`untyped_cst/lower_pattern.mbt`](untyped_cst/lower_pattern.mbt)
+
+正式 `.mbty` 规则要求逗号后还有一个非空 pattern，见 [`lexer/tokens/parser.mbty`](lexer/tokens/parser.mbty)。更合理的修复是让双方都拒绝该输入，而不是让 CST 模仿 handrolled 的 AST。
+
+### 2. 单元素 tuple type 的尾逗号处理不同
+
+输入：
+
+```moonbit
+fn f(x : (Int,)) {}
+```
+
+- handrolled：`Type::Name(Int)`
+- CST：`Type::Tuple([Int])`
+
+相关实现：
+
+- [`handrolled_parser/parser.mbt`](handrolled_parser/parser.mbt)
+- [`untyped_cst/parse_type.mbt`](untyped_cst/parse_type.mbt)
+- [`untyped_cst/lower_type.mbt`](untyped_cst/lower_type.mbt)
+
+正式 `.mbty` 规则同样要求逗号后存在非空 type。
+
+## 二、AST 源码位置不变量仍不成立
+
+默认 AST JSON 配置隐藏位置，见 [`lexer/basic/config.mbt`](lexer/basic/config.mbt)，因此大部分 AST equality 测试不会发现位置差异。
+
+启用位置输出后，已确认的差异包括：
+
+- `fn init {}`、`fn main {}` 的空参数位置：CST 使用固定 `no_location`，handrolled 使用源文件中的空范围；
+- `pub fn f(){}`：CST 的函数位置包含 `pub`，handrolled 从 `fn` 开始；
+- `fn &T::f(){}`：owner type name 是否包含 `&` 不一致；
+- `fn f(x~ : Int){}` 和 enum labelled argument：label/binder 位置是否包含 `~` 不一致；
+- 局部命名函数：CST 使用源码位置，handrolled 使用 `no_location`；
+- 负 pattern、索引/切片、object type 等节点也存在位置范围差异。
+
+即使选择 `LocStyle::Json`，部分辅助位置字段仍没有被 serializer 输出。因此还需要直接比较 AST 位置字段或补全位置序列化，不能只依赖当前 JSON。
+
+## 三、剩余测试覆盖盲区
+
+### 1. 语义 AST runner 主要覆盖成功语料
+
+[`test/sync_test/helper_test.mbt`](test/sync_test/helper_test.mbt) 会比较 handrolled、MoonYacc 和 CST 转 AST，但测试样例主要是三者都能成功解析的 parser fixtures。
+
+[`untyped_cst/ast_equiv_test.mbt`](untyped_cst/ast_equiv_test.mbt) 也主要验证代表性成功输入，不能据此证明任意合法输入的 AST 都等价。
+
+### 2. 默认位置序列化会隐藏差异
+
+[`lexer/basic/loc.mbt`](lexer/basic/loc.mbt) 在默认配置下把 location 序列化为 `null`，使位置差异无法被常规 AST JSON equality 捕获。
+
+### 3. 230 个 token fixtures 未纳入 parser runner
+
+现有 parser runner 有 967 个 fixtures，但 token fixtures 共 1197 个。剩余 230 个主要只用于 lexer 测试，本轮尚未重新完整扫描这些输入。
+
+### 4. 测试生成器输出文件名与实际 runner 不一致
+
+[`test/sync_test/generator/generator.mbt`](test/sync_test/generator/generator.mbt) 输出 `parser.mbt`/`lexer.mbt`，而当前实际运行的是 `parser_test.mbt`/`lexer_test.mbt`，存在重新生成语料后 runner 未同步的风险。
+
+## 四、建议修复优先级
+
+### P0：语义 AST 正确性
+
+1. 按正式 `.mbty` 规则统一拒绝单元素 tuple pattern/type 的尾逗号。
+2. 继续核对 `as` 与 `is` 的解析层级及 lowering 结果。
+
+### P1：AST 位置
+
+1. 明确并测试每类 AST 节点的位置是否包含 modifier、sigil 和 soft keyword。
+2. 统一空参数、owner type、label/binder、局部函数及负 pattern 等位置范围。
+3. 补全位置 serializer，或在测试中直接比较完整 AST location 字段。
+
+### P1：扩大差分覆盖
+
+1. 完整复扫全部 token fixtures。
+2. 对双方成功的输入同时比较语义 AST 和完整位置。
+3. 修正测试生成器与 runner 的文件命名关系。
+4. 为每个剩余最小复现保留固定回归测试。
+
+## 总结
+
+当前 diagnostics、声明截断矩阵、967 个同步 parser tests 和 11 个 manual tests 均已通过。仍需解决的是合法输入的 AST 结构差异、AST 源码位置差异，以及尚未完整覆盖的 token fixtures。
+
+要真正建立目标不变量，需要继续修复上述语义和位置问题，并把差分测试扩展到全部语料。

test/manual_test/extend_soft_keyword_regression_test.mbt

@@ -26,9 +26,10 @@ test "extend soft keyword in remaining lower identifier positions" {
   }
   let hand_json = hand_impls.map(impl_ => impl_.json_repr()).to_json()
   let yacc_json = yacc_impls.map(impl_ => impl_.json_repr()).to_json()
-  if hand_json != yacc_json {
+  let cst_json = cst_impls_json(source, "extend_soft_keyword_regression.mbt")
+  if hand_json != yacc_json || hand_json != cst_json {
     fail(
-      "soft keyword extend produced different ASTs between moonyacc and handrolled parsers",
+      "soft keyword extend produced different ASTs between moonyacc, handrolled, and CST parsers",
     )
   }
 }
@@ -62,9 +63,12 @@ test "extend special constructor pattern location matches yacc" {
   let yacc_json = @basic.show_loc.protect(String, fn() {
     yacc_impls.map(impl_ => impl_.json_repr()).to_json()
   })
-  if hand_json != yacc_json {
+  let cst_json = @basic.show_loc.protect(String, () => {
+    cst_impls_json(source, "extend_special_constructor_pattern_loc.mbt")
+  })
+  if hand_json != yacc_json || hand_json != cst_json {
     fail(
-      "extend special constructor pattern location differs between moonyacc and handrolled parsers:\nhand=\{hand_json.stringify(indent=2)}\nyacc=\{yacc_json.stringify(indent=2)}",
+      "extend special constructor pattern location differs between moonyacc, handrolled, and CST parsers:\nhand=\{hand_json.stringify(indent=2)}\nyacc=\{yacc_json.stringify(indent=2)}\ncst=\{cst_json.stringify(indent=2)}",
     )
   }
 }

test/manual_test/for_condition_test.mbt

@@ -6,20 +6,28 @@ test "single semicolon for condition is accepted" {
     #|    ignore(i)
     #|  }
     #|}
-  let (_, hand_reports) = @parser.parse_string(
+  let (hand_impls, hand_reports) = @parser.parse_string(
     source,
     name="for_condition.mbt",
     parser=Handrolled,
   )
   if !hand_reports.is_empty() {
     fail("handrolled parser rejected single-semicolon for condition")
   }
-  let (_, yacc_reports) = @parser.parse_string(
+  let (yacc_impls, yacc_reports) = @parser.parse_string(
     source,
     name="for_condition.mbt",
     parser=MoonYacc,
   )
   if !yacc_reports.is_empty() {
     fail("moonyacc parser rejected single-semicolon for condition")
   }
+  let hand_json = hand_impls.map(impl_ => impl_.json_repr()).to_json()
+  let yacc_json = yacc_impls.map(impl_ => impl_.json_repr()).to_json()
+  let cst_json = cst_impls_json(source, "for_condition.mbt")
+  if hand_json != yacc_json || hand_json != cst_json {
+    fail(
+      "single-semicolon for condition produced different ASTs between moonyacc, handrolled, and CST parsers",
+    )
+  }
 }

test/manual_test/helper_test.mbt

@@ -1,6 +1,42 @@
+///|
+fn reports_from_view(view : ArrayView[@basic.Report]) -> Array[@basic.Report] {
+  let reports : Array[@basic.Report] = []
+  for report in view {
+    reports.push(report)
+  }
+  reports
+}
+
+///|
+fn cst_impls_json(source : String, name : String) -> Json raise Error {
+  let result = @untyped_cst.parse_string(source, name~, entry=Structure)
+  let diagnostics = reports_from_view(result.diagnostics_view())
+  if !diagnostics.is_empty() {
+    fail(
+      "CST parser rejected \{name}: \{diagnostics.to_json().stringify(indent=2)}",
+    )
+  }
+  let (impls, reports) = result.to_impls()
+  if !reports.is_empty() {
+    fail(
+      "converting CST for \{name} produced diagnostics: \{reports.to_json().stringify(indent=2)}",
+    )
+  }
+  impls.map(impl_ => impl_.json_repr()).to_json()
+}
+
+///|
+fn expect_cst_rejected(source : String, name : String) -> Unit raise Error {
+  let result = @untyped_cst.parse_string(source, name~, entry=Structure)
+  if result.diagnostics_view().is_empty() {
+    fail("CST parser unexpectedly accepted \{name}")
+  }
+}
+
 ///|
 /// Verifies that both the hand-rolled parser and the moonyacc parser produce identical ASTs,
-/// and that no diagnostics are generated for all test cases in `__snapshot__`.
+/// and the CST parser produce identical ASTs, and that no diagnostics are
+/// generated for all test cases in `__snapshot__`.
 fn @test.Test::run(t : Self) -> Unit raise Error {
   let name = t.name()
   let source_file = "test/manual_test/__snapshot__/\{name}.mbt"
@@ -9,6 +45,7 @@ fn @test.Test::run(t : Self) -> Unit raise Error {
   }
   let hand_result = @parser.parse_string(source, name~, parser=Handrolled)
   let yacc_result = @parser.parse_string(source, name~, parser=MoonYacc)
+  let cst_json = cst_impls_json(source, name)
   if !(hand_result.1.is_empty() && yacc_result.1.is_empty()) {
     let buf = StringBuilder::new()
     let _ : Unit = @basic.show_loc.protect(String, fn() {
@@ -24,7 +61,7 @@ fn @test.Test::run(t : Self) -> Unit raise Error {
   }
   let hand_json = hand_result.0.map(impl_ => impl_.json_repr()).to_json()
   let yacc_json = yacc_result.0.map(impl_ => impl_.json_repr()).to_json()
-  if hand_json != yacc_json {
+  if hand_json != yacc_json || hand_json != cst_json {
     @fs.create_dir(debug_dir)
     @fs.write_string_to_file("\{debug_dir}/\{name}.mbt", source)
     @fs.write_string_to_file(
@@ -35,8 +72,12 @@ fn @test.Test::run(t : Self) -> Unit raise Error {
       "\{debug_dir}/\{name}_yacc.json",
       yacc_json.stringify(indent=2),
     )
+    @fs.write_string_to_file(
+      "\{debug_dir}/\{name}_cst.json",
+      cst_json.stringify(indent=2),
+    )
     fail(
-      "parsing \{name}.mbt produced different ASTs between moonyacc and handrolled parsers.",
+      "parsing \{name}.mbt produced different ASTs between moonyacc, handrolled, and CST parsers.",
     )
   }
   let json : Json = hand_json

test/manual_test/impl_method_quantifiers_test.mbt

@@ -30,9 +30,10 @@ test "parse impl method quantifiers" {
   }
   let hand_json = hand_impls.map(impl_ => impl_.json_repr()).to_json()
   let yacc_json = yacc_impls.map(impl_ => impl_.json_repr()).to_json()
-  if hand_json != yacc_json {
+  let cst_json = cst_impls_json(source, "impl_method_quantifiers.mbt")
+  if hand_json != yacc_json || hand_json != cst_json {
     fail(
-      "impl method quantifiers produced different ASTs between moonyacc and handrolled parsers",
+      "impl method quantifiers produced different ASTs between moonyacc, handrolled, and CST parsers",
     )
   }
 }

test/manual_test/loop_removed_test.mbt

@@ -17,4 +17,5 @@ test "loop expression is rejected" {
   if yacc_reports.is_empty() {
     fail("moonyacc parser unexpectedly accepted loop expression")
   }
+  expect_cst_rejected(source, "loop_removed.mbt")
 }

test/manual_test/moon.pkg

@@ -1,6 +1,7 @@
 import {
   "moonbitlang/parser",
   "moonbitlang/parser/syntax",
+  "moonbitlang/parser/untyped_cst",
   "moonbitlang/lexer/basic",
   "moonbitlang/x/fs",
   "moonbitlang/core/test",

test/manual_test/outdated_syntax_removed_test.mbt

@@ -8,18 +8,35 @@ fn expect_rejected(source : String, name : String) -> Unit raise Error {
   if yacc_reports.is_empty() {
     fail("moonyacc parser unexpectedly accepted \{name}")
   }
+  expect_cst_rejected(source, name)
 }
 
 ///|
 fn expect_accepted(source : String, name : String) -> Unit raise Error {
-  let (_, hand_reports) = @parser.parse_string(source, name~, parser=Handrolled)
+  let (hand_impls, hand_reports) = @parser.parse_string(
+    source,
+    name~,
+    parser=Handrolled,
+  )
   if !hand_reports.is_empty() {
     fail("handrolled parser unexpectedly rejected \{name}")
   }
-  let (_, yacc_reports) = @parser.parse_string(source, name~, parser=MoonYacc)
+  let (yacc_impls, yacc_reports) = @parser.parse_string(
+    source,
+    name~,
+    parser=MoonYacc,
+  )
   if !yacc_reports.is_empty() {
     fail("moonyacc parser unexpectedly rejected \{name}")
   }
+  let hand_json = hand_impls.map(impl_ => impl_.json_repr()).to_json()
+  let yacc_json = yacc_impls.map(impl_ => impl_.json_repr()).to_json()
+  let cst_json = cst_impls_json(source, name)
+  if hand_json != yacc_json || hand_json != cst_json {
+    fail(
+      "parsing \{name} produced different ASTs between moonyacc, handrolled, and CST parsers",
+    )
+  }
 }
 
 ///|

test/manual_test/template_object_rhs_test.mbt

@@ -27,9 +27,10 @@ test "parse template object rhs" {
   }
   let yacc_json = yacc_impls.map(impl_ => impl_.json_repr()).to_json()
   let hand_json = hand_impls.map(impl_ => impl_.json_repr()).to_json()
-  if yacc_json != hand_json {
+  let cst_json = cst_impls_json(source, "template_object_rhs.mbt")
+  if yacc_json != hand_json || hand_json != cst_json {
     fail(
-      "template object rhs produced different ASTs between moonyacc and handrolled parsers",
+      "template object rhs produced different ASTs between moonyacc, handrolled, and CST parsers",
     )
   }
 }