Repository navigation
fix: halve publish-time memory for any OpenClaw release without per-release tables - #129
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs maintainer review before merge. What this changesThis PR replaces release-specific SDK qualification tables with declaration certificates and reduces allocations during plugin inspection. Example: Inspect @openclaw/discord against OpenClaw 2026.10.10 on Linux Node 24.
Review scores
ProductKind: Performance · Worth it: Yes · Fix scope: Complete Merge readiness✅ Ready for maintainer review Keep open: the memory improvements remain distinct from main, and the previous correctness findings are addressed. No blocking code findings remain; the current head’s memory gate is still running. Priority: P2 Before mergeNone. FindingsNone. Tests
Agent review detailsHow this fits togetherPlugin Inspector reads packed OpenClaw declarations and plugin source, determines which replacement APIs exist, and produces compatibility findings consumed by publishers and ClawHub. flowchart TD
A[Packed OpenClaw declarations] --> B[Declaration certificates]
B --> C[Exact checker for undecided names]
B --> D[Qualified retirement records]
C --> D
E[Plugin source] --> F[Runtime import analysis]
D --> G[Compatibility report]
F --> G
Technical reviewBest possible solution: Land the certificate and allocation improvements after the existing checks pass, preserving exact fallback answers and stable report contracts. Do we have a high-confidence way to reproduce the issue? Main still invokes the reduced checker for unknown declaration snapshots, and the linked downstream report establishes actual publish failures on 0.4.5; this read-only review ran no target code. Is this the best way to solve the issue? The general certificate path removes per-release maintenance while retaining the existing checker as the authority for uncertain names; the latest shared traversal state fixes the reported cycles without suppressing inference upgrades. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 81ccff0f2c4f. Provenance checked
TestingProof path: shipped entry point. SecurityNone. EvidenceWhat I checked:
Review metrics
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment HistoryReview history (3 earlier review cycles)
Reviewed November 4, 2026, 5:46 AM ET / 10:46 UTC (Revision 4). |
Remove the shipped per-version qualification table. Decide each replacement name from the packed declarations with certificates that need no TypeScript program: available answers are exact, unavailable answers are sound, and any undecided name still goes to the reduced type check. Treat ambient modules without export declarations as export contexts in the reducer as well. Gate CI at 50% of 0.4.5 for the published target, an unseen snapshot and the esbuild-bundled build, and verify certificates against the full type check.
Run the negative certificate pass once (deeply wherever the snapshot can infer), and stop the positive walk once every candidate is proven. Parse runtime-import analysis ASTs with nodes that compute ranges on demand and carry parent and resolved-reference slots instead of a range array, a Map entry and a Set entry per node; the analysis is unchanged.
Store statement spans in typed arrays and single table entries directly.
A large file's syntax tree and scope graph are partly promoted while they are analyzed; dead old-space objects then keep the young remainder alive through the remembered set, so later scavenges promote it too and V8 grows the young generation. Clear the links once both runtime-import passes are done, run the rule pass after them, and key certifier walks by number instead of long concatenated strings. Results are unchanged.
Add a generator for future OpenClaw packages with new SDK entrypoints and declaration shapes (implicit exports, minified re-exports, default exports, declaration merging, generic and conditional aliases, destructured bindings, global augmentation, typeof, namespace exports) and an optional computed-name variant that takes the exact fallback. Gate the future release at 50% of 0.4.5 in CI and verify certificates on it. Certificates now follow typeof, global declarations, export default of an identifier and module-namespace exports, ignore destructuring-annotation member names, and no longer walk a class's instance type from its value (prototype has no declaration, so the type check never does).
Runtime-import classification needs eslint-scope's answers for a few names (require callees, createRequire handles, namespace imports, callback parameters, declarator ids). Drive eslint-scope's unchanged Referencer with a scope manager that tracks declarations and references only for those names, resolving exactly as Scope#__close does; files with direct eval or with, or deep trees, still use the full ScopeManager. A large bundle no longer builds a multi-megabyte scope graph that is promoted during analysis, which made V8 grow the young generation and RSS vary by ~60 MB between runs. Certificates now enumerate template literals over finite unions and certify mapped-type keys built from them, so computed future names are exact. Gate the computed-names future release at 50%, run an undecidable variant through the exact fallback with identical outputs, and verify scope facts on the plugin packages in CI.
84ca602 to
613f878
Compare
|
Maintainer revalidation at current head Both correctness findings from the earlier review still reproduce against this revision:
The latest Publish memory run also fails the bundled Discord gate: 270.4 MiB / 536.1 MiB = 0.504, above the 0.5 limit. The gate must remain intact. Please resolve the effective-member and export-cycle cases with differential regressions, then validate the unchanged memory gate and current-revision Crabpot source smoke. No release is authorized. |
…declaration files Node's type stripper keeps ~25-50 MB of WebAssembly memory for the rest of the process. JavaScript whose tokens contain nothing the TypeScript grammar reads differently strips to itself, so decide that from the acorn parse the runtime-import analysis already needs (and reuse that tree). Validated on 31,915 sources: every accepted one strips to the identical text. Names matched out of packed declaration files were sliced strings that kept whole multi-megabyte files alive with the target surface.
|
Maintainer fixes pushed at
Validation: |
readFile(path, "utf8") decodes a large file chunk by chunk into a concatenated string that V8 flattens into a copy on first use. For the 2.7 MB packed declaration chunk that is ~10 MB of young objects that survive collections, which pushed V8 to double the young generation in some runs (bundled discord grew in 3 of 6 local runs; 0 of 8 after). Decoding the whole buffer gives the identical text.
|
@clawsweeper re-review Current head |
|
🦞👀 Re-review progress:
|
|
@clawsweeper re-review Fixed the negative-walk cycle finding in All 934 local tests and package contents checks pass. Crabpot source-mode report remains identical to main excluding generatedAt. The previous head passed the full hosted memory gate; the unchanged gate must now validate this fix too. |
|
🦞👀 Re-review progress:
|
Summary
Publish-time inspection of discord, matrix and whatsapp peaks at ≤50% of 0.4.5 on Linux Node 24, both unbundled and esbuild-bundled like ClawHub's Convex actions. This holds for OpenClaw 2026.10.10, for an unseen snapshot, and for synthetic future releases with new SDK surface, including replacement names computed by template literals.
One case is not within 50% by design, and it needs a decision (see "Trade-off"): replacement names whose availability depends on TypeScript's assignability, which only a type check can decide.
What changed
Retirement qualification without a type checker (
src/packed-sdk-certificates.js). Each SDK replacement name is decided from the packed declarations:export * as, orexport default <identifier>). Or it is an own member of a type the public walk reaches through steps with unambiguous TypeScript semantics:any/never;typeofof classes (statics only), functions, enums, variables and bindings;asclauses built from templates.${string}),intrinsic, tuples,lengthandprototypeblock negative certificates.Scope facts without a full scope graph (
src/scope-facts.js). Runtime-import classification only asks eslint-scope about a few names:requirecallees,createRequireimports and their handles, namespace import locals, callback parameters and declarator ids. This PR drives eslint-scope's own, unchangedReferencer, so which identifiers are references or definitions, in which scope, with which flags and in which order, stays eslint-scope's. The scope manager behind it only keeps declarations and references for those names, and resolves the wayScope#__closedoes:arguments;Files with direct
eval,with, or nesting deeper than 500 levels still use the fullScopeManager. A throw still yields the same fallback.For the 351 KB discord bundle this removes a ~6 MB scope graph that used to live, and be promoted, during analysis. That was what made V8 double the young generation in some runs: the bimodal bundled discord result of 258 vs 315 MiB in run 41797514068 attempt 2.
Lifetimes. Each file's analysis is released after its last use. The rule pass runs after the runtime-import passes. Member-chain checks no longer allocate. The certifier parses 8 KB pieces once each, without caching. Its tables are compact.
No type stripper for plain JavaScript (
src/plain-javascript.js). Runtime-import classification ran every file through Node'sstripTypeScriptTypes, which loads swc as WebAssembly and keeps ~25–50 MB of linear memory for the rest of the process. Strip mode only blanks TypeScript syntax, and JavaScript can only contain TypeScript syntax where the TypeScript grammar reads the same tokens differently. The check refuses each such place from the tokens of the acorn parse the analysis needs anyway (and that tree is then reused):</<<that closes with>over tokens that can occur in types (f<T>(x),f<g<T>>(x),a<b>before a line break, tagged templates,new);cond ? (a): T => b,case (a): b => c;declare/namespace/global/type/… at a statement start followed by a word, literal or brace (swc acceptsdeclare\nconst x = 1across the line break);!after a line break that could be a non-null assertion, HTML-like comments, and a byte order mark (swc drops it).Anything refused, and every TypeScript file, is stripped exactly as before. Sources swc rejects as invalid (
with, legacy octals, top-levelreturn) give the same result either way, and transform mode rejects them identically.Names copied out of declaration files.
apiRegistrars,hookNames,manifestFieldsandmanifestContractFieldswere V8 slices of the packed.d.tsfiles, so the target surface kept a 5.2 MB and a 0.9 MB declaration file alive through inspection.Files decoded in one piece (
src/read-text.js).readFile(path, "utf8")decodes a large file chunk by chunk into a concatenated string that V8 flattens into a copy on first use: the 2.7 MB packed declaration chunk cost ~10 MB of young objects that survived collections. Decoding the whole buffer gives the identical text as one string. This is what moves discord away from the young-generation cliff (below).Bug fix in merged fix: halve publish-time inspection memory for packed OpenClaw targets #128 code. Export-context
.d.tsfiles are now kept whole by the reducer.Linux CI (
ubuntu-latest, Node 24.22.1, npm tarballs, 0.4.5 in the same job)Run 41813458081 on
8de7260(4061d3don top only moves shared helpers; its run is 41814332643). Peak RSS in MiB, median (range). Ratio is the candidate median over the 0.4.5 median; the worst-run column divides the candidate's highest run by the 0.4.5 median.Every gated row compares target-surface and findings hashes with 0.4.5, and every row is identical.
Equivalence evidence
collectRuntimeModuleImportsagainst main's pure-eslint-scope implementation, all three option sets, output order included: 25,882 unique files (300.7 MB) from ClawHub'snode_modulesand the global npm modules, plus 4,893 files (43 MB) from the plugin packages, the inspector's dependencies and the 0.4.3/0.4.5 baselines. Together that's 92,325 comparisons, 0 mismatches.scripts/verify-scope-facts.mjscompares the facts themselves (each requested variable's identifiers, definition count, references, read/write/init/static flags, and declared-variable lookups) with the fullScopeManager: 4,099 files, 0 differences. CI runs it on the plugin packages.require, defaults vs body declarations, block/catch/for/switch/class scopes, class heritage and fields, named function expressions,createRequirehandles, callbacks, hoisting, destructuring, exports and labels: 2,676 sources, 8,028 comparisons, 0 mismatches.test/scope-facts.test.jscovers 13 cases.bench-v2-style validation over ClawHub'snode_modules, the global npm modules, the plugin packages, the baselines and the inspector's dependencies): 31,915 unique sources (348.6 MB); 27,012 accepted (275.3 MB), and every accepted source strips to the identical text (one is rejected by swc as invalid, which gives the same result). The validation found one real gap, a byte order mark, which is now refused.test/plain-javascript.test.jschecks that each JavaScript/TypeScript ambiguity swc changes is refused and that accepted samples strip to themselves.scripts/verify-packed-sdk-certificates.mjs, every name spelled in a snapshot): 18 published releases plus the future fixtures, about 650,000 names, 0 disagreements.test/packed-sdk-certificates.test.jshas 14 tests, including future-shaped and computed snapshots, and an end-to-end check that retirement records equal filtering by the full type check, with and without the fallback.npm run check: 929 tests pass.Future fixtures
scripts/make-future-openclaw-fixture.mjsderives future packages from the fetched OpenClaw.Plain: a new
./plugin-sdk/future-runtimeentry with no export declarations, a./plugin-sdk/future-defaultentry with a default export, and new chunks with minified re-exports. Through them it exposes:declare globalinterface andtypeofusage.0.4.5's full check finds 21 retirement records instead of 18, and the certificates match with nothing undecided.
--computed-names: adds{ [K in Verbs as \${K}Async`]: … }`. The names are enumerated exactly; gated at 50%.--undecidable: addsT extends { kind: K } ? … : …over object types. This needs assignability, so it takes the exact fallback.Trade-off that needs a decision: exact answers for type-dependent names
The undecidable fixture keeps outputs identical to 0.4.5, but only by type-checking the SDK closure, which costs 74–75% of 0.4.5 on Linux (78–83% on
fa700ce, before files were decoded in one piece). This is not a tuning gap:The options:
Remaining limits
8de7260, bundled discord grew in 3 of 6 local runs (273 → ~295 MiB; macOS, same Node) and unbundled discord in 1 of 6, even though no CI row grew. After decoding files in one piece, it grew in 0 of 8 local runs, as did matrix and whatsapp (8 each bundled, 6 each unbundled). In CI, no candidate row's young generation went past 64 MiB. Unbundled discord now reaches the 32 MiB semi-space only during inspection, with ~11 MB of survivals after it against the ~32 MB that would trigger the next doubling.ClawHub
No npm publish, deploy or consumer bump. After a release, ClawHub can unpin from 0.4.3 and bump, and update #4362's test, which expects all 50 unfiltered records.