Add documentation ('''...''') as sugar over the doc.doc tagged value - #5008
Add documentation ('''...''') as sugar over the doc.doc tagged value#5008rafaelbey wants to merge 5 commits into
Conversation
Ports legend-pure fb29e8f5929 (#1299). A '''...''' literal immediately preceding a declaration desugars, at parse time, into a meta::pure::profiles::doc doc tagged value, so a .pure file legend-pure accepts now parses here too. There is no new profile, no new metamodel and no protocol change: desugaring happens in DomainParseTreeWalker, and everything downstream - compiler, PureModelContextData JSON, every existing doc consumer - sees an ordinary TaggedValue. Documentation is a parser rule listed explicitly at each declaration that accepts it (class, property, qualified property, association, enum, enum value, function), not a lexer-level construct. That is what keeps the identical literal in expression position an ordinary multi-line string, and it is why attachment needs no adjacency heuristic. The engine folds text blocks into a single STRING token - deliberately, since that is what let 4feb08b avoid touching 275 STRING references across ~50 grammars - so the block shape is checked in the walker, which also yields a better message than an ANTLR mismatch. Canonicalization reuses processTextBlock(raw, false), which is already exactly legend-pure's MultilineTextLayout, and adds only the removal of leading and trailing blank lines. There is no escape processing: documentation is prose, and unescaping it would silently rewrite a regex, a Markdown escape and a Windows path. An element may not carry both documentation and an explicit doc.doc. Because tag references are unresolved at parse time, the conflict check matches the profile as written - bare doc or fully qualified - while the composer promotes both spellings. The composer converts unconditionally, so an existing {doc.doc = '...'} reformats into a block on save. That is intentional. Documentation has no escape mechanism at all, so isRenderableAsDocumentation is the guard and anything it rejects stays a tagged value: an embedded ''', a carriage return, trailing whitespace on a line, a leading or trailing blank line, an empty value, two doc tags on one element, and a line beginning ### (the section split runs before any grammar and is not string-aware). The regenerated XSD fixtures exercise this at scale - of ~1000 FpML/RDU/Xetra doc strings, 37 fall back, all of them empty values from empty <documentation/> elements. The doc tagged value's source range is computed in the walker rather than through ParseTreeWalkerSourceInformation. A token reports only the line it starts on, and the shared helper derives the end column from the start column plus the whole token length - wrong for any token whose text spans lines. Fixing it there, as legend-pure did, regresses the section index: island and section content tokens end with a newline rather than after one, which yields an end column of 0. A documentation literal always spans lines and always ends on ''', so it has no such edge case. The general multi-line end coordinate stays wrong for CString and island tokens; that is pre-existing, not introduced here. Left out, and documented as gaps rather than half-supported: Profile, Measure and native function (the grammar gives them no annotation slots, Profile has no taggedValues field in the protocol, and nativeFunction has no walker), and ###Relational and the other DSLs, which define their own taggedValues rules. Note the latter is better than legend-pure, where a '''...''' relational tagged value parses and then NPEs. Documents all of this in docs/engineering/architecture/type-system.md.
Extends the previous commit from ###Pure to the ten other element declarations that carry stereotypes and tagged values: Database, Schema, Table, column and View in ###Relational; DataSpace; Service and HostedService; FunctionJar; DataQualityValidation and DataQualityRelationValidation; SnowflakeApp and SnowflakeM2MUdf; BigQueryFunction; MemSqlFunction; and Data. The rule is now simply that wherever an element accepts tagged values, it accepts documentation. Rather than paste the construct into ten grammars and ten walkers, this moves it to the places they already share. The `documentation` rule moves from DomainParserGrammar into M3ParserGrammar, so the eight grammars that import it inherit the rule and only name it in their element rule; RelationalParserGrammar and DataParserGrammar import CoreParserGrammar and carry their own stereotypes/taggedValues, so they declare a local copy. The walker side - the text-block check, the doc.doc conflict check, canonicalization and the source range - moves into PureGrammarParserUtility.taggedValuesWithDocumentation, which every walker now calls and DomainParseTreeWalker no longer duplicates. It takes the literal's token rather than the context because ANTLR generates a distinct DocumentationContext class per grammar even for an inherited rule. The composer helpers were already public statics that every DSL composer depends on for renderAnnotations. Seven of these DSLs already have a documentation or description field of their own - Service, HostedService and FunctionJar have documentation; DataSpace, Snowflake, BigQuery and MemSql have description. A block does not populate those. It is the doc tagged value in every grammar, so the same syntax means the same thing everywhere, the DSL-specific field stays a separate slot, and an element may carry both. Relational is the one place where the block is not at column 0, so the composer emits it at the element's own indentation, which is also what pins the indentation the parser strips back off - Schema at two spaces, Table at four, a column at six. This makes the engine accept documentation in ###Relational, which legend-pure does not: there a '''...''' relational tagged value parses and then NPEs in RelationalGraphBuilder.visitTaggedValueNew.
Workflow Telemetry - Build CI / BuildWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Build
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785531363000, 1785531368000
Checkout repo : 1785531368000, 1785531375000
Cache Maven dependencies : 1785531375000, 1785531386000
Set up JDK : 1785531386000, 1785531397000
Check Java version : 1785531397000, 1785531397000
Configure git : 1785531397000, 1785531397000
Download deps and plugins : 1785531397000, 1785531420000
Restore legend-pure snapshot : done, 1785531420000, 1785531420000
Set pure version override : 1785531420000, 1785531420000
Collect Workflow Telemetry : 1785531420000, 1785531420000
Set profiles that required github secrets : 1785531420000, 1785531420000
Purge stale legend-engine artifacts from local Maven repo : 1785531420000, 1785531420000
Build (PR) : 1785531420000, 1785537595000
Build (Master + Docker Snapshot) : done, 1785537595000, 1785537595000
Upload build output : 1785537595000, 1785538176000
Upload m2 repository : 1785538176000, 1785538201000
Purge engine artifacts before cache save : 1785538201000, 1785538201000
Set Test Matrix : 1785538201000, 1785538201000
Upload CI Event : 1785538201000, 1785538202000
|
Workflow Telemetry - Build CI / Test Module - sqlserverWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - sqlserver
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538233000, 1785538234000
Checkout repo : 1785538234000, 1785538236000
Before Test : 1785538236000, 1785538374000
Restore legend-pure snapshot : done, 1785538374000, 1785538374000
Included Modules : 1785538374000, 1785538374000
Test : 1785538374000, 1785538494000
Upload Test Results : 1785538494000, 1785538495000
|
Workflow Telemetry - Build CI / Test Module - postgresWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - postgres
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538226000, 1785538227000
Checkout repo : 1785538227000, 1785538232000
Before Test : 1785538232000, 1785538376000
Restore legend-pure snapshot : done, 1785538376000, 1785538376000
Included Modules : 1785538376000, 1785538376000
Test : 1785538376000, 1785538520000
Upload Test Results : 1785538520000, 1785538521000
|
Workflow Telemetry - Build CI / Test Module - spannerWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - spanner
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538227000, 1785538229000
Checkout repo : 1785538229000, 1785538233000
Before Test : 1785538233000, 1785538392000
Restore legend-pure snapshot : done, 1785538392000, 1785538392000
Included Modules : 1785538392000, 1785538392000
Test : 1785538392000, 1785538536000
Upload Test Results : 1785538536000, 1785538537000
|
Workflow Telemetry - Build CI / Test Module - graphqlWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - graphql
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538233000, 1785538235000
Checkout repo : 1785538235000, 1785538239000
Before Test : 1785538239000, 1785538438000
Restore legend-pure snapshot : done, 1785538438000, 1785538438000
Included Modules : 1785538438000, 1785538438000
Test : 1785538438000, 1785538544000
Upload Test Results : 1785538544000, 1785538546000
|
Workflow Telemetry - Build CI / Test Module - clickhouseWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - clickhouse
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538227000, 1785538228000
Checkout repo : 1785538228000, 1785538234000
Before Test : 1785538234000, 1785538415000
Restore legend-pure snapshot : done, 1785538415000, 1785538415000
Included Modules : 1785538415000, 1785538415000
Test : 1785538415000, 1785538547000
Upload Test Results : 1785538547000, 1785538548000
|
Workflow Telemetry - Build CI / Test Module - duckdbWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - duckdb
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538229000, 1785538230000
Checkout repo : 1785538230000, 1785538233000
Before Test : 1785538233000, 1785538415000
Restore legend-pure snapshot : done, 1785538415000, 1785538415000
Included Modules : 1785538415000, 1785538415000
Test : 1785538415000, 1785538546000
|
Workflow Telemetry - Build CI / Test Module - relationalStoreWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - relationalStore
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538233000, 1785538234000
Checkout repo : 1785538234000, 1785538237000
Before Test : 1785538237000, 1785538375000
Restore legend-pure snapshot : done, 1785538375000, 1785538375000
Included Modules : 1785538375000, 1785538375000
Test : 1785538375000, 1785538578000
Upload Test Results : 1785538578000, 1785538579000
|
Workflow Telemetry - Build CI / Test Module - h2Workflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - h2
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538233000, 1785538235000
Checkout repo : 1785538235000, 1785538238000
Before Test : 1785538238000, 1785538408000
Restore legend-pure snapshot : done, 1785538408000, 1785538408000
Included Modules : 1785538408000, 1785538408000
Test : 1785538408000, 1785538583000
|
Workflow Telemetry - Build CI / Test Module - trinoWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - trino
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538227000, 1785538228000
Checkout repo : 1785538228000, 1785538233000
Before Test : 1785538233000, 1785538373000
Restore legend-pure snapshot : done, 1785538373000, 1785538373000
Included Modules : 1785538373000, 1785538373000
Test : 1785538373000, 1785538593000
Upload Test Results : 1785538593000, 1785538594000
|
Workflow Telemetry - Build CI / Test Module - oracleWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - oracle
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538227000, 1785538228000
Checkout repo : 1785538228000, 1785538233000
Before Test : 1785538233000, 1785538386000
Restore legend-pure snapshot : done, 1785538386000, 1785538386000
Included Modules : 1785538386000, 1785538386000
Test : 1785538386000, 1785538601000
Upload Test Results : 1785538601000, 1785538601000
|
Workflow Telemetry - Build CI / Test Module - memsqlWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - memsql
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538227000, 1785538227000
Checkout repo : 1785538227000, 1785538232000
Before Test : 1785538232000, 1785538379000
Restore legend-pure snapshot : done, 1785538379000, 1785538379000
Included Modules : 1785538379000, 1785538379000
Test : 1785538379000, 1785538647000
|
Workflow Telemetry - Build CI / Test Module - elasticsearchWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - elasticsearch
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538229000, 1785538230000
Checkout repo : 1785538230000, 1785538234000
Before Test : 1785538234000, 1785538518000
Restore legend-pure snapshot : done, 1785538518000, 1785538518000
Included Modules : 1785538518000, 1785538519000
Test : 1785538519000, 1785538676000
Upload Test Results : 1785538676000, 1785538678000
|
Workflow Telemetry - Build CI / Test Module - deephavenWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - deephaven
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538228000, 1785538230000
Checkout repo : 1785538230000, 1785538235000
Before Test : 1785538235000, 1785538420000
Restore legend-pure snapshot : done, 1785538420000, 1785538420000
Included Modules : 1785538420000, 1785538420000
Test : 1785538420000, 1785538688000
|
Workflow Telemetry - Build CI / Test Module - coreWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - core
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538226000, 1785538228000
Checkout repo : 1785538228000, 1785538233000
Before Test : 1785538233000, 1785538374000
Restore legend-pure snapshot : done, 1785538374000, 1785538374000
Included Modules : 1785538374000, 1785538374000
Test : 1785538374000, 1785538714000
|
Workflow Telemetry - Build CI / Test Module - sqlWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - sql
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538232000, 1785538236000
Checkout repo : 1785538236000, 1785538241000
Before Test : 1785538241000, 1785538432000
Restore legend-pure snapshot : done, 1785538432000, 1785538432000
Included Modules : 1785538432000, 1785538432000
Test : 1785538432000, 1785538824000
Upload Test Results : 1785538824000, 1785538825000
|
Workflow Telemetry - Build CI / Test Module - ide-toolsWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - ide-tools
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538234000, 1785538235000
Checkout repo : 1785538235000, 1785538239000
Before Test : 1785538239000, 1785538436000
Restore legend-pure snapshot : done, 1785538436000, 1785538436000
Included Modules : 1785538436000, 1785538436000
Test : 1785538436000, 1785538833000
Upload Test Results : 1785538833000, 1785538835000
|
Workflow Telemetry - Build CI / Test Module - javaBindingWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - javaBinding
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538227000, 1785538227000
Checkout repo : 1785538227000, 1785538232000
Before Test : 1785538232000, 1785538403000
Restore legend-pure snapshot : done, 1785538403000, 1785538403000
Included Modules : 1785538403000, 1785538403000
Test : 1785538403000, 1785538957000
|
Workflow Telemetry - Build CI / Test Module - serverWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - server
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538227000, 1785538229000
Checkout repo : 1785538229000, 1785538234000
Before Test : 1785538234000, 1785538403000
Restore legend-pure snapshot : done, 1785538403000, 1785538403000
Included Modules : 1785538403000, 1785538403000
Test : 1785538403000, 1785539499000
|
Workflow Telemetry - Build CI / Run Unit TestsWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Run Unit Tests
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538226000, 1785538228000
Checkout repo : 1785538228000, 1785538233000
Before Test : 1785538233000, 1785538394000
Restore legend-pure snapshot : done, 1785538394000, 1785538394000
Excluded Modules : 1785538394000, 1785538394000
Test : crit, 1785538394000, 1785539546000
Upload Test Results : done, 1785539546000, 1785539546000
|
Workflow Telemetry - Build CI / Test Module - pythonWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - python
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538226000, 1785538227000
Checkout repo : 1785538227000, 1785538232000
Before Test : 1785538232000, 1785538378000
Restore legend-pure snapshot : done, 1785538378000, 1785538378000
Included Modules : 1785538378000, 1785538378000
Test : 1785538378000, 1785539810000
Upload Test Results : 1785539810000, 1785539811000
|
Workflow Telemetry - Build CI / Test Module - databricksWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - databricks
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538226000, 1785538228000
Checkout repo : 1785538228000, 1785538232000
Before Test : 1785538232000, 1785538383000
Restore legend-pure snapshot : done, 1785538383000, 1785538383000
Included Modules : 1785538383000, 1785538383000
Test : 1785538383000, 1785539874000
Upload Test Results : 1785539874000, 1785539875000
|
Workflow Telemetry - Build CI / Test Module - snowflakeWorkflow telemetry for commit 24bdede7abfaf644b3363dacfddc22554766755b Step Tracegantt
title Test Module - snowflake
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785538233000, 1785538235000
Checkout repo : 1785538235000, 1785538237000
Before Test : 1785538237000, 1785538395000
Restore legend-pure snapshot : done, 1785538395000, 1785538395000
Included Modules : 1785538395000, 1785538395000
Test : 1785538395000, 1785539956000
|
…poser flag Promoting every doc.doc tagged value to a '''...''' documentation block reformatted existing single-line sources on save. Make the promotion conditional instead: a multi-line value always composes as a block (it has no readable single-line form), while a single-line value keeps its tagged-value formatting unless the composing context opts in through the new alwaysRenderDocumentation flag (default off) on PureGrammarComposerContext / DEPRECATED_PureGrammarComposerCore / RelationalGrammarComposerContext, carried through every builder conversion path and threaded to all eleven composers. The renderability guard is unchanged and applies in both modes. Also wire SnowflakeM2MUdf's composer for documentation (the parser already accepted it) and preserve the composing context in the deprecated composer's DataElement path instead of building a default one. With the default off, the previously updated round-trip and generation expectations (domain, service, relational, data-space, JSON schema, XSD fixtures) revert to their pre-feature text - none contained a multi-line doc value - which also restores the dataspace analytics artifact expectations that broke CI. TestDocumentationRoundtrip now pins multi-line blocks as fixed points, exercises the fallback triggers with multi-line values, and covers the default single-line conversion and both flag behaviours.
Workflow Telemetry - Build CI / BuildWorkflow telemetry for commit 014708d309a0995c7a64027ff8dc5846da9d9902 Step Tracegantt
title Build
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785621161000, 1785621163000
Checkout repo : 1785621163000, 1785621169000
Cache Maven dependencies : 1785621169000, 1785621178000
Set up JDK : 1785621178000, 1785621184000
Check Java version : 1785621184000, 1785621184000
Configure git : 1785621184000, 1785621184000
Download deps and plugins : 1785621184000, 1785621213000
Restore legend-pure snapshot : done, 1785621213000, 1785621213000
Set pure version override : 1785621213000, 1785621213000
Collect Workflow Telemetry : 1785621213000, 1785621213000
Set profiles that required github secrets : 1785621213000, 1785621213000
Purge stale legend-engine artifacts from local Maven repo : 1785621213000, 1785621213000
Build (PR) : 1785621213000, 1785628505000
Build (Master + Docker Snapshot) : done, 1785628505000, 1785628505000
Upload build output : 1785628505000, 1785628871000
Upload m2 repository : 1785628871000, 1785628898000
Purge engine artifacts before cache save : 1785628898000, 1785628898000
Set Test Matrix : 1785628898000, 1785628898000
|
Workflow Telemetry - Build CI / Test Module - sqlserverWorkflow telemetry for commit 014708d309a0995c7a64027ff8dc5846da9d9902 Step Tracegantt
title Test Module - sqlserver
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785628938000, 1785628939000
Checkout repo : 1785628939000, 1785628941000
Before Test : 1785628941000, 1785629103000
Restore legend-pure snapshot : done, 1785629103000, 1785629103000
Included Modules : 1785629103000, 1785629103000
Test : 1785629103000, 1785629223000
|
Workflow Telemetry - Build CI / Test Module - databricksWorkflow telemetry for commit 014708d309a0995c7a64027ff8dc5846da9d9902 Step Tracegantt
title Test Module - databricks
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785628934000, 1785628935000
Checkout repo : 1785628935000, 1785628939000
Before Test : 1785628939000, 1785629145000
Restore legend-pure snapshot : done, 1785629145000, 1785629145000
Included Modules : 1785629145000, 1785629145000
Test : 1785629145000, 1785630917000
Upload Test Results : 1785630917000, 1785630918000
|
TaggedValue.value becomes a CString so the protocol records whether the
literal was authored as a multi-line ('''...''') block, replacing the
composer-side alwaysRenderDocumentation flag and the multi-line content
heuristic: the parser sets multiLine from the token shape alone and the
composer obeys it, so a documentation block stays a block (including
single-line content) and an ordinary tagged value stays a tagged value.
{doc.doc = '''...'''} literals promote to documentation; a flagged value
the canonicalization could not read back unchanged still falls back to
the escaped form.
The wire shape is unchanged for existing models: a custom serializer
emits a plain JSON string unless multiLine is set (then an object
carrying _type/multiLine/value), and a custom deserializer converges
both shapes - plain old string or CString object - into the new POJO.
TestTaggedValueCompatibility pins both directions, and every grammar
round-trip test now also proves the flag survives the JSON hop. The flag
is protocol-only: compiling to the graph drops it, so graph-regenerated
protocol composes docs as ordinary tagged values.
All parser tagged-value sites (domain, data, and the nine DSL walkers)
build the value through toCString; the doc-sugar builder marks it
multi-line. Graph builders, the query store, and the compiler read
value.value; the three composer contexts lose the flag machinery
added by the previous commit.
…tations Every composer that supports documentation repeated the same fragile pair: render the documentation block, then remember to strip the promoted doc tagged value out of the annotations via withoutDocumentation. Fold the pattern into a single HelperDomainGrammarComposer.renderDeclarationPrefix(keyword, [indent,] stereotypes, taggedValues) and migrate all eighteen call sites - domain elements and members, Data, DataSpace, Service, DataQuality, Database, Schema, Table, View, and the five function activators. Relational columns keep the explicit pair: their grammar puts annotations after the column name, so there is no contiguous prefix to render.
Workflow Telemetry - Build CI / BuildWorkflow telemetry for commit e9fb32c6ff685ca36821b034773e74552bbf4fde Step Tracegantt
title Build
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785695799000, 1785695805000
Checkout repo : 1785695805000, 1785695811000
Cache Maven dependencies : 1785695811000, 1785695821000
Set up JDK : 1785695821000, 1785695832000
Check Java version : 1785695832000, 1785695832000
Configure git : 1785695832000, 1785695832000
Download deps and plugins : 1785695832000, 1785695851000
Restore legend-pure snapshot : done, 1785695851000, 1785695851000
Set pure version override : 1785695851000, 1785695851000
Collect Workflow Telemetry : 1785695851000, 1785695851000
Set profiles that required github secrets : 1785695851000, 1785695851000
Purge stale legend-engine artifacts from local Maven repo : 1785695851000, 1785695851000
Build (PR) : 1785695851000, 1785701292000
Build (Master + Docker Snapshot) : done, 1785701292000, 1785701292000
Upload build output : 1785701292000, 1785701640000
Upload m2 repository : 1785701640000, 1785701663000
Purge engine artifacts before cache save : 1785701663000, 1785701663000
Set Test Matrix : 1785701663000, 1785701663000
Upload CI Event : 1785701663000, 1785701664000
|
Workflow Telemetry - Build CI / BuildWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Build
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785701699000, 1785701702000
Checkout repo : 1785701702000, 1785701709000
Cache Maven dependencies : 1785701709000, 1785701724000
Set up JDK : 1785701724000, 1785701730000
Check Java version : 1785701730000, 1785701730000
Configure git : 1785701730000, 1785701730000
Download deps and plugins : 1785701730000, 1785701755000
Restore legend-pure snapshot : done, 1785701755000, 1785701755000
Set pure version override : 1785701755000, 1785701755000
Collect Workflow Telemetry : 1785701755000, 1785701755000
Set profiles that required github secrets : 1785701755000, 1785701755000
Purge stale legend-engine artifacts from local Maven repo : 1785701755000, 1785701755000
Build (PR) : 1785701755000, 1785708656000
Build (Master + Docker Snapshot) : done, 1785708656000, 1785708656000
Upload build output : 1785708656000, 1785709033000
Upload m2 repository : 1785709033000, 1785709060000
Purge engine artifacts before cache save : 1785709060000, 1785709060000
Set Test Matrix : 1785709060000, 1785709060000
|
Workflow Telemetry - Build CI / Test Module - sqlserverWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - sqlserver
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709104000, 1785709105000
Checkout repo : 1785709105000, 1785709108000
Before Test : 1785709108000, 1785709255000
Restore legend-pure snapshot : done, 1785709255000, 1785709255000
Included Modules : 1785709255000, 1785709255000
Test : 1785709255000, 1785709376000
Upload Test Results : 1785709376000, 1785709376000
|
Workflow Telemetry - Build CI / Test Module - graphqlWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - graphql
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709099000
Checkout repo : 1785709099000, 1785709104000
Before Test : 1785709104000, 1785709266000
Restore legend-pure snapshot : done, 1785709266000, 1785709266000
Included Modules : 1785709266000, 1785709266000
Test : 1785709266000, 1785709384000
Upload Test Results : 1785709384000, 1785709385000
|
Workflow Telemetry - Build CI / Test Module - spannerWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - spanner
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709099000
Checkout repo : 1785709099000, 1785709103000
Before Test : 1785709103000, 1785709267000
Restore legend-pure snapshot : done, 1785709267000, 1785709267000
Included Modules : 1785709267000, 1785709267000
Test : 1785709267000, 1785709402000
Upload Test Results : 1785709402000, 1785709403000
|
Workflow Telemetry - Build CI / Test Module - duckdbWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - duckdb
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709104000, 1785709105000
Checkout repo : 1785709105000, 1785709108000
Before Test : 1785709108000, 1785709273000
Restore legend-pure snapshot : done, 1785709273000, 1785709273000
Included Modules : 1785709273000, 1785709273000
Test : 1785709273000, 1785709409000
|
Workflow Telemetry - Build CI / Test Module - postgresWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - postgres
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709099000
Checkout repo : 1785709099000, 1785709104000
Before Test : 1785709104000, 1785709264000
Restore legend-pure snapshot : done, 1785709264000, 1785709264000
Included Modules : 1785709264000, 1785709264000
Test : 1785709264000, 1785709410000
Upload Test Results : 1785709410000, 1785709411000
|
Workflow Telemetry - Build CI / Test Module - h2Workflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - h2
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709100000
Checkout repo : 1785709100000, 1785709104000
Before Test : 1785709104000, 1785709271000
Restore legend-pure snapshot : done, 1785709271000, 1785709271000
Included Modules : 1785709271000, 1785709271000
Test : 1785709271000, 1785709436000
Upload Test Results : 1785709436000, 1785709438000
|
Workflow Telemetry - Build CI / Test Module - elasticsearchWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - elasticsearch
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709104000, 1785709106000
Checkout repo : 1785709106000, 1785709110000
Before Test : 1785709110000, 1785709323000
Restore legend-pure snapshot : done, 1785709323000, 1785709323000
Included Modules : 1785709323000, 1785709323000
Test : 1785709323000, 1785709463000
Upload Test Results : 1785709463000, 1785709464000
|
Workflow Telemetry - Build CI / Test Module - oracleWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - oracle
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709101000
Checkout repo : 1785709101000, 1785709105000
Before Test : 1785709105000, 1785709251000
Restore legend-pure snapshot : done, 1785709251000, 1785709251000
Included Modules : 1785709251000, 1785709252000
Test : 1785709252000, 1785709464000
Upload Test Results : 1785709464000, 1785709465000
|
Workflow Telemetry - Build CI / Test Module - relationalStoreWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - relationalStore
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709100000
Checkout repo : 1785709100000, 1785709106000
Before Test : 1785709106000, 1785709309000
Restore legend-pure snapshot : done, 1785709309000, 1785709309000
Included Modules : 1785709309000, 1785709309000
Test : 1785709309000, 1785709489000
Upload Test Results : 1785709489000, 1785709490000
|
Workflow Telemetry - Build CI / Test Module - trinoWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - trino
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709100000
Checkout repo : 1785709100000, 1785709105000
Before Test : 1785709105000, 1785709254000
Restore legend-pure snapshot : done, 1785709254000, 1785709254000
Included Modules : 1785709254000, 1785709254000
Test : 1785709254000, 1785709493000
Upload Test Results : 1785709493000, 1785709494000
|
Workflow Telemetry - Build CI / Test Module - memsqlWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - memsql
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709105000, 1785709107000
Checkout repo : 1785709107000, 1785709111000
Before Test : 1785709111000, 1785709284000
Restore legend-pure snapshot : done, 1785709284000, 1785709284000
Included Modules : 1785709284000, 1785709284000
Test : 1785709284000, 1785709576000
Upload Test Results : 1785709576000, 1785709577000
|
Workflow Telemetry - Build CI / Test Module - deephavenWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - deephaven
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709104000, 1785709105000
Checkout repo : 1785709105000, 1785709108000
Before Test : 1785709108000, 1785709264000
Restore legend-pure snapshot : done, 1785709264000, 1785709264000
Included Modules : 1785709264000, 1785709264000
Test : 1785709264000, 1785709582000
|
Workflow Telemetry - Build CI / Test Module - ide-toolsWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - ide-tools
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709105000, 1785709107000
Checkout repo : 1785709107000, 1785709110000
Before Test : 1785709110000, 1785709281000
Restore legend-pure snapshot : done, 1785709281000, 1785709281000
Included Modules : 1785709281000, 1785709281000
Test : 1785709281000, 1785709668000
Upload Test Results : 1785709668000, 1785709669000
|
Workflow Telemetry - Build CI / Test Module - clickhouseWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - clickhouse
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709107000, 1785709108000
Checkout repo : 1785709108000, 1785709111000
Before Test : 1785709111000, 1785709380000
Restore legend-pure snapshot : done, 1785709380000, 1785709380000
Included Modules : 1785709380000, 1785709380000
Test : crit, 1785709380000, 1785709699000
Upload Test Results : done, 1785709699000, 1785709699000
|
Workflow Telemetry - Build CI / Test Module - coreWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - core
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709100000
Checkout repo : 1785709100000, 1785709104000
Before Test : 1785709104000, 1785709268000
Restore legend-pure snapshot : done, 1785709268000, 1785709268000
Included Modules : 1785709268000, 1785709268000
Test : 1785709268000, 1785709700000
|
Workflow Telemetry - Build CI / Test Module - sqlWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - sql
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709099000, 1785709100000
Checkout repo : 1785709100000, 1785709104000
Before Test : 1785709104000, 1785709283000
Restore legend-pure snapshot : done, 1785709283000, 1785709283000
Included Modules : 1785709283000, 1785709283000
Test : 1785709283000, 1785709733000
|
Workflow Telemetry - Build CI / Test Module - javaBindingWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - javaBinding
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709099000, 1785709100000
Checkout repo : 1785709100000, 1785709104000
Before Test : 1785709104000, 1785709283000
Restore legend-pure snapshot : done, 1785709283000, 1785709283000
Included Modules : 1785709283000, 1785709283000
Test : 1785709283000, 1785709822000
|
Workflow Telemetry - Build CI / Test Module - snowflakeWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - snowflake
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709099000
Checkout repo : 1785709099000, 1785709104000
Before Test : 1785709104000, 1785709261000
Restore legend-pure snapshot : done, 1785709261000, 1785709261000
Included Modules : 1785709261000, 1785709261000
Test : 1785709261000, 1785710377000
|
Workflow Telemetry - Build CI / Test Module - serverWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - server
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709100000
Checkout repo : 1785709100000, 1785709105000
Before Test : 1785709105000, 1785709305000
Restore legend-pure snapshot : done, 1785709305000, 1785709305000
Included Modules : 1785709305000, 1785709305000
Test : 1785709305000, 1785710417000
Upload Test Results : 1785710417000, 1785710419000
|
Workflow Telemetry - Build CI / Test Module - pythonWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - python
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709100000
Checkout repo : 1785709100000, 1785709104000
Before Test : 1785709104000, 1785709268000
Restore legend-pure snapshot : done, 1785709268000, 1785709268000
Included Modules : 1785709268000, 1785709268000
Test : 1785709268000, 1785710759000
Upload Test Results : 1785710759000, 1785710760000
|
Workflow Telemetry - Build CI / Test Module - databricksWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - databricks
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709100000, 1785709101000
Checkout repo : 1785709101000, 1785709105000
Before Test : 1785709105000, 1785709265000
Restore legend-pure snapshot : done, 1785709265000, 1785709265000
Included Modules : 1785709265000, 1785709265000
Test : 1785709265000, 1785710875000
Upload Test Results : 1785710875000, 1785710877000
|
Workflow Telemetry - Build CI / Run Unit TestsWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Run Unit Tests
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785709098000, 1785709099000
Checkout repo : 1785709099000, 1785709104000
Before Test : 1785709104000, 1785709272000
Restore legend-pure snapshot : done, 1785709272000, 1785709272000
Excluded Modules : 1785709272000, 1785709272000
Test : 1785709272000, 1785711004000
|
Workflow Telemetry - Build CI / Test Module - clickhouseWorkflow telemetry for commit 59b2dae153b1f445f94be35125e5adffdb96ae46 Step Tracegantt
title Test Module - clickhouse
dateFormat x
axisFormat %H:%M:%S
Set up job : milestone, 1785759179000, 1785759182000
Checkout repo : 1785759182000, 1785759188000
Before Test : 1785759188000, 1785759364000
Restore legend-pure snapshot : done, 1785759364000, 1785759364000
Included Modules : 1785759364000, 1785759364000
Test : 1785759364000, 1785759492000
Upload Test Results : 1785759492000, 1785759493000
|
Engine is adding multi-line ('''...''') string literals to the Pure
grammar and records them in the protocol in two places:
- CString gains an optional `multiLine` flag, omitted from the wire
when false (finos/legend-engine#4998)
- TaggedValue's `value` becomes a CString, so it is sent as a plain
string unless the value was authored multi-line, in which case it
is an object carrying the flag (finos/legend-engine#5008)
Both changes are backward compatible on engine's side. Studio, however,
drops unknown keys on deserialization, so without this it would silently
rewrite a multi-line documentation as an escaped single-line literal on
the next save.
Tagged values keep a flat `value: string` with a sibling `multiLine`
flag rather than mirroring engine's CString typing - the union is
handled entirely in `V1_taggedValueModelSchema`, which leaves every
`taggedValue.value` consumer untouched. The flag is hashed on both the
metamodel and the protocol side so that toggling it registers as a
change; note it has to be stringified first, as `hash.js` digests any
boolean to the hash of the empty string. It is left out of
`V1_CString`'s hash to keep that compatible with
`PrimitiveInstanceValue`, whose metamodel has no counterpart.
Ports legend-pure #1299 (
fb29e8f5929). A'''...'''literal immediately preceding a declaration is syntactic sugar for themeta::pure::profiles::docdoctagged value, so a.purefile legend-pure accepts now parses here too.is identical after parsing to
One protocol change, wire-compatible for existing models.
TaggedValue.valuebecomes aCString, so the protocol records whether the literal was authored as a multi-line block (multiLine, the same flag #4998 added to string literals). On the wire the value stays a plain JSON string — byte-identical to the old shape — unlessmultiLineis set, in which case it is an object ({"_type": "string", "multiLine": true, "value": ...}); a custom deserializer converges both shapes, so every existing model deserializes unchanged and only models using documentation blocks put the new shape on the wire. Desugaring still happens in the parser: the compiler and everydocconsumer see an ordinaryTaggedValue.Where it attaches
Wherever an element accepts tagged values, it accepts documentation — 14 declarations across 11 grammars:
###PureClass,Enumand its values,Association,function, properties, qualified properties###RelationalDatabase,Schema,Table, column,View###DataSpaceDataSpace###Service/###HostedServiceService###FunctionJarFunctionJar###DataQualityDataQualityValidation,DataQualityRelationValidation###SnowflakeSnowflakeApp,SnowflakeM2MUdf###BigQueryFunction/###MemSqlFunctionBigQueryFunction,MemSqlFunction###DataDataThe
documentationrule lives inM3ParserGrammar.g4, so the grammars importing it inherit the rule and only name it in their element rule;RelationalParserGrammar.g4andDataParserGrammar.g4importCoreParserGrammarand carry their ownstereotypes/taggedValues, so they declare a local copy.PureGrammarParserUtility.taggedValuesWithDocumentationis the single walker-side implementation — it takes the literal's token rather than the context, because ANTLR generates a distinctDocumentationContextclass per grammar even for an inherited rule.Documentation is a parser rule listed explicitly at each declaration, not a lexer construct. That is what keeps the identical literal in expression position an ordinary multi-line string, and why attachment needs no adjacency heuristic. The engine folds text blocks into a single
STRINGtoken — deliberately, since that is what let #4998 avoid touching 275STRINGreferences across ~50 grammars — so the block shape is checked in the walker, which also gives a better message than an ANTLR mismatch.Behaviour worth reviewing
Documentation does not populate a DSL's own
documentation:/description:field. Service, HostedService and FunctionJar havedocumentation; DataSpace, Snowflake, BigQuery and MemSql havedescription. A block is thedoctagged value in every grammar, so the same syntax means the same thing everywhere; the DSL-specific field stays a separate slot and an element may carry both.Content is literal — no escape processing.
\n,\'and\\are content. Unescaping prose would silently rewrite a regex (\d+→d+), a Markdown escape (\*→*) and a Windows path (C:\temp→C:followed by a tab). Leading and trailing blank lines are dropped, which is what makes a block and the equivalent explicitdoc.dochold the same string. Otherwise the layout is shared with #4998's string literals.An element may not carry both documentation and an explicit
doc.doc— a parser error rather than a silent drop. Because tag references are unresolved at parse time, the check matches the profile as written (baredocor fully qualified);my::pkg::doc.docis a different profile and not a conflict.The composer renders whichever form the value was authored in. The parser sets
multiLinefrom the token shape alone and the composer obeys it — neither inspects the value, mirroring the string-literal rule from #4998. A documentation block stays a block (including single-line content); an ordinary tagged value stays a tagged value even when its value contains newlines, so existing sources do not reformat on save; and a{doc.doc = '''...'''}literal, which carries the flag, composes to the block form documentation is sugar for. Documentation has no escape mechanism at all, soisRenderableAsDocumentationstill guards promotion and anything rejected stays a tagged value: an embedded''', a carriage return, trailing whitespace on a line, a leading or trailing blank line (a block literal closed on its own line carries a trailing\n, which a documentation block cannot represent), an empty value, twodoctags on one element, and a line beginning###(the section split runs before any grammar and is not string-aware). The flag is protocol-only — compiling to the graph drops it — so protocol regenerated from a compiled model composes docs as ordinary tagged values.Notes
The doc tagged value's source range is computed in the walker rather than through
ParseTreeWalkerSourceInformation. A token reports only the line it starts on, and the shared helper derives the end column from the start column plus the whole token length — wrong for any token whose text spans lines. Fixing it there, as legend-pure did, regresses the section index: island and section content tokens end with a newline rather than after one, which yields an end column of 0. A documentation literal always spans lines and always ends on''', so it has no such edge case. The general multi-line end coordinate stays wrong forCStringand island tokens — pre-existing, not introduced here.Profile,Measureandnative functionare left out and documented as gaps rather than half-supported: the grammar gives them no annotation slots,Profilehas notaggedValuesfield in the protocol, andnativeFunctionhas no walker. Graph-fetch projections are out of scope too — legend-pure did not touch them.This makes the engine accept documentation in
###Relational, which legend-pure does not: there a'''...'''relational tagged value parses and then NPEs inRelationalGraphBuilder.visitTaggedValueNew.Testing
New
TestTaggedValueCompatibility(7) pins the wire shape in both directions: legacy plain-string JSON converges into the new POJO,multiLine=falseserializes byte-identically to the old shape, and both shapes round-trip. NewTestDocumentationParsing(23) andTestDocumentationRoundtrip(27): blocks are fixed points (including single-line content), tagged values are fixed points (including multi-line content),{doc.doc = '''...'''}promotes, and each renderability-guard trigger is driven from hand-built protocol JSON — the only way a flagged-but-unrenderable value can arise. The round-trip harness serializes and re-reads the model between parse and compose, so every round-trip test also proves the flag survives the JSON hop. Per-DSL round-trip coverage and compilation tests assert the sugared and explicit forms resolve to the sameTaginstance.Because values not authored as blocks keep their form, existing expectations (domain/service/relational/data-space round-trips, JSON schema and XSD generation fixtures, dataspace analytics artifacts) are untouched by this PR.
Green: core grammar 369, compiler 461, the eleven affected DSL/format modules, and the dataspace analytics generation suite. Checkstyle clean.
Documented in
docs/engineering/architecture/type-system.md.🤖 Generated with Claude Code