fix: validate attribute types at declaration and insert (#1527, #1528, #1529, #1530)

[dimitri-yatsenko]

Closes #1527, closes #1528, closes #1529, closes #1530.

Four bugs with one shape: the type system accepted spellings it could not honour, and
failed late — at the driver or at the server — instead of at declaration. Fixing them
separately would have meant four passes over the same twenty lines, so they are here
together.

#1527 — native blob silently stored str(array)

__make_placeholder handed the value to the driver untouched for a codec-less blob
attribute. PyMySQL has no encoder for ndarray and falls back to escape_str, so an
array was stored as its text repr — elided in the middle for anything large — with no
error on insert and none on fetch. It now requires bytes/bytearray/memoryview
and names <blob> in the error. This is the one with actual data loss.

The declaration warning for native blob types also now says the column stores raw
bytes and points at <blob>, instead of the generic "consider a core DataJoint type
for better portability", which reads as cosmetic.

Note the failure was MySQL-specific: on PostgreSQL the same insert already raised
can't adapt type, because that adapter registers numpy scalars only. The guard makes
the two backends agree.

#1528 — decimal(M,D) unsigned rejected outright

A regression from 0.14.x, which had one permissive pattern for both spellings. 2.x
promoted decimal to a core type with a strict pattern and left the old permissive
branch under the name NUMERIC — which kept numeric and dropped decimal. The
result was that numeric(2,2) unsigned passed while the canonical spelling of the
same SQL type raised. NUMERIC now covers decimal/numeric/dec/fixed with
unsigned/zerofill; the core DECIMAL pattern is still matched first, so
decimal(M,D) remains a core type.

Heading derives its numeric flag from these same patterns and did not consult
NUMERIC, so a modified decimal was classified as neither numeric nor string. Fixed
in the same place.

Found in element-optogenetics, which declares four decimal(2, 2) unsigned
proportions and cannot be declared at all on 2.x.

#1529 — migration wrote core-type markers that do not exist

NATIVE_TO_CORE_TYPE mapped unsigned integers to uint8…uint64 and mediumint to
int24, none of which match_type accepts. Phase 2 stamped :uint32: into column
comments and Heading then silently dropped the marker, so the effect was cosmetic
rather than destructive — but the labels meant nothing, and the guide documented a
different mapping than the code performed.

Unsigned columns now widen to the next signed type that holds their full range, which
is what migrate-to-v20.md already said. bigint unsigned is the one case with no
lossless target — values above 2**63-1 do not fit in int64 — so it maps to
int64 and logs a warning naming the column. That is the decision most worth a second
opinion in this PR.

#1530 — misspelled types classified as native integers

match_type used re.match, and the $ in the INTEGER pattern bound to the
serial alternative only, so int24, tinyinteger and intbanana all matched on
their prefix and were emitted into the DDL verbatim — producing a MySQL syntax error
instead of Unsupported attribute type. Now fullmatch, with the alternation anchored
as a whole.

tests/integration/test_declare.py::test_unsupported_int_datatype already asserted
this behaviour and was passing only because the server rejected the DDL; it now passes
for the right reason.

I did not move the TYPE_PATTERN call sites in heading.py to fullmatch, which
#1530 suggested. Those match against types reported by the server, which are not
normalized — PostgreSQL returns double precision and timestamp without time zone,
and both currently classify on their leading word. fullmatch there would break the
PostgreSQL backend. Normalizing adapter-reported types is the real fix and is a larger
change; I left a comment at the call site saying so.

Also: bare blob

NATIVE_BLOB required a size prefix, so blob — a MySQL type in its own right, and
one migrate.BLOB_TYPES already recognized — was rejected while tinyblob and
longblob were accepted. The size prefix is now optional. This was found by the new
cross-reference test rather than by reading, and it is real: element-moseq declares
three bare blob attributes.

Testing

tests/unit/test_type_patterns.py covers the seam between the three places that name
a type. The assertions that would have caught these:

• every value in NATIVE_TO_CORE_TYPE round-trips through match_type (migrate.py stamps :uint32:/:int24: markers for core types that do not exist; guide documents a different mapping #1529, bare blob)
• near-miss spellings raise (match_type uses re.match with an unanchored INTEGER alternative, so misspelled types (int24, intbanana) classify as native integers #1530)
• the modified and aliased decimal forms classify as native (Regression: decimal(M,D) with unsigned/zerofill or as dec/fixed is rejected, where numeric(M,D) unsigned passes #1528)
• the native spellings that already worked still classify identically, so the
  strictness change cannot quietly narrow what is accepted

Plus integration tests for the insert guard, a bare blob round-trip, and declaring
and reading back decimal(2, 2) unsigned.

Full suite green on both backends: 1021 passed, 10 skipped. ruff, ruff-format and
mypy clean at the versions pinned in .pre-commit-config.yaml.

Follow-up

The documentation half needs a datajoint-docs PR: migrate-to-v20.md:1180 currently
promises decimal(M,D) → decimal(M,D) # unchanged, and :1080 documents the
unsigned-integer mapping this PR aligns the code to. type-system.md should also say
what happens to native modifiers. Not in this repo, so not in this PR.

[MilagrosMarin]

Verified the four fixes. The fullmatch change is safe — match_type() is only reached from declare.py:857/:957, both on the declaration path, never on server-reported types, and the patterns involved already end in $. The insert guard correctly gates on attr.is_blob and attr.codec is None.

On the bigint unsigned call you flagged — traced it through:

migrate_columns emits MODIFY COLUMN {native_type} … COMMENT, so the physical column is preserved and stored data is untouched; values above 2**63-1 still fetch fine. The effect is descriptive: heading.py:517 sets original_type = "int64", and Table.describe() renders original_type or attr.type — so describe() reports int64 for a column that is physically bigint unsigned.

That's where it bites: the describe→recreate→copy path produces a signed bigint, and values above 2**63-1 don't survive that copy. Nothing is lost at migration time, and the warning is well-placed, but the label is what describe() echoes downstream.

Worth weighing as an alternative: leave bigint unsigned out of NATIVE_TO_CORE_TYPE and give it an explicit warn-and-skip branch. The column then stays honestly native and describe() keeps reporting bigint unsigned. It would need the explicit branch — unmapped types currently fall through the "silently skipped" path, and silence would be worse than what you have now. Your call; both are defensible, but the describe() round-trip is the consequence I'd weigh it against.

Smaller: the new heading.py comment says prefix matching is what lets PostgreSQL spellings like "double precision" and "timestamp without time zone" match on their leading word. Testing both directly, re.match returns None for each — FLOAT and TEMPORAL both end in $, and every pattern in those tuples is anchored, so .match() and .fullmatch() are equivalent there. No runtime effect; just a rationale future maintainers would take at face value.

[dimitri-yatsenko]

Addressed the review:

1. bigint unsigned — ruled: the mapping stays; the warning is the fix. There is no lossless signed target, and leaving the type unmapped would strand the column outside the core type system permanently — a different permanent cost. What changed per your trace: the warning now names the actual hazard (describe() reports int64, so a describe→recreate→copy round trip produces a signed column and values above 2**63-1 do not survive the copy — physical column and stored data untouched by migration itself) and names the lossless escape hatch: declare decimal(20,0), which holds the full unsigned range.

2. The heading.py comment you disproved is corrected — it now states that .match() is equivalent to .fullmatch() there because every pattern in those tuples is anchored, rather than claiming prefix matching does work it doesn't.