Skip to content

Truncate TIMESTAMPTZ arithmetically in ICU date_trunc for fixed-length parts - #8

Merged
redox merged 2 commits into
altertable-1.5.5from
su/boost-date-trunc
Sep 30, 2026
Merged

redox merged 2 commits into
altertable-1.5.5from
su/boost-date-trunc

Conversation

@redox

@redox redox commented Sep 30, 2026

Copy link
Copy Markdown
Member

NOTE: this is a fork-only change. The ICU date_trunc code on upstream duckdb/main has changed since this fork point, so this commit will not be upstreamed as-is. Before proposing anything upstream, check whether main still has the per-row ICU cost measured below; it may already be addressed there, and the patch would need rework against the new code either way.

Problem

date_trunc(part, TIMESTAMPTZ) went through an icu::Calendar for every row: setTime, set fields, recompute. With a constant 'day' part on 10M rows in America/New_York, that took ~0.2s against ~0.01s for naive TIMESTAMP.

Change

When the part is a constant of fixed wall-time length (microseconds, milliseconds, second/epoch, minute, hour, day and its aliases), bind swaps in ArithmeticTruncFunction, which works on µs directly:

  • Below a minute, the result is floor(utc, unit). tzdata offsets are whole seconds, so the time zone and calendar cannot affect the result. This applies to every calendar.
  • Minute: keep the instant's offset (ICU's PreserveOffsets), floor the wall time, subtract the same offset.
  • Hour and day: floor the wall time, then re-resolve the offset at the truncated wall time, like ICU does. Gaps take the offset before a transition and overlaps the offset after it, which matches ICU's default UCAL_WALLTIME_LAST.

Minute and above are only enabled for the gregorian calendar. Everything else (variable parts, coarser parts, other calendars, zones that don't expose their transitions) keeps the existing ICU path, whose function body is unchanged apart from a small TruncWithICU helper.

The offsets come from a new ZoneOffsets table (icu-zone-offsets.{hpp,cpp}) built at bind time from icu::BasicTimeZone transitions. It is a sorted list of constant-offset segments with sentinels at both ends. One lookup serves both instants and wall times, and a one-segment cursor skips the binary search for clustered input. Zones with recurring rules are covered up to 2250; later instants, and instants within three days of the int64 limits, fall back to ICU per row. Build rejects zones where wall times would not increase monotonically, or where an offset is a day or more. The second check is what makes a single bounds check enough to rule out overflow. The table is immutable and shared between bind data copies through a shared_ptr.

Performance (Apple Silicon, release build)

  • 10M rows, America/New_York, all threads: ICU path 0.202s, arithmetic path 0.0075s. Naive TIMESTAMP is 0.0086s.
  • 100M sorted rows, one thread, per row: day 2.2 ns, hour 2.2 ns, minute 2.0 ns, second 1.5 ns. A plain max(ts) scan is 0.42 ns.
  • 20M shuffled rows, one thread: 27 ns per row, against 255 ns for ICU. Random order misses the cursor, so each row does two binary searches over about 700 segments. A bucket index could fix this, but it was left out to keep ZoneOffsets simple.

Tests

test_icu_datetrunc.test gets explicit cases (UTC, a fixed offset, Asia/Kolkata history, New York spring-forward and fall-back, a session TimeZone change between queries, a non-gregorian calendar). It also gets a differential check: a part stored in a column forces the ICU path, and every constant-part result must match it. That runs across 11 zones and the islamic-civil calendar, over ~13.5k instants covering transition windows, 1883, 1945, the 1970 epoch, the 2250 end of the table and the infinities. Deliberately introduced bugs (no hour re-resolution, an off-by-one floor, an off-by-one segment end) each make it fail.

Four micro-benchmarks under benchmark/micro/timestamp/ compare naive TIMESTAMP with TIMESTAMPTZ in UTC, a fixed offset and America/New_York.

…h parts

NOTE: this is a fork-only change. The ICU date_trunc code on upstream
duckdb/main has changed since this fork point, so this commit will not be
upstreamed as-is. Before proposing anything upstream, check whether main
still has the per-row ICU cost measured below; it may already be addressed
there, and the patch would need rework against the new code either way.

Problem

date_trunc(part, TIMESTAMPTZ) went through an icu::Calendar for every row:
setTime, set fields, recompute. With a constant 'day' part on 10M rows in
America/New_York, that took ~0.2s against ~0.01s for naive TIMESTAMP.

Change

When the part is a constant of fixed wall-time length (microseconds,
milliseconds, second/epoch, minute, hour, day and its aliases), bind swaps
in ArithmeticTruncFunction, which works on µs directly:

- Below a minute, the result is floor(utc, unit). tzdata offsets are whole
  seconds, so the time zone and calendar cannot affect the result. This
  applies to every calendar.
- Minute: keep the instant's offset (ICU's PreserveOffsets), floor the wall
  time, subtract the same offset.
- Hour and day: floor the wall time, then re-resolve the offset at the
  truncated wall time, like ICU does. Gaps take the offset before a
  transition and overlaps the offset after it, which matches ICU's
  default UCAL_WALLTIME_LAST.

Minute and above are only enabled for the gregorian calendar. Everything
else (variable parts, coarser parts, other calendars, zones that don't
expose their transitions) keeps the existing ICU path, whose function body
is unchanged apart from a small TruncWithICU helper.

The offsets come from a new ZoneOffsets table (icu-zone-offsets.{hpp,cpp})
built at bind time from icu::BasicTimeZone transitions. It is a sorted list
of constant-offset segments with sentinels at both ends. One lookup serves
both instants and wall times, and a one-segment cursor skips the binary
search for clustered input. Zones with recurring rules are covered up to
2250; later instants, and instants within three days of the int64 limits,
fall back to ICU per row. Build rejects zones where wall times would not
increase monotonically, or where an offset is a day or more. The second
check is what makes a single bounds check enough to rule out overflow.
The table is immutable and shared between bind data copies through a
shared_ptr.

Performance (Apple Silicon, release build)

- 10M rows, America/New_York, all threads:
  ICU path 0.202s, arithmetic path 0.0075s. Naive TIMESTAMP is 0.0086s.
- 100M sorted rows, one thread, per row: day 2.2 ns, hour 2.2 ns,
  minute 2.0 ns, second 1.5 ns. A plain max(ts) scan is 0.42 ns.
- 20M shuffled rows, one thread: 27 ns per row, against 255 ns for ICU.
  Random order misses the cursor, so each row does two binary searches
  over about 700 segments. A bucket index could fix this, but it was left
  out to keep ZoneOffsets simple.

Tests

test_icu_datetrunc.test gets explicit cases (UTC, a fixed offset,
Asia/Kolkata history, New York spring-forward and fall-back, a session
TimeZone change between queries, a non-gregorian calendar). It also gets a
differential check: a part stored in a column forces the ICU path, and
every constant-part result must match it. That runs across 11 zones and
the islamic-civil calendar, over ~13.5k instants covering transition
windows, 1883, 1945, the 1970 epoch, the 2250 end of the table and the
infinities. Deliberately introduced bugs (no hour re-resolution, an
off-by-one floor, an off-by-one segment end) each make it fail.

Four micro-benchmarks under benchmark/micro/timestamp/ compare naive
TIMESTAMP with TIMESTAMPTZ in UTC, a fixed offset and America/New_York.

Co-authored-by: Cursor <cursoragent@cursor.com>
@redox
redox merged commit 66aa0e4 into altertable-1.5.5 Sep 30, 2026
49 of 69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants