Truncate TIMESTAMPTZ arithmetically in ICU date_trunc for fixed-length parts - #8
Merged
Merged
Conversation
…h parts
NOTE: this is a fork-only change. The ICU date_trunc code on upstream
duckdb/main has changed since this fork point, so this commit will not be
upstreamed as-is. Before proposing anything upstream, check whether main
still has the per-row ICU cost measured below; it may already be addressed
there, and the patch would need rework against the new code either way.
Problem
date_trunc(part, TIMESTAMPTZ) went through an icu::Calendar for every row:
setTime, set fields, recompute. With a constant 'day' part on 10M rows in
America/New_York, that took ~0.2s against ~0.01s for naive TIMESTAMP.
Change
When the part is a constant of fixed wall-time length (microseconds,
milliseconds, second/epoch, minute, hour, day and its aliases), bind swaps
in ArithmeticTruncFunction, which works on µs directly:
- Below a minute, the result is floor(utc, unit). tzdata offsets are whole
seconds, so the time zone and calendar cannot affect the result. This
applies to every calendar.
- Minute: keep the instant's offset (ICU's PreserveOffsets), floor the wall
time, subtract the same offset.
- Hour and day: floor the wall time, then re-resolve the offset at the
truncated wall time, like ICU does. Gaps take the offset before a
transition and overlaps the offset after it, which matches ICU's
default UCAL_WALLTIME_LAST.
Minute and above are only enabled for the gregorian calendar. Everything
else (variable parts, coarser parts, other calendars, zones that don't
expose their transitions) keeps the existing ICU path, whose function body
is unchanged apart from a small TruncWithICU helper.
The offsets come from a new ZoneOffsets table (icu-zone-offsets.{hpp,cpp})
built at bind time from icu::BasicTimeZone transitions. It is a sorted list
of constant-offset segments with sentinels at both ends. One lookup serves
both instants and wall times, and a one-segment cursor skips the binary
search for clustered input. Zones with recurring rules are covered up to
2250; later instants, and instants within three days of the int64 limits,
fall back to ICU per row. Build rejects zones where wall times would not
increase monotonically, or where an offset is a day or more. The second
check is what makes a single bounds check enough to rule out overflow.
The table is immutable and shared between bind data copies through a
shared_ptr.
Performance (Apple Silicon, release build)
- 10M rows, America/New_York, all threads:
ICU path 0.202s, arithmetic path 0.0075s. Naive TIMESTAMP is 0.0086s.
- 100M sorted rows, one thread, per row: day 2.2 ns, hour 2.2 ns,
minute 2.0 ns, second 1.5 ns. A plain max(ts) scan is 0.42 ns.
- 20M shuffled rows, one thread: 27 ns per row, against 255 ns for ICU.
Random order misses the cursor, so each row does two binary searches
over about 700 segments. A bucket index could fix this, but it was left
out to keep ZoneOffsets simple.
Tests
test_icu_datetrunc.test gets explicit cases (UTC, a fixed offset,
Asia/Kolkata history, New York spring-forward and fall-back, a session
TimeZone change between queries, a non-gregorian calendar). It also gets a
differential check: a part stored in a column forces the ICU path, and
every constant-part result must match it. That runs across 11 zones and
the islamic-civil calendar, over ~13.5k instants covering transition
windows, 1883, 1945, the 1970 epoch, the 2250 end of the table and the
infinities. Deliberately introduced bugs (no hour re-resolution, an
off-by-one floor, an off-by-one segment end) each make it fail.
Four micro-benchmarks under benchmark/micro/timestamp/ compare naive
TIMESTAMP with TIMESTAMPTZ in UTC, a fixed offset and America/New_York.
Co-authored-by: Cursor <cursoragent@cursor.com>
utay
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOTE: this is a fork-only change. The ICU date_trunc code on upstream duckdb/main has changed since this fork point, so this commit will not be upstreamed as-is. Before proposing anything upstream, check whether main still has the per-row ICU cost measured below; it may already be addressed there, and the patch would need rework against the new code either way.
Problem
date_trunc(part, TIMESTAMPTZ) went through an icu::Calendar for every row: setTime, set fields, recompute. With a constant 'day' part on 10M rows in America/New_York, that took ~0.2s against ~0.01s for naive TIMESTAMP.
Change
When the part is a constant of fixed wall-time length (microseconds, milliseconds, second/epoch, minute, hour, day and its aliases), bind swaps in ArithmeticTruncFunction, which works on µs directly:
Minute and above are only enabled for the gregorian calendar. Everything else (variable parts, coarser parts, other calendars, zones that don't expose their transitions) keeps the existing ICU path, whose function body is unchanged apart from a small TruncWithICU helper.
The offsets come from a new ZoneOffsets table (icu-zone-offsets.{hpp,cpp}) built at bind time from icu::BasicTimeZone transitions. It is a sorted list of constant-offset segments with sentinels at both ends. One lookup serves both instants and wall times, and a one-segment cursor skips the binary search for clustered input. Zones with recurring rules are covered up to 2250; later instants, and instants within three days of the int64 limits, fall back to ICU per row. Build rejects zones where wall times would not increase monotonically, or where an offset is a day or more. The second check is what makes a single bounds check enough to rule out overflow. The table is immutable and shared between bind data copies through a shared_ptr.
Performance (Apple Silicon, release build)
Tests
test_icu_datetrunc.test gets explicit cases (UTC, a fixed offset, Asia/Kolkata history, New York spring-forward and fall-back, a session TimeZone change between queries, a non-gregorian calendar). It also gets a differential check: a part stored in a column forces the ICU path, and every constant-part result must match it. That runs across 11 zones and the islamic-civil calendar, over ~13.5k instants covering transition windows, 1883, 1945, the 1970 epoch, the 2250 end of the table and the infinities. Deliberately introduced bugs (no hour re-resolution, an off-by-one floor, an off-by-one segment end) each make it fail.
Four micro-benchmarks under benchmark/micro/timestamp/ compare naive TIMESTAMP with TIMESTAMPTZ in UTC, a fixed offset and America/New_York.