You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge?
#25692 (fixing #25690) made date_bin report Unordered output when its source argument is Timestamp(Second | Millisecond | Microsecond, _), Time64(Microsecond), or has an unknown (DataType::Null) range type. The reason is that date_bin scales these values to nanoseconds, and a per-row scaling overflow becomes NULL, so sorted input can turn into [NULL, value, NULL].
As discussed in #25692 (comment), this forces extra work for common queries over data that never gets near the overflow range:
ORDER BY date_bin(..., time) over data sorted by time re-sorts the whole input.
GROUP BY date_bin(..., time) over sorted input can no longer use sorted/streaming aggregation.
Affected today:
Direct s/ms/us timestamp and Time64(Microsecond) columns.
Not affected: date_bin(i, col) where col is Timestamp(Nanosecond, _).
The guard is also incomplete in the other direction. Nanosecond inputs can still produce per-row NULLs in the binning step, for example compute_distance computes time_delta - stride, which overflows for time_diff near i64::MIN; source - origin can overflow with an explicit origin; and month strides can go out of range. So the current rule adds sorts for coarse precisions without fully guaranteeing NULL placement for nanoseconds.
Describe the solution you'd like
Stop producing NULL for valid in-range source values, so that date_bin is monotone over non-null input for every unit, and restore ordering propagation for all timestamp/time representations:
For s/ms/us sources, compute the bin in wider arithmetic (i128) or directly in the source precision, instead of scaling to nanoseconds first and mapping overflow to NULL.
Decide on the remaining edge cases (bins that fall outside the representable range near i64::MIN/i64::MAX, explicit origins, month strides) consistently for all units. They need either an explicit ordering argument (e.g. only the extreme bins can fail, so NULLs cluster at one end) or proven input bounds.
Describe alternatives you've considered
Only give up ordering when the input range is proven to possibly overflow. Column ranges are currently unbounded in ExprProperties, so this needs bounds propagation from statistics first.
A config flag to disable the guard. A flag that trades correctness for performance is hard to reason about, so I'd prefer fixing the arithmetic.
Is your feature request related to a problem or challenge?
#25692 (fixing #25690) made
date_binreportUnorderedoutput when its source argument isTimestamp(Second | Millisecond | Microsecond, _),Time64(Microsecond), or has an unknown (DataType::Null) range type. The reason is thatdate_binscales these values to nanoseconds, and a per-row scaling overflow becomesNULL, so sorted input can turn into[NULL, value, NULL].As discussed in #25692 (comment), this forces extra work for common queries over data that never gets near the overflow range:
ORDER BY date_bin(..., time)over data sorted bytimere-sorts the whole input.GROUP BY date_bin(..., time)over sorted input can no longer use sorted/streaming aggregation.Affected today:
Time64(Microsecond)columns.date_bin(i, date_trunc('hour', ts)). perf: preserve scalar UDF output types in property analysis #25668 would give these a typed range, which recovers the nanosecond case.Not affected:
date_bin(i, col)wherecolisTimestamp(Nanosecond, _).The guard is also incomplete in the other direction. Nanosecond inputs can still produce per-row
NULLs in the binning step, for examplecompute_distancecomputestime_delta - stride, which overflows fortime_diffneari64::MIN;source - origincan overflow with an explicit origin; and month strides can go out of range. So the current rule adds sorts for coarse precisions without fully guaranteeing NULL placement for nanoseconds.Describe the solution you'd like
Stop producing
NULLfor valid in-range source values, so thatdate_binis monotone over non-null input for every unit, and restore ordering propagation for all timestamp/time representations:NULL.i64::MIN/i64::MAX, explicit origins, month strides) consistently for all units. They need either an explicit ordering argument (e.g. only the extreme bins can fail, soNULLs cluster at one end) or proven input bounds.Describe alternatives you've considered
ExprProperties, so this needs bounds propagation from statistics first.Additional context
evaluate_bounds.date_binbounds, which made the original bug visible throughdate_trunc.