Trigger A: Existing Delta Table Has a Different Scale (Highest Likelihood)

Your delta merge reads the existing Delta table as DECIMAL(38, S) — written previously by Spark (which frequently infers DECIMAL(38, 18) from data), by a prior Daft version, or via a different connector. The new MySQL data arrives as DECIMAL(38, 20).

Daft's supertype computation (supertype.rs:204):

Some(DataType::Decimal128(max(p1, p2), max(s1, s2)))

// DECIMAL(38, 18) + DECIMAL(38, 20) → DECIMAL(38, 20)

This forces old rows to be upscaled from S=18 to S=20. Any row with |value| > i128::MAX / 10² (≈ 1.7×10^36 internal, ≈ 1.7×10^18 actual) will overflow → NULL. Since DECIMAL(38, 18) allows values up to 10^20, roughly 98% of the range (10^18 to 10^20) is the danger zone.

Trigger B: PyArrow Scale Inference at MySQL Read Time

Python MySQL connectors (mysql-connector-python, PyMySQL) return decimal.Decimal objects with trailing zeros stripped. For example, a MySQL DECIMAL(38, 20) value 1000000000000000000.00000000000000000000 comes back as Decimal('1000000000000000000') (scale=0).

PyArrow infers the type from the actual values per partition. If a Ray partition happens to contain only integer-valued decimals (scale=0 after stripping), PyArrow creates DECIMAL(19, 0) for that partition. When Daft reconciles with other partitions (via supertype) or you call .cast(DataType.decimal128(38, 20)), the upscaling is fallible.

Overflow threshold for scale=0 → scale=20: |value| > 10^18 (which equals the precision cap of DECIMAL(38, 20) itself, meaning ALL values near the maximum are at risk).