https://www.getdaft.io logo
Join Slack
Powered by
PR ready for review adding support for pyiceberg v...
# daft-dev
g
PR ready for review adding support for pyiceberg v0.11.0 cc @Desmond Cheong https://github.com/Eventual-Inc/Daft/pull/6200
Looks like 
0.11.0
has some incompatibilities with pyarrow 
8.0.0
that is in the test matrix
d
😮 these version incompatibilities are such a pain...
g
technically, 
pyiceberg
has always had a minimum of 
9.0.0
for awhile, now set to 
17.0.0
in latest version. maybe because it is an optional dep, this doesn't create install issues 🤷
8.0.0
is also pretty old from 2022, do y'all have some users in environments that can't upgrade easily?
d
Hm if 9.0.0 is the new minimum we should just follow that too
g
looking back to pyiceberg 
0.5.x
, that was the minimum https://github.com/apache/iceberg-python/blob/891b4c7f4214fb9118080ce2215a210a770a5019/pyproject.toml#L60. I think if people want to effectively use 
daft
with 
pyiceberg
, they need a newer version
for the test matrix, assuming using 
9.0.0
, I think there will still need to be a conditional to install an older version of 
pyiceberg
for that case
which will also require a conditional in some of the changes in the PR above based on the pyiceberg version
nvm, looks like things might work by just moving to 
9.0.0
, testing.... the original version range on the dev version used for testing is 
pyarrow = { version = ">=9.0.0,<18.0.0", optional = true }
, which includes the minimum version required in latest 
pyiceberg
Got everything working by ensuring pyiceberg version is compatible with pyarrow version
4 Views
Open in Slack
PreviousNext
Powered by