Hello friends! I am trying to ingest delta table in S3. I am currently experiencing this error:

'PyDeltaTableError: Failed to load checkpoint: Failed to read checkpoint content: Failed to read S3 object content: Request ID: None '
           'Body: <?xml version="1.0" encoding="UTF-8"?>\n'
           '<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. '
           'Please send all future requests to this '
           'endpoint.</Message><Endpoint><http://databricks-lake-dev-us-west-2.s3-us-west-2.amazonaws.com|databricks-lake-dev-us-west-2.s3-us-west-2.amazonaws.com></Endpoint><Bucket>databricks-lake-dev-us-west-2</Bucket><RequestId>N79MDMFJ56EK9V74</RequestId><HostId>h0jvugKsFOWOKy/EPr8NELkO85lO7YQYBKR0H33LqZ7U3HkjFB2iUOM2Ne/3reGDzbzKxfEYPMg=</HostId></Error>\n'

I noticed people are experiencing similar issues with the delta-rs module when they don't pass in the correct s3 region to the DeltaTable class. I went ahead experimented with this behavior in my local Jupyter Notebook. I noticed when I pass in the correct region, my notebook will instantiate a DeltaTable object properly. When I pass the wrong region, I get the exact same error as the one I get when ingesting with Datahub. This leads me to believe that Datahub code is not handling aws region correctly. I then dug into the Datahub source code a bit and realized that the

read_delta_table

method is missing the region parameter. I believe that is the reason I am getting this failure. Can someone please confirm my suspicion? I am happy to open an issue on github or pair up any time today to investigate further! Thanks in advance!

@careful-pilot-86309 ^

@plain-guitar-45103: great catch! if you want to raise a PR we would be glad to accept that bug fix 🙂

Np, I will try and take a stab at it first. If I run into trouble I will open up an issue

@plain-guitar-45103 In my tests, even if i put region as "abc", tests pass. have you setup any other auth mechanism for aws like env variables / creds file?