https://slingdata.io logo
I try to use sling to transfer tables
# sling-help
s
I try to use sling to transfer tables from sqlserver to fabric using bulk loading. If I login using my interactive login (az login) everything works, but i need to use a SPN to connecti to the fabric WH. Using an spn I am not able to get the COPY INTO command to work properly. The error message is: mssql: Content of directory on path 'https://XXXXXX.dfs.core.windows.net/dls-edp-test-01/slingtest/sling_temp/sling.dimsupplier_sling_tmp/1768995259092/*' cannot be listed. More information: I have configured the staging like this: abfs_endpoint: XXXXXXX.dfs.core.windows.net abfs_filesystem: dls-edp-test-01 abfs_parent: slingtest The upload of the parquet-file to the staging area works. I can restore the deleted files with azure storage explorer, and everything seems fine. However, the COPY INTO command gives the error above, even if I run it manually from SSMS logged in with my SPN. If I change the COPY INTO command by changing the URL from XXXXXXX.dfs.core.windows.net to XXXXXXX.dfs.blob.windows.net, the COPY command succeeds. If I try using the blob version as the abfs_endpoint the upload of the parquet file fails. So, it seems I need to use dfs when writing to the staging area, but blob when reading. Any ideas?
f
hey @stegus64 looked into this, and added a new property
copy_into_endpoint
that you can specify in your env.yaml entry. Should be availabe next release (see docs https://docs.slingdata.io/connections/database-connections/fabric#copy-into-endpoint-issues). Planning to release 1.5.5 tomorrow, let me know how that works out. Thanks for raising.
s
Hello! This sounds excellent! I have actually created a fork of sling and made exactly the same change. So I know this will work. Thank you for your quick attention!
A small question about the new documentation. Does this mean that the override with copy_into_endpoint is not necessary when using .dfs.core.windows.net ? https://cdn.discordapp.com/attachments/1463507768261480625/1465258335820320973/image.png?ex=697873aa&is=6977222a&hm=665772fbceba575b8450c83848d34569d2a7b28e68fdae688d4fea5a4adb7cac&
f
hey @stegus64 actually, after thinking about it, this logic was removed. The user will need to explicitly specify the
copy_into_endpoint
endpoint to replace, so that there are no surprises.
s
ok, that sounds reasonable. Maybe you should update the documentation to make it less confusing?
f
yes, already updated
s
By the way, I tried the new 1.5.5 release, and it works perfectly after configuring copy_into_endpoint. Good job!
2 Views