https://slingdata.io logo
There seems to be a bug when using
# sling-help
s
There seems to be a bug when using fileformat csv when uploading data to fabric. Sling generates a COPY INTO command like this:
Copy code
COPY INTO "sling"."dimemployee_tmp" ("WKEmployee", "EmployeeKey", "EmployeeName", "pnr", "AgreementFromDate", "AgreementToDate", "AgreementName", "AgreementHourly", "EmploymentType", "src", "period_fromdate", "period_todate", "DW_ValidFrom", "DW_ValidTo")
FROM 'XXXX/part.01.0001.csv.gz'
WITH (
  FILE_TYPE = 'CSV',
  FIELDTERMINATOR = ',',
  ROWTERMINATOR = '\n',
  FIRSTROW = 2,
  COMPRESSION = 'Gzip',
  ENCODING = 'UTF8'
)
But it always inserts 0 rows. It turns out that the syntax '\n' is not supported when reading from csv.gz, instead you need to write it like this:
Copy code
COPY INTO "sling"."dimemployee_tmp" ("WKEmployee", "EmployeeKey", "EmployeeName", "pnr", "AgreementFromDate", "AgreementToDate", "AgreementName", "AgreementHourly", "EmploymentType", "src", "period_fromdate", "period_todate", "DW_ValidFrom", "DW_ValidTo")
FROM 'XXXX/part.01.0001.csv.gz'
WITH (
  FILE_TYPE = 'CSV',
  FIELDTERMINATOR = ',',
  ROWTERMINATOR = '0x0A', // <--------- HERE
  FIRSTROW = 2,
  COMPRESSION = 'Gzip',
  ENCODING = 'UTF8'
)
This works fine.
f
thank you @stegus64 , fixed for next relaase. You can actually customize the template if you want, see https://docs.slingdata.io/concepts/replication/templates just create a
fabric.yaml
with:
Copy code
yaml
core:
  copy_from_onelake_csv: |
    COPY INTO {table} ({tgt_columns})
    FROM '{onelake_path}'
    WITH (
      FILE_TYPE = 'CSV',
      FIELDTERMINATOR = ',',
      ROWTERMINATOR = '0x0A',
      FIRSTROW = 2,
      COMPRESSION = 'Gzip',
      ENCODING = 'UTF8'{credential_expr}
    )