https://slingdata.io logo
Powered by
Replication overriding explicit column types
# sling-help
m
Hello @flarco I'm running a replication from an API spec, where I'm explicitly specifying the columns with their types in the stream map. I have the column 
system_sku: string
that for some reason is being converted to an integer (I guess that's the type Sling inferred from the first 1,000 or so records). The replication is failing because the cast fails, but I don't understand why it is performing the cast if I'm telling it is a string column. The records are inserted without issue to the temp table, but the problem is when inserting to the target table from the temp table. In the following comments are the relevant bits from the debut output (edited for brevity, and chunked because of Discord message length limit).
Copy code
2026-03-03 01:21:18 DBG using: {"columns":[{"name":"item_id","type":"integer"},{"name":"system_sku","type":"string"},{"name":"description","type":"string"},{"name":"time_stamp","type":"datetime"}],"mode":"incremental","select":null,"transforms":null}
2026-03-03 01:21:18 DBG using source options: {"empty_as_null":true,"datetime_format":"AUTO","max_decimals":-1}
2026-03-03 01:21:18 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":false,"adjust_column_type":false,"column_casing":"normalize","table_tmp":"\"main\".\"item_sling_duckdb_tmp\""}

2026-03-03 01:21:29 DBG casting column 'description' as 'string'
2026-03-03 01:21:29 DBG casting column 'item_id' as 'integer'
2026-03-03 01:21:29 DBG casting column 'system_sku' as 'string'
2026-03-03 01:21:29 DBG casting column 'time_stamp' as 'datetime'

2026-03-03 01:21:29 DBG create temp table if not exists "main"."item_sling_duckdb_tmp" ("description" text,
  "item_id" integer,
  "system_sku" text,
  "time_stamp" datetime)

2026-03-03 01:21:29 DBG insert into "main"."item_sling_duckdb_tmp" ("description", "item_id", "system_sku", "time_stamp") select * from read_csv('http://localhost:49422/data', delim=',', header=True, columns={'description':'text', 'item_id':'integer', 'system_sku':'text', 'time_stamp':'datetime'}, max_line_size=2000000, parallel=false, quote='"', escape='"', nullstr='\N', auto_detect=false)
Sling pages through the API for ~2 minutes getting ~12,000 rows, and then:
Copy code
2026-03-03 01:23:43 DBG select count(*) cnt from "main"."item_sling_duckdb_tmp"
2026-03-03 01:23:44 DBG inserting system_sku [VARCHAR] into system_sku [BIGINT]
2026-03-03 01:23:44 DBG inserting time_stamp [TIMESTAMP] into time_stamp [TIMESTAMP WITH TIME ZONE]
2026-03-03 01:23:44 DBG insert into "lightspeed"."item" ("description", "item_id", "system_sku", "time_stamp", "upc") select "description" as "description", "item_id" as "item_id", case when "system_sku" = 'true' then 1 when "system_sku" = 'false' then 0 else cast("system_sku" as integer) end as "system_sku", "time_stamp"::TIMESTAMP WITH TIME ZONE as "time_stamp" from "main"."item_sling_duckdb_tmp"
2026-03-03 01:23:44 DBG drop table if exists "main"."item_sling_duckdb_tmp"
2026-03-03 01:23:44 DBG table "main"."item_sling_duckdb_tmp" dropped
2026-03-03 01:23:44 INF execution failed

Perhaps setting a higher 'SLING_SAMPLE_SIZE' environment variable could help? This represents the number of records to process in order to infer column types (especially for file sources). The default is 900. Try 2000 or even higher.
You can also manually specify the column types with the `columns` input, (see https://docs.slingdata.io/concepts/replication/columns)
Furthermore, you can try the `target_options.adjust_column_type` setting to allow Sling to automatically alter the column type on the target side.

fatal:
--- proc.go:285 main ---
--- sling_cli.go:491 main ---
--- sling_cli.go:527 cliInit ---
--- cli.go:287 CliProcess ---
~ failure running replication (see docs @ 
https://docs.slingdata.io/sling-cli)
--- sling_run.go:289 processRun ---
And this is the final error message:
Copy code
--------------------------- item ---------------------------
~ Failed to execute SQL
--- duckdb.go:906 1 ---
Conversion Error: Could not convert string '210000000085' to INT32 when casting from source column system_skuLINE 1: ..." = 'true' then 1 when "system_sku" = 'false' then 0 else cast("system_sku" as integer) end as "system_sku", "tim...                                                                     ^
For query => insert into "lightspeed"."item" ("description", "item_id", "system_sku", "time_stamp", "upc") select "description" as "description", "item_id" as "item_id", case when "system_sku" = 'true' then 1 when "system_sku" = 'false' then 0 else cast("system_sku" as integer) end as "system_sku", "time_stamp"::TIMESTAMP WITH TIME ZONE as "time_stamp" from "main"."item_sling_duckdb_tmp"
So I found out the issue. The table already existed in the target DB and was trying to cast the values to the existing columns' data type. Solution was droping the DB and recreating from scratch.
2 Views
PreviousNext
Powered by