https://slingdata.io logo
Powered by
We're testing out the sling platform. I'
# sling-help
a
We're testing out the sling platform. I'm trying to move ~300M rows from s3 to clickhouse. My job seems to be hanging.
using the sling data platform. sling project jobs trigger job_xz35xwk00llodu21
I'm running in docker on my mac m1
I did a couple of smaller tests which ran fine. Is there some kind of rough limit I should be aware of here?
Copy code
2026-04-03 17:12:00 DBG Sling version: 1.5.13 (linux amd64)
2026-04-03 17:12:00 DBG type is file-db
2026-04-03 17:12:00 DBG using: {"columns":null,"mode":"full-refresh","select":null,"transforms":null}
2026-04-03 17:12:00 DBG using source options: {"empty_as_null":false,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"max_decimals":11}
2026-04-03 17:12:00 DBG using target options: {"batch_limit":100000,"datetime_format":"auto","file_max_rows":0,"max_decimals":11,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"normalize","table_tmp":"`sling`.`riid_tmp`"}
2026-04-03 17:12:00 INF connecting to target database (clickhouse)
2026-04-03 17:12:00 DBG opened "clickhouse" connection (conn-clickhouse-clickhouse-0x9)
2026-04-03 17:12:00 INF reading from source file system (s3)
2026-04-03 17:12:00 DBG opened "s3" connection (conn-s3-bQZ)
2026-04-03 17:12:00 DBG copying 3 files from remote to /tmp/duck.temp.1775236320768.VHm for local processing
2026-04-03 17:14:10 DBG opened "file" connection (conn-file-zvN)
2026-04-03 17:14:10 DBG reading datastream from s3://raw-demo-data-evidence-studio/riid [format=parquet, nodes=3]
2026-04-03 17:14:11 DBG select * from read_parquet(['riid/riiid_train.parquet', 'riid/riiid_train_3.parquet', 'riid/riiid_train_2.parquet']) [zvN]

 
2026-04-03 17:36:44 -- Task stalled - no update received for over 45 seconds
f
Hey @Archie (Evidence.dev) looking at the internal logs, it seems the duckdb is spiking the RAM up to 100%. It might be hitting OOM error. Does your agent have swap setup? maybe that could help with the data size memory spillage. since it's duckdb processing the parquet files
a
will add some swap and retry
f
also, can you monitor the agent memory/swap usage?
a
yeah I's seeing a big spike. going to allocate more swap and try again
hmm same issue. my mac is arm64 and the image is amd64. perhaps i need to try it on a proper linux box?
f
sling already prefixes 
SET preserve_insertion_order = false;
but not memory_limit or threads
also, there is arm64 image, but linux also
slingdata/sling:latest-arm64
a
i didnt see it on dockerhub?
f
a
ah sorry being blind!
thanks, retrying
okay, it's running! 5M rows so far!
still ticking along, 90M rows!
seems like getting the right image was the critical thing
this has overall been nice intro to product. to offer one piece of feedback, I found the term "agent" to be confusing because now everyone uses the term to refer to an LLM that can call tools. in my mind I'm thinking of them as "workers"
my first 2 mins in the tool: "I dont care about using AI so I'm not going to create an agent. Now how do I setup a sync from postgres to clickhouse?"
f
Indeed, I've thought about this, was named before all LLM agent wave. Will consider more, thanks.
a
i see there is an API for checking job status. is there, or are there any plans for webhooks that fire when a job completes?
ah i see you have hooks for this
f
one feature that's not implemented yet is having the job trigger call wait for completion (by polling interval). But that'd be via sling cli though (
sling project jobs trigger --wait job_123456
). Let me know if you think that'd be useful
a
do you have a helm chart?
f
Ah no, not yet . Will look into that soon
a
4 Views
PreviousNext
Powered by