Hi @flarco I am trying to load a bunch of parquet files from a bunch of s3 buckets into one single redshift table and it is not working for me Here are my

env.yaml

and

replication.yaml

files and this is the debug trace from the replication run

sling run -r ortto_activities_s3_to_redshift_load.yaml --debug
2026-01-14 14:54:44 INF Sling CLI | https://slingdata.io
2026-01-14 14:54:44 DBG CLI Pro token validated
2026-01-14 14:54:44 DBG opened "s3" connection (conn-s3-LDM)
2026-01-14 14:54:44 INF Sling Replication | S3_DATA_SERVICES -> REDSHIFT_DEV | s3://datalake-external-eunorth01-ortto-norway/invalid-sms--is/
2026-01-14 14:54:44 DBG Sling version: 1.5.4 (windows amd64)
2026-01-14 14:54:44 DBG type is file-db
2026-01-14 14:54:44 DBG using: {"columns":null,"mode":"full-refresh","select":null,"transforms":null}
2026-01-14 14:54:44 DBG using source options: {"empty_as_null":false,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"max_decimals":-1}
2026-01-14 14:54:44 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"normalize","table_tmp":"\"public\".\"sms_invalid_tmp\""}
2026-01-14 14:54:44 INF connecting to target database (redshift)
2026-01-14 14:54:44 DBG opened "redshift" connection (conn-redshift-redshift_dev-2YB)
2026-01-14 14:54:44 INF reading from source file system (s3)
2026-01-14 14:54:44 DBG opened "s3" connection (conn-s3-XsD)
2026-01-14 14:54:45 INF no files found
2026-01-14 14:54:45 DBG closed "redshift" connection (conn-redshift-redshift_dev-2YB)
2026-01-14 14:54:45 INF execution succeeded

Do i need to create different s3 bucket sources for different s3 buckets?? Am i using the correct mode when loading in to same table using different streams? https://cdn.discordapp.com/attachments/1210406123744796705/1460997123331063922/ortto_activities_s3_to_redshift_load.yaml?ex=69796dda&is=69781c5a&hm=81e5bd1c9208c1e1cde3b4c9f02ff955c61e00bf56679205e9e109aa007569a1& https://cdn.discordapp.com/attachments/1210406123744796705/1460997123918139563/sample_env.yaml?ex=69796dda&is=69781c5a&hm=f2c556ddf05a9ee9be17c286ed785a382989c1bae5b6914388cea251b3e5171e&

hey @lsscodes , yes one connection per S3 bucket. and no need to put the full URI, just the path:

source: S3_DATA_SERVICES
target: REDSHIFT_DEV

env:
  SLING_THREADS: 3
  SLING_LOADED_AT_COLUMN: false

streams:
  invalid-sms--is/:
    object: 'public.sms_invalid'
    mode: full-refresh
    
    src_options:
      format: parquet
      empty_as_null: true

i have multiple buckets loading data into the same table, what should be the mode? or how would the replication.yaml look like in that case

since they come from diff buckets, you can just one replication per bucket. but if it's the same path in different buckets, I would use an env var to define your S3 connection instead of env.yaml:

export S3_DATA_SERVICES='{type: s3, bucket: slingdata-ingestion, access_key_id: ACCESS_KEY_ID, secret_access_key: "SECRET_ACCESS_KEY"}'

sling run -r replication.yaml

and just change the env var with diff bucket name. use

invalid-sms--is/

as the stream name, with the same object table name

i have different buckets but the prefix is still the "invalid-sms--is/" and i want data from different buckets into same table on the intial load and i dont have a primary key

one sec, let me test something

@flarco any update for me 😉

hey @lsscodes, apologies for the delay. yes, I was able to look at this, and you are correct that it doesn't work. This should be fixed for next release. in the meantime, you can do as I previously mentioned (re-using the same replication.yaml env providing a different source conn). Another path is to create a connection per bucket:

S3_DATA_SERVICES_NORWAY

,

S3_DATA_SERVICES_SWEDEN

, etc. and then set the source and env in your replication to:

source: 'S3_DATA_SERVICES_{REGION}'
target: REDSHIFT_DEV

env:
  REGION: ${REGION}

streams:
  invalid-sms--is/:
    object: 'public.sms_invalid_{REGION}'

next, having a pipeline.yaml:

steps:
  - type: group
    loop: [norway, sweden]
    steps:
      - replication: path/to/replication.yaml
        env:
          REGION: '{loop.value}'

  - type: query
    connection: redshift_dev
    query: truncate table exists public.sms_invalid

  - type: group
    loop: [norway, sweden]
    steps:
      - type: query
        connection: redshift_dev
        query: |
          insert into public.sms_invalid
          select * from public.sms_invalid_{loop.value};
          drop table if exists public.sms_invalid_{loop.value};

Then run this pipeline like this:

sling run -p pipeline.yaml

This should do what you want as a workaround. In next release, it will be much better with just one replication file, allowing multi-buckets for one connection:

source: S3_DATA_SERVICES
target: REDSHIFT_DEV

streams:
  sms_invalid:
    mode: full-refresh
    files:
      - s3://datalake-external-eunorth01-ortto-norway/invalid-sms--is/
      - s3://datalake-external-eunorth01-ortto-sweden/invalid-sms--is/
    object: 'public.sms_invalid'

@flarco thanks for the temp fix. i will give it a try right away and would wait for next release

ah put

replication: file://path/to/replication.yaml

, i forgot the

file://

prefix

i did a few more changes and use one single pipeline file https://cdn.discordapp.com/attachments/1460997117534273705/1463096655137079329/pipeline.yaml?ex=69709671&is=696f44f1&hm=c2d47150a43b566bb7466147e829e93e84c211bba76145ceecbc80b811569751&

yes, this is the way it works to profile the data. There are definitely optimization possible with this. The way it works now allows more flexibilty for example GCP or Azure buckets -> local temp folder -> S3 -> Redshift.

and what about this