I want to use chunking with incremental, and I think there is a bug.

defaults:
  mode: incremental
  object: dbo.{stream_table}
  source_options:
    chunk_size: 1000

  target_options:
    use_bulk: true
    add_new_columns: true
    adjust_column_type: true
    column_casing: source
    column_typing:
        string:
          length_factor: 2

# -----------------------------
# Streams (tables)
# -----------------------------
streams:
  dbo.M3_MITMAS:
    primary_key: ["DW_Rowid"]
    update_key: DW_Rowid
    where: "DW_rowid < 10000"

DW_rowid is an identity column, and we only insert new data inte this table, there are no updates or deletes. The table is quite large, about 9M rows. The code above is for testing small subset. Sling generates this code to calculate the number of chunks:

select min("DW_Rowid") as min_val, max("DW_Rowid") as max_val from "dbo"."M3_MITMAS"

The problem is that the where configuration above is not used in this select, which means that sling thinks there is about 9000 chunks, which causes problems. In my test I was expecting only 10 chunks with 1000 rows each.

@stegus64 thanks for reporting, fixed for next release. try to to use raw sql as a workaround for now:

yaml
streams:
  dbo.M3_MITMAS:
    primary_key: ["DW_Rowid"]
    update_key: DW_Rowid
    sql: select * from dbo.M3_MITMAS where DW_rowid < 10000

great! Thanks!