https://slingdata.io logo
Powered by
# sling-help
  • Hi! Im replicating Postgres table to
    i
    Hi! Im replicating Postgres table to Clickhouse, in my setup sling automatically configure target table with NULL-able columns, is there a way to control that (which column is NUllbale or not) ? I tried this, but stil see colums Nullable(T) in click
    Copy code
    streams:
  bi.bi_tasks:
    mode: full-refresh
    primary_key: id
    update_key: create_ts
    columns:
      id: uuid | value is not null
      create_ts: timestamptz | value is not null
      is_deleted: bool | value is not null
    f
    • 2
    • 4
  • Good Evening. I have a MySQL database
    a
    Good Evening. I have a MySQL database with a mediumblob column containing the binary contents of a PDF file. I am using sling to move this to another MySQL database, however it seems that sling converts this to mediumtext column. I believe a step or process within sling is corrupting the data - my measure for this, is that on the original database I can select the contents of the mediumblob column into a dumpfile and open it as a valid PDF, but if I try the same on the destination database, whilst the PDF opens and has the correct number of pages, there is no contents.
    f
    • 2
    • 5
  • Hi, In our ETLs the customer wants a
    i
    Hi, In our ETLs the customer wants a column of timestamp for all the tables target, sling has SLING_EXEC_ID_COLUMN . but that is a UUID . If I understand correctly SLING_LOADED_AT_COLUMN is the now() of each insert / update . is there a variable of the exec_id_timestamp ? the begin timestamp ? the end timestamp ? Thanks
    f
    • 2
    • 2
  • Hi, i am working on ELT to postgres -
    i
    Hi, i am working on ELT to postgres - looking at this example : I have a unique key that is composite of 2 columns . If I understand correctly - this will create two indexes: 1. One for col1 and One for col2 my_schema.another_table: target_options: table_keys: index: [col1, col2]
    f
    • 2
    • 12
  • suggestion: a new mode:
    i
    suggestion: a new mode: mode: delete insert (in transaction) : begin; delete from stg.small_dim; insert into stg.small_dim select ... from source; commit; Why? Because DELETE does not block normal SELECT in the same aggressive way as TRUNCATE. If we need to query the STG while ELT is working, in our case it is every 15 minutes . if we need to verify or dbt starts with large transformation it can get locked. to put it in pre hook means the delete is not in transactions. What do you think ?
    f
    • 2
    • 7
  • h
    Hi guys, I use sling 1.5.19, and when I try to sync with hubspot I got `could not fetch spec 
    hubspot-v3
    (status: 400) => Missing CLI token or API key` I used my key to access API via curl and all works fine. What am I doing wrong?
    Copy code
    $ cat .env.sling | sed 's/na1.*"/na1-...."/' | grep HUB
HUBSPOT='{ type: api, spec: hubspot-v3, secrets: { access_token: "pat-na1-...." } }'
$ sling conns test MY_PG
5:56PM INF success!
$ sling conns test HUBSPOT
fatal:
~ could not fetch api spec: hubspot-v3
could not fetch spec `hubspot-v3` (status: 400) => Missing CLI token or API key
  • h
    Oh, I see what is happening, this is not free stuff. Okay, thanks.
  • j
    .
  • Hi, i am trying to create an incremental
    i
    Hi, i am trying to create an incremental that brings data until the start date of the sling job . Because our process brings many tables and it takes several minutes We don't want the fact tables to bring data that was written after the beginning of the job. how can we get sling_loaded_at ? If we use pipeline it will be the same value for all replicas? sql: | SELECT * FROM {stream_table} WHERE updated_at > COALESCE({incremental_value}, '1900-01-01 00:00:00') - INTERVAL 2 HOUR and updated_at < '{env._sling_loaded_at}' Tried it in several ways : put the env at the bottom of the file , '{{env._sling_loaded_at}}', and a hook. hooks: start: - type: store key: env._sling_loaded_at value: "{timestamp.YYYY}-{timestamp.MM}-{timestamp.DD} {timestamp.HH}:{timestamp.mm}:{timestamp.ss}"
    f
    • 2
    • 4
  • maria db has connection strings vars
    i
    maria db has connection strings vars that can be set. for example : tinyInt1isBit=false , zeroDateTimeBehavior=convertToNull, characterEncoding=UTF-8, useUnicode=true where can i add this to the connection string ? I think that it is better to bring it right from the source instead of transform it . no ?
    f
    • 2
    • 1
  • Hi, question about variables. i have a
    i
    Hi, question about variables. i have a table with a list of tables and parameters , for example limit and hours_back . so one table can be SELECT * FROM {stream_table} WHERE updated_at > COALESCE({incremental_value}, '1900-01-01 00:00:00') - INTERVAL {hours_back} HOUR and updated_at < '{env._sling_loaded_at}' limit {limit} i am working on incremental for 10 tables, i put this on the default of the replication. with a pre hook that bring the variables from the params_table on the target how can i load the variables and use them in the replication ? i managed to do it In a preceding pipeline but then I lose the flexibility of running only one stream (and still get its variables) If it is not possible , is there a way to load the table to environment variables before the replication and then use them in the replication ? Some other option ?
    f
    • 2
    • 2
  • Hi, when using select: ['**', '-column1
    i
    Hi, when using select: ['**', '-column1','-column2' ] then i use {fields} in the SQL . how can i specify the tables ? because the sql has 3 joins but i don't want any columns from those. and if i wanted only 1 column from the join in the select . how can i specify from which table the columns are? i tried to put table_name."*" same for -table_name2.'*' . of course i always can specify the columns manually but its missing the globs that are great.
    f
    • 2
    • 1
  • Hi @flarco
    m
    Hi @flarco I am currently trying to configure sling for use with Jira. Using the sling provided api spec, I am getting the following error:
    Copy code
    --- spec.go:105 LoadSpec ---
~ queue validation failed
--- spec.go:180 validateQueues ---
undeclared queue(s) found:
  - queue 'board_ids' used by endpoint(s): sprints, boards
  - queue 'issue_keys' used by endpoint(s): issue_changelogs, issues, issue_comments, issue_worklogs
  - queue 'project_keys' used by endpoint(s): project_components, projects, project_versions

Please declare them in the 'queues' section at the spec level
    I have attempted to add them to the spec level, but the documentation isn't clear on exactly how to go about this. Are you able to provide any additional clarity on this? Thank you! <3
    f
    • 2
    • 2
  • Hi, about on failure . i did some tests
    i
    Hi, about on failure . i did some tests and i don't exactly understand the behavior . _case 1: _ replication with a table source that doesn't exists. - result | Error 1146 (42S02): Table 'bi_Assets_1' doesn't exist bi_Assets_1 | bi_Assets_1 | fatal: 12:05PM INF Sling Replication Completed in 16s |DASH -> target_stg | 28 Successes | 1 Failures 12:05PM WRN failed streams: ["bi_Assets_1"] fatal: ~ failure running replication (see docs @ https://docs.slingdata.io/sling-cli) Replication failure (1 streams failed). it didn't fail in the middle or stopped the replication. also i write the log to a table with '{run.status}', and {run.stream.table} and this stream doesn't exist in the log, I expected to have the source stream that doesn't exist. with an error. _case 2: _ I went on and created a pipeline that runs this replication, with on_failure: abort it did the same thing. didn't abort . didn't fail the run bi_Assets_1 | fatal: bi_Assets_1 | ~ SQL Error bi_Assets_1 | Error 1146 (42S02): Table 'Assets_1' doesn't exist 12:15PM INF Sling Replication Completed in 16s | DASH -> target_stg | 28 Successes | 1 Failures 12:15PM WRN failed streams: ["bi_Assets_1"] fatal: ~ error running replication Replication failure (1 streams failed) no log, no abort. _case 3: _ I created a loop over 4 tables in the target DB and send the names as parameters to the replication streams: - "{loop.value.table_name}" i put there a table that doesn't exist as stream in the replication . I didn't get any error. the process ignored it completely. I would expect at least a warning about the stream I try to run that doesn't exist. How exactly should the abort work? I thought it will stop the process all together. Especially if it is a pipeline with abort.
    f
    • 2
    • 3
  • Hi @flarco
    m
    Hi @flarco We are in the process of doing some BQ -> BQ work and have run into an interesting issue. We don't seem to be able to get nested schemas working. Usually we do 
    dataset_name.*
    but this time we need to something like 
    dataset_group.dataset_name.*
    and we are getting all kinds of trouble.
    Copy code
    sling run \
                    --src-conn <company>-bigquery-prod \
                    --src-stream "<dataset_group>.events_${CURRENT_DATE}" \
                    --tgt-conn <project>-bigquery-prod \
                    --tgt-object "<app>_analytics_${CURRENT_YEAR}.sftp_logs_${CURRENT_DATE}" \
                    --mode full-refresh
    Copy code
    <company>-bigquery-prod:
        type: bigquery
        project: <project>
        location: EU
        gc_bucket: <company>-sling-temp-bucket
        key_body: 'redacted'
    and this gives us
    Copy code
    fatal:
~ Could not unload.
~ Error in Export Task
{Location: ""; Message: "Operation cannot be performed on a nested schema. Field: event_params"; Reason: "invalid"}
    f
    • 2
    • 3
  • Hi, we use pipeline with a loop from a
    i
    Hi, we use pipeline with a loop from a database query. each table gets several variables from the table row . for each row from the table we run its own stream on the replication how can we run specific tables manually ? We dont want to change the main is_active columns because it serves the scheduled process. 1. best option is if there is a --tag or --stream name ? 2. fails: when one of the streams fail with abort the all pipeline stops . I prefer the behavior of replication that it will fail the specific stream and at the end the status of the pipeline will be failed. for now we put warn. the process is success even if one of the stream failed and then we need to query the log and find the error to alert the developers. thanks steps: # All active medium tables - type: query id: cfg connection: target_stg query: | SELECT table_name, cycle_limit, hours_back, cutoff FROM zstg.vw_z_refresh WHERE is_active = 1 and size_group = 'medium' ORDER BY table_name - type: group id: refresh loop: state.cfg.result steps: - type: replication path: elts/fin/fin_inc.yaml streams: - "{loop.value.table_name}" env: CUTOFF: "{loop.value.cutoff}" HOURS_BACK: "{loop.value.hours_back}" CYCLE_LIMIT: "{loop.value.cycle_limit}" on_failure: warn
    f
    • 2
    • 7
  • Question about sling platform - Feature
    m
    Question about sling platform - Feature to receive alert notification (slack, email) on source connection schema change, is available in standard or advanced subscription?
    f
    • 2
    • 2
  • Hello,
    e
    Hello, We are trying to use SLING_STATE. However, when Sling writes the latest timestamp to the state (file or table), it generates an invalid timestamp format, as shown below. “2026-08-05 14:45:00 +0200 +02:00” The corresponding log entry is as follows: 2026-08-06 07:30:56 DBG insert into “l0_sling”.“sling” (“source_stream”, “target_object”, “value”, “column_type”) values (‘HO_QLIKSENSE. VS_H4M_BODEN_T’, ‘“l0_sling” p52_ho2qlik_vs_h4m_boden_t”, ‘2026-08-06 06:45:00 +0200 +02:00’, ‘timestamp’) [TAPT_SLING-H8d] The source is an Oracle database, and the field is of type DATE
    f
    • 2
    • 1
  • s
    message has been deleted
  • g
    message has been deleted
  • m
    message has been deleted
  • Hi everyone, can anyone help me
    m
    Hi everyone, can anyone help me understand why my job on the Sling platform has an execution status of "Queued"?
    f
    • 2
    • 7
  • We are trying to use Sling CLI-Pro to do
    j
    We are trying to use Sling CLI-Pro to do CDC from AWS RDS Postgres to Snowflake. Unfortunately, with AWS RDS we don't have superuser privledges, so we can issue an ALTER ROLE sling_user REPLICATION; How can we do CDC from RDS?
    f
    • 2
    • 2
  • I'm getting this (attached) error when
    j
    I'm getting this (attached) error when trying to run CDC using CLI PRO. We DO have UUIDs in our Postgres DB, but they can be just treated as strings, we don't care. https://cdn.discordapp.com/attachments/1210406123744796705/1548052978794102804/message.txt?ex=6aa5a82f&is=6aa456af&hm=e5827819a5e570f8037d7fee3fd37d82fb9f82b3fee657c5569de2405986a704&
    f
    • 2
    • 2
  • Also, assuming we can get everything CDC
    j
    Also, assuming we can get everything CDC'ing out of our Postgres into Snowflake, we will definitely become a paying customer. It WOULD be convenient if you guys just had a way to pay for the CLI through the AWS Marketplace (they have SASS style stuff there, so I think you could do it). Our company is fine paying for things through AWS Marketplace, but it's difficult to get the company to expense on a credit card.
    f
    • 2
    • 3
  • Hi, can I use the paid sling pro CLI in
    j
    Hi, can I use the paid sling pro CLI in an AWS subnet with no internet access? Does it need an outside connection to auth the pro licence?
    f
    • 2
    • 2
  • `SLING_SCHEMA_MIGRATION` makes invalid
    i
    SLING_SCHEMA_MIGRATION
    makes invalid DDL for a StarRocks target (v1.6.3) Hi! We use Sling CLI 1.6.3 (official build, Pro trial) from Postgres to StarRocks. With 
    SLING_SCHEMA_MIGRATION=primary_key
    , Sling finds the source primary key, but StarRocks rejects the DDL:
    Copy code
    create table ... (`_sling_row_id` varchar(65533), `id` bigint, ..., PRIMARY KEY (`id`)) distributed by hash(`_sling_row_id`)
    Error 1064: Getting syntax error at line 5, column 2 With 
    all
    , 
    DEFAULT uuid()
    and 
    DEFAULT current_timestamp NOT NULL
    also fail. Do you plan StarRocks support for schema migration? We want to sync all tables of a source without a primary_key for each stream.
    f
    • 2
    • 3
  • Same Sling 1.6.3 CLI (Pro trial). The
    i
    Same Sling 1.6.3 CLI (Pro trial). The source is Postgres and the target is StarRocks (shared-data mode). With mode: 
    incremental
    , an explicit 
    primary_key
    , and 
    target_options.delete_missing: soft
    , every stream fails after the load: ~ could not delete missing records could not find column id in table 
    sling_spike
    .
    public_orders
    The column exists in the target table. The same replication without 
    delete_missing
    succeeds. Is 
    delete_missing
    supported for StarRocks? Or does the trial token limit it? I can send the full replication YAML and the log.
    f
    • 2
    • 3
  • **`delete_missing: soft` does not
    i
    delete_missing: soft
    does not clear 
    _sling_deleted_at
    when a row comes back (v1.6.4, StarRocks)     Sling 1.6.4 (official build, Pro) copies from Postgres to StarRocks (shared-data mode). Each stream has 
    mode: incremental
    , an explicit 
    primary_key
    , an 
    update_key
    , and 
    target_options.delete_missing: soft
    . Steps: 1. Run the replication. All rows load. 2. Delete the row 
    id = 4
    in Postgres, and run again. Sling sets 
    _sling_deleted_at
    on that row. This is correct. 3. Insert the row 
    id = 4
    again, with a new 
    updated_at
    , and run again. Expected: the upsert writes the row, and 
    _sling_deleted_at
    is 
    NULL
    . Actual: the upsert writes the row, but 
    _sling_deleted_at
    keeps the old value. So the row stays marked as deleted. Is this intended? Or can the upsert clear the mark? I can send the replication YAML and the log.
    f
    • 2
    • 2
  • **StarRocks stream load fails when a
    i
    StarRocks stream load fails when a column name is a reserved word (Sling 1.6.4) If a source column has a name that StarRocks reserves (for example 
    default
    , 
    key
    , or 
    rows
    ), the stream load fails. StarRocks rejects the load before it reads data (LoadBytes: 0). Sling then marks the stream as failed. Cause: StreamLoad in core/dbio/database/database_starrocks.go removes the quote characters from each column name:
    Copy code
    return strings.ReplaceAll(conn.Quote(name), q, "")
    Then it sends the bare names in the "columns" header. StarRocks cannot parse a reserved word there. To reproduce: 1. Make a Postgres table with a column named "default". 2. Copy it to StarRocks with fe_url set and use_bulk: true. The INSERT path (no fe_url) quotes the names correctly. Can you keep the backticks in the "columns" header? Or is there a reason to remove them? Thanks!
    f
    • 2
    • 5