https://linen.dev logo
Join Slack
Powered by
# best-practices
  • s
    Hi Team, Hope you are doing well. I am getting one issue when I am using tap-postgres and target-jsonl for my custom connector. Can anyone please help. > _(meltanoEnv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend> meltano invoke tap-postgres_ > 2025-03-21T081517.883761Z [warning ] Failed to create symlink to 'meltano.exe': administrator privilege required > 2025-03-21T081517.917544Z [info ] Environment 'dev' is active > 2025-03-21 134520,737 | INFO | tap-postgres | Skipping deselected stream 'public-accounts'. > _(meltanoEnv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>_ > > meltano=# \dp public.accounts > Access privileges > Schema | Name | Type | Access privileges | Column privileges | Policies > --------+----------+-------+----------------------------+-------------------+---------- > public | accounts | table | postgres=arwdDxtm/postgres | | > (1 row) > > meltano=# > > Below is my meltano.yml configuration. > - name: tap-postgres > namespace: tap_postgres > pip_url: ./connectors/tap-postgres > executable: tap-postgres > config: > database: meltano > host: localhost > port: 5432 > user: postgres > password: ******** > filter_schemas: [public] > select_all_tables: true # Or false, and use a selection JSON > select_all_fields: true # If false, use a selection JSON to specify fields > sqlalchemy_url: "postgresql://postgres:"******"@localhost:5432/meltano" > > meltano=# > 2025-03-21 134520,737 | INFO | tap-postgres | Skipping deselected stream 'public-accounts'. > _(meltanoEnv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>_
    r
    • 2
    • 5
  • p
    In this public article I summarized my performance tests of Meltano. It would be much appreciated if people who are more experienced looked at what I've done and commented whether the numbers quoted make sense. I tested throughput of Oracle and Postgres connection in various types of data. https://medium.com/@pp_85623/testing-performance-of-elt-data-ingest-meltano-oracle-and-postgresql-471989ff82df
  • s
    Hi @Everyone, Good Day ! I am trying to run a meltano pipeline using environment variables. So I set the env variables $env:MELTANO_SETTING_TAP_POSTGRES_DATABASE="meltano"; $env:MELTANO_SETTING_TAP_POSTGRES_HOST="localhost"; $env:MELTANO_SETTING_TAP_POSTGRES_PORT="5432"; $env:MELTANO_SETTING_TAP_POSTGRES_USER="postgres"; $env:MELTANO_SETTING_TAP_POSTGRES_PASSWORD="****"; $env:MELTANO_SETTING_TAP_POSTGRES_FILTER_SCHEMAS="public"; $env:MELTANO_SETTING_TAP_POSTGRES_SQLALCHEMY_URL="postgresql://postgres:siba1234@localhost:5432/meltano" and commented the config sections in the meltano.yml file, then I locked the plugin. When I executed 2025-04-02T034115.605289Z [warning ] Failed to create symlink to 'meltano.exe': administrator privilege required 2025-04-02T034115.616596Z [info ] The default environment 'dev' will be ignored for 
    meltano config
    . To configure a specific environment, please use the option 
    --environment=<environment name>
    . (meltanoEnv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend> I am getting no values. Is it the right way to set configuration if I want to use env variables. The purpose is not to store the config parameters in meltano.yml file. note: there is no .env file under my meltano project root directory. Thanks in advance !
    r
    v
    • 3
    • 7
  • s
    Hi Team, Getting these warnings while running an ELT pipeline. Can anyone suggest what needs to be done from my side. 
    2025-04-02T07:37:31.279724Z [warning  ] Certificate did not match expected hostname: <http://sp.meltano.com|sp.meltano.com>. Certificate: {'subject': ((('commonName', '_.<http://ops.snowcatcloud.com|ops.snowcatcloud.com>'),),), 'issuer': ((('countryName', 'US'),), (('organizationName', 'Amazon'),), (('commonName', 'Amazon RSA 2048 M02'),)), 'version': 3, 'serialNumber': '0668C0E7C8CD0F1A31A21E5DDD2FD67D', 'notBefore': 'Mar  7 00:00:00 2025 GMT', 'notAfter': 'Apr  5 23:59:59 2026 GMT', 'subjectAltName': (('DNS', '_.<http://ops.snowcatcloud.com|ops.snowcatcloud.com>'),), 'OCSP': ('*<http://ocsp.r2m02.amazontrust.com>*',), 'caIssuers': ('*<http://crt.r2m02.amazontrust.com/r2m02.cer>*',), 'crlDistributionPoints': ('*<http://crl.r2m02.amazontrust.com/r2m02.crl',)>}*
    2025-04-02T07:37:31.283308Z [warning  ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=0)) after connection broken by 'SSLError(CertificateError("hostname '<http://sp.meltano.com|sp.meltano.com>' doesn't match '_.<http://ops.snowcatcloud.com|ops.snowcatcloud.com>'"))': /com.snowplowanalytics.snowplow/tp2_
    _2025-04-02T073732.200577Z [warning ] Certificate did not match expected hostname: sp.meltano.com. Certificate: {'subject': ((('commonName', '_`.ops.snowcatcloud.com'),),), 'issuer': ((('countryName', 'US'),), (('organizationName', 'Amazon'),), (('commonName', 'Amazon RSA 2048 M02'),)), 'version': 3, 'serialNumber': '0668C0E7C8CD0F1A31A21E5DDD2FD67D', 'notBefore': 'Mar 7 000000 2025 GMT', 'notAfter': 'Apr 5 235959 2026 GMT', 'subjectAltName': (('DNS', '_.ops.snowcatcloud.com'),), 'OCSP': ('*http://ocsp.r2m02.amazontrust.com*',), 'caIssuers': ('*http://crt.r2m02.amazontrust.com/r2m02.cer*',), 'crlDistributionPoints': ('*http://crl.r2m02.amazontrust.com/r2m02.crl',)}*_` 
    _2025-04-02T07:37:32.204045Z [warning  ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=0)) after connection broken by 'SSLError(CertificateError("hostname '<http://sp.meltano.com|sp.meltano.com>' doesn't match '_.<http://ops.snowcatcloud.com|ops.snowcatcloud.com>'"))': /com.snowplowanalytics.snowplow/tp2
    2025-04-02T073733.138758Z [warning ] Certificate did not match expected hostname: sp.meltano.com. Certificate: {'subject': ((('commonName', '`_.ops.snowcatcloud.com'),),), 'issuer': ((('countryName', 'US'),), (('organizationName', 'Amazon'),), (('commonName', 'Amazon RSA 2048 M02'),)), 'version': 3, 'serialNumber': '0668C0E7C8CD0F1A31A21E5DDD2FD67D', 'notBefore': 'Mar 7 000000 2025 GMT', 'notAfter': 'Apr 5 235959 2026 GMT', 'subjectAltName': (('DNS', '_.ops.snowcatcloud.com'),), 'OCSP': ('*http://ocsp.r2m02.amazontrust.com*',), 'caIssuers': ('*http://crt.r2m02.amazontrust.com/r2m02.cer*',), 'crlDistributionPoints': ('*http://crl.r2m02.amazontrust.com/r2m02.crl',)}*` 
    2025-04-02T07:37:33.141577Z [warning  ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=0)) after connection broken by 'SSLError(CertificateError("hostname '<http://sp.meltano.com|sp.meltano.com>' doesn't match '_.<http://ops.snowcatcloud.com|ops.snowcatcloud.com>'"))': /com.snowplowanalytics.snowplow/tp2_
    _2025-04-02T073734.072331Z [warning ] Certificate did not match expected hostname: sp.meltano.com. Certificate: {'subject': ((('commonName', '_`.ops.snowcatcloud.com'),),), 'issuer': ((('countryName', 'US'),), (('organizationName', 'Amazon'),), (('commonName', 'Amazon RSA 2048 M02'),)), 'version': 3, 'serialNumber': '0668C0E7C8CD0F1A31A21E5DDD2FD67D', 'notBefore': 'Mar 7 000000 2025 GMT', 'notAfter': 'Apr 5 235959 2026 GMT', 'subjectAltName': (('DNS', '*.ops.snowcatcloud.com'),), 'OCSP': ('*http://ocsp.r2m02.amazontrust.com*',), 'caIssuers': ('*http://crt.r2m02.amazontrust.com/r2m02.cer*',), 'crlDistributionPoints': ('*http://crl.r2m02.amazontrust.com/r2m02.crl',)}*` 
    (meltanoEnv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
    a
    p
    r
    • 4
    • 8
  • s
    Hi Team, I am trying to create a pipeline from tap-snowflake to target-salesforce. Below is the snippet of my root meltano.yml file for both tap-snowflake and target-salesforce.
    Copy code
    - name: tap-snowflake
    namespace: tap_snowflake
    pip_url: ./connectors/tap-snowflake
    executable: tap-snowflake
    capabilities:
    - state
    - catalog
    - discover
    - about
    - stream-maps
    settings:
    - name: account
      kind: string
      value: aigoiop-hq79023
      description: The Snowflake account identifier.
    - name: user
      kind: string
      value: udey
      description: The Snowflake username.
    - name: password
      kind: string
      value: ***********
      description: The Snowflake password.
      sensitive: true
    - name: database
      kind: string
      value: PARTHAN_DB
      description: The Snowflake database name.
    - name: warehouse
      kind: string
      value: COMPUTE_WH
      description: The Snowflake warehouse name.
    select:
    - public-account.* 
-------------------------------------------------------------------
  - name: target-salesforce
    namespace: target_salesforce
    pip_url: ./connectors/target-salesforce
    executable: target-salesforce
    capabilities:
    - about
    - stream-maps
    - schema-flattening
    config:
      username: udey@vcs.sandbox
      password: ********
      security_token: nanLbbN3lexEw70gK7tLrzP4s
      api_type: sandbox
      #sobject: account
      action: insert
      stream_maps:
        public-employees:   
          target: Account     
          #key_properties: []   
          mappings:
          - source: name      
            target: Name
    I am getting an error as below for this stream_maps.
    Copy code
    2025-04-02T12:36:00.006657Z [info     ]        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-salesforce name=target-salesforce producer=False run_id=a2dbeee1-5078-41a8-a7d9-aba40dd70d46 stdio=stderr string_id=target-salesforce
2025-04-02T12:36:00.008151Z [warning  ] Received state is invalid, incremental state has not been updated
2025-04-02T12:36:00.098874Z [info     ] Incremental state has been updated at 2025-04-02 12:36:00.098808+00:00.
2025-04-02T12:36:00.100101Z [info     ] TypeError: unhashable type: 'list' cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-salesforce name=target-salesforce producer=False run_id=a2dbeee1-5078-41a8-a7d9-aba40dd70d46 stdio=stderr string_id=target-salesforce
2025-04-02T12:36:00.220899Z [error    ] Loader failed
2025-04-02T12:36:00.221982Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Loader failed
(meltanoEnv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
    Seems there is some issue with the stream_maps ! Can anyone please guide
    ✅ 1
    r
    • 2
    • 9
  • t
    I need to scp a file off a server then parse that into json with my tap. I'm leaning towards an extension but it's looking I could just call scp directly in my extractor. Is there a reason to go with an extension? Is there already a solution for SCP?
    r
    • 2
    • 3
  • s
    Hi Team, Good Day ! I am working Snowflake->Salesforce connector.
    Copy code
    - name: tap-snowflake
    namespace: tap_snowflake
    pip_url: ./connectors/tap-snowflake
    executable: tap-snowflake
    capabilities:
    - state
    - catalog
    - discover
    - about
    - stream-maps
    settings:
    - name: account
      kind: string
      value: aigoiop-hq79023
      description: The Snowflake account identifier.
    - name: user
      kind: string
      value: udey
      description: The Snowflake username.
    - name: password
      kind: string
      value: ********
      description: The Snowflake password.
      sensitive: true
    - name: database
      kind: string
      value: PARTHAN_DB
      description: The Snowflake database name.
    - name: warehouse
      kind: string
      value: COMPUTE_WH
      description: The Snowflake warehouse name.
    select:
    - public-account.name 
  

  - name: target-salesforce
    namespace: target_salesforce
    pip_url: ./connectors/target-salesforce
    executable: target-salesforce
    capabilities:
    - about
    - stream-maps
    #- schema-flattening
    config:
      username: udey@vcs.sandbox
      password: ********
      security_token: nanLbbN3lexEw70gK7tLrzP4s
      api_type: sandbox
      sobject: Employee__c
      action: insert
      stream_maps:
        '*':   # Stream from the tap
          __alias__: Employee__c
          mappings:
            "name": "Name"
    Somehow this "mappings" configuration is not working. Throwing error as
    Copy code
    2025-04-04T10:11:29.098748Z [warning  ] Received state is invalid, incremental state has not been updated
2025-04-04T10:11:29.163005Z [info     ] Incremental state has been updated at 2025-04-04 10:11:29.162967+00:00.
2025-04-04T10:11:29.163935Z [info     ] TypeError: unhashable type: 'dict' cmd_type=elb consumer=True job_name=dev:tap-snowflake-to-target-salesforce name=target-salesforce producer=False run_id=0f693284-543d-4331-8029-ac397f2c6d83 stdio=stderr string_id=target-salesforce
2025-04-04T10:11:29.182189Z [info     ] 2025-04-04 15:41:28,998 | INFO     | snowflake.connector.connection | Snowflake Connector for Python Version: 3.13.2, Python Version: 3.13.2, Platform: Windows-11-10.0.22631-SP0 cmd_type=elb consumer=False job_name=dev:tap-snowflake-to-target-salesforce name=tap-snowflake producer=True run_id=0f693284-543d-4331-8029-ac397f2c6d83 stdio=stderr string_id=tap-snowflake
2025-04-04T10:11:29.183433Z [info     ] 2025-04-04 15:41:28,999 | INFO     | snowflake.connector.connection | Connecting to GLOBAL Snowflake domain cmd_type=elb consumer=False job_name=dev:tap-snowflake-to-target-salesforce name=tap-snowflake producer=True run_id=0f693284-543d-4331-8029-ac397f2c6d83 stdio=stderr string_id=tap-snowflake
2025-04-04T10:11:29.184298Z [info     ] 2025-04-04 15:41:28,999 | INFO     | snowflake.connector.connection | This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity. cmd_type=elb consumer=False job_name=dev:tap-snowflake-to-target-salesforce name=tap-snowflake producer=True run_id=0f693284-543d-4331-8029-ac397f2c6d83 stdio=stderr string_id=tap-snowflake
2025-04-04T10:11:29.241665Z [error    ] Loader failed
2025-04-04T10:11:29.242795Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Loader failed
(meltanoEnv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
    So I am trying to map name (From Snowflake) to Name (To Salesforce) field. [Camelcase]. 
    TypeError: unhashable type: 'dict'
    Did not get much examples on internet related to this "mappings".
    ✅ 1
    r
    • 2
    • 2
  • t
    I am very confused here. I'm trying to run scp as a utility and I can't get it to work. This works
    Copy code
    - name: scp
      namespace: scp
      commands:
        get_file:
          executable: scp
          args: -h
    This doesn't
    Copy code
    - name: test
      namespace: test
      commands:
        get_file:
          executable: /bin/bash
          args: -c "./test.sh"
    Neither does this
    Copy code
    - name: test
      namespace: test
      commands:
        get_file:
          executable: ssh
          args: -h
    Here is my output from the frist scp test
    Copy code
    [tanner@sato ubb-meltano]$ mel run scp
2025-04-08T16:50:21.070409Z [info     ] Environment 'dev' is active   
2025-04-08T16:50:21.117648Z [info     ] usage: scp [-346ABCOpqRrsTv] [-c cipher] [-D sftp_server_path] [-F ssh_config] cmd_type=command name=scp stdio=stderr
2025-04-08T16:50:21.117864Z [info     ]            [-i identity_file] [-J destination] [-l limit] [-o ssh_option] cmd_type=command name=scp stdio=stderr
2025-04-08T16:50:21.117973Z [info     ]            [-P port] [-S program] [-X sftp_option] source ... target cmd_type=command name=scp stdio=stderr
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

'NoneType' object is not subscriptable
    The other tests produce the same "NoneType" error but they don't print the command's help message. test.sh is just 
    echo hello
    . I made an scp extension that used to be in my utilities. It looked like this
    Copy code
    - name: scp-ext
      namespace: scp-ext
      pip_url: '../scp-ext'
      executable: scp
    I wonder if that's cached and that's why the scp command is the only thing that works? I've been banging my head against this for days. Any help would really be appreciated
    ✅ 1
  • r
    You invoke a command in the format 
    <plugin name>:<command name>
    , so in your case
    Copy code
    meltano run scp:get_file
    or have you already tried that?
  • t
    That's what I was missing. Thank you
    👍 1
  • t
    Yup, it's right there in the documents. My bad
    👍 1
  • o
    What’s the best practice for ingesting multiple taps in parallel to separate datasets, and then merging them into a final 
    stg
    dataset after completion? Use case: Ingesting data from multiple Shopify stores. Right now, we run one Meltano pipeline per store, which: • Extracts raw data into a shared 
    raw_shopify
    dataset in BigQuery • Creates common views in a single 
    stg_shopify
    dataset This setup causes some issues. Ideally, we want to: 1. Ingest each store's raw data into its own dataset (e.g. 
    raw_shopify_store1
    , 
    raw_shopify_store2
    , etc.) in parallel 2. Run per-store transforms into separate staging datasets (e.g. 
    stg_shopify_store1
    , etc.) 3. Run a final transform step that unions everything into a central 
    stg_shopify
    dataset Is there a clean way to do this in Meltano? Any recommendations or patterns others are using?
    e
    • 2
    • 2
  • s
    Hi All, I am trying to understand how Meltano internals are working. I am not able to figure out why one of the cases below is working but not both. • I am using Multiprocessing to call, child stream with parent keys. the data sent to Target is getting into race condition and the data posted is broken at times like ◦ expected sample, I have not added signer format but consider this in singer format and type record ▪︎ 
    {"key1": "value1" }, {"key2" : "value2"}
    ◦ random race condition data sample: the second record is emitted and written to the target before completing the first record writing. ▪︎ 
    {"key1": {"key2" : "value2"}
    ◦ I was under the assumption this was happening because of batching at tap and data written to jsonl ◦ This is happening even on removing the batching at tap • I tried Multiprocessing outside tap and call the meltano el as a subprocess for each chunk of data like below . This works without race condition.
    Copy code
    def run(time_range):
    try:
        is_backfill = os.environ.get("TAP_ZOOM_IS_BACKFILL")
        start_time, end_time = time_range
        start_time = shlex.quote(start_time)
        # start = start_time.replace(" ", "").replace(":", "")
        end_time = shlex.quote(end_time)
        cmd = (
            f"export MELTANO_STATE_BACKEND_URI='<s3://iflow-prod/state/backfill/zoom/>'"
            f"export TAP_ZOOM_FROM_DATE={start_time} TAP_ZOOM_TO_DATE={end_time} TAP_ZOOM_IS_BACKFILL={is_backfill}; "
            f"source .venv/bin/activate ; "
            f"meltano el tap-zoom target-s3-zoom --force;"
        )
        subprocess.run(cmd, shell=True, check=True)
        return {"time_range": time_range, "status": "success"}
    except Exception as e:
        return {"time_range": time_range, "status": "error", "error": str(e)}
    I was wondering if both the approaches spin an individual stdout pipe for each process spun but why is it getting into race condition in case 1 and not in case 2. My understanding is meltano sends data to stdout as per the target emit code, • might be a silly question but. How is Meltano differentiating logs that are emitted and singer records emitted? • when I spin a separate process this stdout should be different from the main process stdout right or else, is it the same stdout pipe. thanks for the time to read through any help is much appreciated, Thanks and have a great day
    e
    • 2
    • 6
  • a
    Random thought problem: Let’s say I have a table with 1billion+ rows, and for the longest time, I’ve been replicating it on an auto-incrementing 
    id
    column. However, the table is mutable at the source and has an 
    updated_at
    column, so that means I’m not catching changes in the source table once I’ve pulled the at-the-moment value of a row. So my situation is this: • I want to update meltano config from using 
    id
    as my replication-key to using 
    updated_at
    as my replication key, with 
    id
    as a value in 
    table-key-properties
    • I don’t want to start from the beginning of time, because it’s absolutely too much data to handle, so I’ve got to manually set a 
    date
    So the question is — how would you go about it?
    e
    • 2
    • 8
  • t
    We need to run a 
    show arp
    command on all routers at our ISP and get that data into our warehouse. Ansible is really good at communicating with network devices. It has profiles for each type and is able to recognize when a command starts/ends and parses that data for you. I don't think there's an equivalent tap for that with meltano. I'm wondering what the best way is to merge the two. Maybe I could make a tap that will call out to ansible and ansible can write what I want to a json file then my meltano tap can read from that json and pump it in to a raw table. Seems kind of weird at that point because all my tap is doing is just reading from a file. I could have ansible write directly to my postgres db but that feels like it'd be stepping on Meltano's toes. Looking for input
    👀 1
    v
    • 2
    • 7
  • d
    What is the best practice for setting up a different sync schedule for a subset of tables in a database? Should that be a separate project or maybe just set that up as an environment in the same project? Use case is that we have a table that receives periodic large bulk inserts and don't want that interfere with the other tables that have small changes which we want to replicate on a faster interval.
    m
    • 2
    • 2
  • s
    Hi Team, I am getting one issue with "meltano invoke tap-mysql". MySQL is installed on my local machine. (localhost)
    Copy code
    - name: tap-mysql
    namespace: tap_mysql
    pip_url: ./connectors/tap-mysql
    executable: tap-mysql
    capabilities:
    - about
    - batch
    - stream-maps
    - schema-flattening
    - discover
    - catalog
    - state
    settings:
    - name: host
      kind: string
      value: localhost
    - name: port
      kind: integer
      value: 3306  # Or whatever port your PostgreSQL is running on
    - name: user
      value: root
    - name: password
      kind: string
      value: *******  # Use an environment variable!
      sensitive: true
    - name: database
      kind: string
      value: world
    - name: is_vitess
      kind: boolean
      value: false
    Error:
    Copy code
    (sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend> meltano invoke tap-mysql
2025-05-08T13:38:51.175173Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2025-05-08T13:38:51.189778Z [info     ] Environment 'dev' is active   
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Catalog discovery failed: command ['C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano\\extractors\\tap-mysql\\venv\\Scripts\\tap-mysql.exe', '--config', 'C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano\\run\\tap-mysql\\tap.79a26421-4773-4b39-a35d-577aa37522b8.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Scripts\tap-mysql.exe\__main__.py", line 7, in <module>
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\click\core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\click\core.py", line 1081, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\click\core.py", line 949, in make_context
    self.parse_args(ctx, args)
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\click\core.py", line 1417, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\click\core.py", line 2403, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\click\core.py", line 2365, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\singer_sdk\tap_base.py", line 554, in cb_discover
    tap.run_discovery()
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\singer_sdk\tap_base.py", line 309, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\singer_sdk\tap_base.py", line 329, in catalog_json_text
    return dump_json(self.catalog_dict, indent=2)
                     ^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\tap_mysql\tap.py", line 333, in catalog_dict
    result["streams"].extend(self.connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-mysql\venv\Lib\site-packages\singer_sdk\connectors\sql.py", line 998, in discover_catalog_entries
    (reflection.ObjectKind.TABLE, False),
     ^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'sqlalchemy.engine.reflection' has no attribute 'ObjectKind'

(sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
    Copy code
    (sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend> python -m pip show SQLAlchemy
Name: SQLAlchemy
Version: 2.0.39
Summary: Database Abstraction Library
Home-page: <https://www.sqlalchemy.org>
Author: Mike Bayer
Author-email: <mailto:mike_mp@zzzcomputing.com|mike_mp@zzzcomputing.com>
License: MIT
Location: C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\sibaVenv\Lib\site-packages
Requires: greenlet, typing-extensions
Required-by: alembic, meltano
(sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend> meltano --version
meltano, version 3.7.4
(sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
    e
    • 2
    • 4
  • t
    I need to drop my raw schema every time before running running a specific tap. In my research I learned about macros. I wrote this one based on a macro I saw from a blog post:
    Copy code
    {%- macro drop_schema() -%}
    {%- set drop_query -%}
        drop schema {{ target.schema }}
    {%- endset -%}
    {% do run_query(drop_query) %}
{%- endmacro -%}
    I'm assuming I should be able to do something like this: 
    mel run dbt:run-operation:drop_schema sonar warehouse
    but I get an error saying it can't find drop_schema. I have it in 
    ./macros/
    . I'm assuming I need to put it in my meltano.yml under my dbt transfromer section. Maybe it should go under utilities? I'm at a loss
    r
    • 2
    • 5
  • s
    Hi Team, I am facing one issue with tap-sftp.
    Copy code
    (venv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
 meltano --log-level=debug invoke tap-sftp
2025-05-14T06:09:15.617690Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2025-05-14T06:09:15.624021Z [debug    ] Meltano 3.6.0, Python 3.12.3, Windows (AMD64)
2025-05-14T06:09:15.632565Z [debug    ] Looking up time zone info from registry
2025-05-14T06:09:15.651188Z [info     ] Environment 'dev' is active   
2025-05-14T06:09:15.707195Z [debug    ] Creating DB engine for project at 'C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend' with DB URI 'sqlite:/C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano/meltano.db'
2025-05-14T06:09:16.045430Z [debug    ] Skipped installing extractor 'tap-sftp'
2025-05-14T06:09:16.046429Z [debug    ] Skipped installing 1/1 plugins
2025-05-14T06:09:16.231095Z [debug    ] Created configuration at C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\run\tap-sftp\tap.31d90bf6-66b2-4bc2-9d17-9305905bbcdf.config.json
2025-05-14T06:09:16.235112Z [debug    ] Could not find tap.properties.json in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\tap.properties.json, skipping.
2025-05-14T06:09:16.238126Z [debug    ] Could not find tap.properties.cache_key in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\tap.properties.cache_key, skipping.
2025-05-14T06:09:16.240124Z [debug    ] Could not find state.json in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\state.json, skipping.
2025-05-14T06:09:16.248129Z [debug    ] Invoking: ['C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano\\extractors\\tap-sftp\\venv\\Scripts\\tap-sftp.exe', '--config', 'C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano\\run\\tap-sftp\\tap.31d90bf6-66b2-4bc2-9d17-9305905bbcdf.config.json']
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\pkey.py:59: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
  "cipher": algorithms.TripleDES,
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\transport.py:219: CryptographyDeprecationWarning: Blowfish has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.Blowfish and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 45.0.0.
  "class": algorithms.Blowfish,
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\transport.py:243: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
  "class": algorithms.TripleDES,
2025-05-14T06:09:17.479363Z [debug    ] Deleted configuration at C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\run\tap-sftp\tap.31d90bf6-66b2-4bc2-9d17-9305905bbcdf.config.json
(venv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
    I have created a basic configuration in meltano.yml file.
    Copy code
    - name: tap-sftp
    namespace: tap_sftp
    pip_url: ./connectors/tap-sftp
    executable: tap-sftp
    config:
      host: 10.148.155.30
      port: 22
      username: ubuntu
      start_date: 2025-05-13
      private_key_file: bridgex.pem
      tables:
      - table_name: single_file_test
        search_prefix: /home/ubuntu
        search_pattern: 'wget-log'
    Copy code
    (venv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
 meltano config tap-sftp list
2025-05-14T06:16:25.653934Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2025-05-14T06:16:25.677427Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.

Custom, possibly unsupported by the plugin:
host [env: TAP_SFTP_HOST] current value: '10.148.155.30' (from `meltano.yml`)
port [env: TAP_SFTP_PORT] current value: 22 (from `meltano.yml`)
username [env: TAP_SFTP_USERNAME] current value: 'ubuntu' (from `meltano.yml`)
start_date [env: TAP_SFTP_START_DATE] current value: '2025-05-13' (from `meltano.yml`)
private_key_file [env: TAP_SFTP_PRIVATE_KEY_FILE] current value: 'bridgex.pem' (from `meltano.yml`)
tables [env: TAP_SFTP_TABLES] current value: [{'table_name': 'single_file_test', 'search_prefix': '/home/ubuntu', 'search_pattern': 'wget-log'}] (from `meltano.yml`)
(venv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
    Can anyone please help.
    r
    s
    • 3
    • 21
  • a
    When I started with my meltano deployment I didn't have a separate systemdb, so used Azure for state store. Now I've seen the light, and moved to a system db. Is there any good reason to maintain a separate state store in Azure storage? Or is it just unnecessary complication?
    v
    e
    • 3
    • 4
  • k
    Hi everyone, I’ve been using the tap-postgres from MeltanoLabs version for doing hourly log-based replication. I wonder is there any way to parameterized the replication slot name? Instead of hardcoded 
    tappostgres
    can it be named something else? I might come into scenario where I must replicate the same source to two separate destination while data migration is happening. Thank you!
    v
    e
    • 3
    • 5
  • s
    Hi Team, I am using the "tap-sftp" from singer https://github.com/singer-io/tap-sftp Getting one issue saying
    Copy code
    paramiko.ssh_exception.SSHException: Incompatible ssh peer (no acceptable host key)
    For this I made a change in the client.py
    Copy code
    self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')
    Copy code
    def __try_connect(self):
        if not self.__active_connection:
            try:
                self.transport = paramiko.Transport((self.host, self.port))
                self.transport.use_compression(True)
                self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')
                self.transport.connect(username = self.username, pkey = self.key)
                self.sftp = paramiko.SFTPClient.from_transport(self.transport)
            except (AuthenticationException, SSHException) as ex:
                self.transport.close()
                self.transport = paramiko.Transport((self.host, self.port))
                self.transport.use_compression(True)
                self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')
                self.transport.connect(username= self.username, pkey = None)
                self.sftp = paramiko.SFTPClient.from_transport(self.transport)
            self.__active_connection = True
            # get 'socket' to set the timeout
            socket = self.sftp.get_channel()
            # set request timeout
            socket.settimeout(self.request_timeout)
    Even after making this change, its not resolving the issue.
    ✅ 1
    r
    • 2
    • 1
  • s
    Looking to modularize my Meltano project. Is using 
    pkl
    modules still considered a good practice here? I know there are some limitations to 
    include_paths
    .
    ✅ 1
    e
    • 2
    • 2
  • s
    Team, Do we have tap-onedrive from meltano side ? Has anyone ever tried it ?
    e
    c
    • 3
    • 3
  • b
    is there a best practice approach to apply a conditional when using an extractor?
    e
    • 2
    • 1
  • s
    Hi Team, I would like to introduce data encryption to my pipeline. Is there any default encryption mechanism available from meltano ?
    e
    • 2
    • 1
  • t
    https://github.com/MeltanoLabs/tap-csv Is there a way to configure this tap to drop the table before it runs every time?
    e
    • 2
    • 3
  • s
    Hi everyone, Is there a simple way to change the timezone for the logging? I’d like to keep everything as it is—just use my local timezone instead of the default. Could you give me a hint or point me in the right direction? Thank you in advance!
    r
    e
    • 3
    • 4
  • s
    Hello! I am heavily considering running Meltano inside of Dagster. I've never used Dagster before, so all tips and recommendations are welcome if anyone has any. Or maybe things I should know before proceeding down this route? I am currently just running Meltano standalone inside of an Ubuntu instance.
    e
    a
    j
    • 4
    • 7
  • s
    Hi everyone, I'm planning to build a common module to validate tap and target configurations (credentials, connection, etc.) before a pipeline is run. The main challenge is that every connector has a different API or method for testing a connection. My first thought is a large 
    switch
    statement to handle each connector individually, but that doesn't seem scalable. Has anyone approached this problem before? I'd love to hear any recommendations or design patterns you'd suggest instead of a massive 
    switch
    case. In any case if meltano offers something which can solve my issue ? Thanks!
    👀 1
    r
    a
    • 3
    • 9