Meltano #singer-taps

Denny Pradipta

12/17/2024, 7:46 AM

Hello, I am new to Meltano, trying to looking for alternatives for ETL tools I have this meltano.yml

Copy code

version: 1
default_environment: prod
project_id: 98dec0e3-6278-42e8-bcb8-8f1f18c72dcb
environments:
- name: dev
- name: staging
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-mongodb
        config:
          mongodb_connection_string:
            <mongodb://user:password@172.16.0.242:22017/collection?authSource=admin>
          database: indorelawan
      loaders:
      - name: target-postgres
        config:
          database: meltano
          user: meltano
          host: 172.16.0.248
          add_record_metadata: true
          port: 5433
plugins:
  extractors:
  - name: tap-mongodb
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-mongodb.git>
    metadata:
      "*":
        replication-method: "INCREMENTAL"
        replication-key: "_id"
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres

When I run

meltano --environment=prod run tap-mongodb target-postgres

, I got

Copy code

TypeError: Value after * must be an iterable, not Logger cmd_type=elb consumer=False job_name=prod:tap-mongodb-to-target-postgres name=tap-mongodb producer=True run_id=54cdd660-d0e4-4269-ae94-8860113e38cd stdio=stderr string_id=tap-mongodb

Any ideas why?

Kevin Phan

12/17/2024, 10:28 PM

getting this error

Copy code

2024-12-17T22:22:00.910855Z [error    ] The extractor generated a message exceeding the message size limit of 5.0MiB (half the buffer size of 10.0MiB).
2024-12-17T22:22:00.911147Z [error    ] To let this message be processed, increase the 'elt.buffer_size' setting to at least double the size of the largest expected message, and try again.
2024-12-17T22:22:00.911327Z [error    ] To learn more, visit <https://docs.meltano.com/reference/settings#eltbuffer_size>
2024-12-17T22:22:00.934366Z [error    ] Output line length limit exceeded
2024-12-17T22:22:00.934764Z [error    ] Block run completed.

if i set an env var

MELTANO_ELT_BUFFER_SIZE

in my cluster to 100mib, does meltano automatically grab that value?

Kevin Phan

12/17/2024, 11:48 PM

been getting this alot lately so ill post again for vis. So i have a CDC tap as such

Copy code

- name: tap-reporting-db-source
    inherit_from: tap-postgres-pipelinewise
    namespace: tap_postgres_pipelinewise
    config:
      # <<: *reportingdb_credentials
    select:
    - public-external_balances.*
    - public-direct_deposits.*

    - public-transactions_history.user_id
    - public-transactions_history.currency
    - public-transactions_history.created_at
    - public-transactions_history.updated_at

when i do

meltano select tap-reporting-db-source --list

i get the pic attached. I dont see the

Copy code

public-transactions_history.transaction

field yet i see it in snowflake. i full loaded the table without the transaction column and then reverted to CDC where for the full load rows, the transaction is null but for new rows it appears. Not entirely sure why its doing this. Is there a config im missing or something i am doing wrong here?

Pawel Plaszczak

12/26/2024, 5:01 PM

hey hello! I am evaluating Meltano for our ETL project. Quite excited about what I read in the docs, but then the reality hit 😉 First question: what is the currently recommended way of connecting Meltano to Oracle database as source? I am quite confused here. My platform is Ubuntu, connecting to remote Oracle source. I followed instructions to install extractor tap-oracle. This resulted in series of warnings and errors. I resolved some of those by installing dependencies, but ended up with cryptic "plugin configuration is invalid / Exception: Unrecognized replication_method None". I cannot go further. I also saw that in the command I can turn on debug-type logging:

Copy code

meltano config tap-oracle test

but its output is cryptic. So for now, I abandonned this route, because I found another: I spotted that the errors included warnings that said that cx-oracle was outdated, and recommended python-oracle driver instead. So I thought maybe this would be a better route. I went to the python-oracle page: https://python-oracledb.readthedocs.io/en/latest/user_guide/installation.html and installed python-oracle and tested that it connects to my database. So I have partial success but what next? Now - okay, python-oracle works, but now how to come back to Meltano and make this working driver part of the Meltano configuration? How to make Meltano upload Oracle table using this working python-pracle config? Shall I still somehow use the tap-oracle extractor (how)? I also saw that there are alternative implementation of Oracle connector on this page: https://hub.meltano.com/extractors/tap-oracle/ but since I've already been spending quite some time, I prefer to ask before trying. In short, what's the recommended and the most standard way to connect Meltano to Oracle database as source of data? Oracle (source) is remote, while the local data destination (on the same machine as Meltano install) can be whatever - initially, for the prototyping phase, I'd say PostgreSQL

Prashant Vikram

01/02/2025, 6:03 AM

Subject: Compatibility Issues with
pipelinewise-tap-postgres
and
pipelinewise-target-snowflake
Hi everyone, We're using PipelineWise taps and targets, specifically: -

pipelinewise-tap-postgres==2.1.0

pipelinewise-target-snowflake==2.2.0

Since Snowflake is phasing out support for Python connector versions

< 3.0.0

, I upgraded

pipelinewise-target-snowflake

to version

2.3.0

, as the older version (

2.2.0

) uses Python connector

2.7.0

in the backend. However, I'm now facing compatibility issues. My problem seems identical to this open issue: 🔗 [PipelineWise tap-postgres Issue #228](https://github.com/transferwise/pipelinewise-tap-postgres/issues/228) Since pipelinewise-tap-postgres has been archived and the issue is still unresolved: 1. Has anyone else encountered this problem? 2. Are there any known solutions or workarounds for this? Any guidance would be greatly appreciated! Thanks in advance!

Ben Kentzer

01/06/2025, 4:21 PM

Hello! Working through using the hot glue tap for NetSuite, and there are a couple of tables that we need to extract that aren't listed in the source code (so I'm not convinced we'll be able to pull them without development). Has anyone extended the hot glue tap-netsuite, or is there another option to pull in the tables that we need? It looks like we need "Items" (included already), "Classes" (not included) and "Transaction_lists" - as well as potentially one or two others. Any ideas? Thanks Ben

Pawel Plaszczak

01/10/2025, 8:45 AM

How can I replicate a table with some offset? I have a very large table and for test purposes would like to only the recent records. I use tap-oracle. I think that replication_key_value variable is for this. For test purposes, I created table PERSONS, and with three records PERSON_ID with values 1,2,3. I wanted to set offset in such a way, that only the record 3 would be replicated. I first tried to set up replication-key-value in meltano.yml, but the variable kept being ignored by meltano. I then tried another method: I saw that I could pass a STATE object to meltano invoke command. So I captured the state (json) from the previous meltano run in a file state.log:

"type":"STATE",

"value":{

"bookmarks":{

"PLASZPA-PERSONS":{

"last_replication_method":"INCREMENTAL",

"replication_key":"PERSON_ID",

"replication_key_value":3

and then I invoked tap-oracle with this state:

meltano invoke tap-oracle -s state.log

But again, the replication-key-value was ignored - all three records with PERSON_ID values 1,2, and 3 got replicated. Is this because this variable is not supported by tap-oracle?

👀 1

Michal Ondras

01/13/2025, 8:59 PM

hi, is this operational https://hub.meltano.com/extractors/tap-mysql--meltanolabs/ ? should we default wise (default)?

Michal Ondras

01/13/2025, 11:54 PM

hi, tap-postgres on latest all of sudden can't download

✅ 1

magnus_avitsland

01/14/2025, 8:52 AM

Hi, I cannot find a tap for Azure BlobStorage, I can only find a loader for it (https://hub.meltano.com/loaders/target-azureblobstorage).

Shachi Bista

01/16/2025, 1:09 PM

Hi! I am trying to move a lot of different ETL pipelines into meltano and one of our requirements is to tap specific files (json) from a github repo. Is there a tap that already allows you to tap specific files from a github repository? I did not find anything that meets my requirement, the closest was

tap-smoke-test

but I am not sure if it fits our needs.

Pawel Plaszczak

01/17/2025, 4:29 PM

Hello, I now try to copy data from Postgres to any destination. I am using tap-postgres, default version. How can I filter the data? I don't want to copy all tables in the schema, this would be insane. I want to copy just some elected tables, and just some records. None of the known methods worked for me. This is the meltano.yml:

- name: tap-postgres

variant: meltanolabs

pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>

config:

sqlalchemy_url: ****

filter_schemas:

- myschema

metadata:

myschema-MYTABLE:

replication-method: INCREMENTAL

replication-key: KW_UPDATE_DATE

I tried the following: • I tried to set filter_tables parameters but the tap does not allow it. • I tried replication_key_value (thinking I can block copying of particular tables by setting the value in the future) but it got ignored. • I tried to run invoke with parameter -s state.json but that parameter is not allowed • I looked at other variants of tap-postgres but they seem even more concise and not supportive of extra options My problem is that the current version of tap-postgres is copying the entire schema and all the data in all the tables in the schema. This is a no-go... Thank you.

Jun Pei Liang

01/22/2025, 9:35 PM

does anyone have an update on https://github.com/singer-io/tap-dynamics-365?

Jun Pei Liang

01/22/2025, 9:36 PM

much appreciated!

Jacob Ukokobili

01/23/2025, 5:55 PM

@Edgar Ramírez (Arch.dev) Could you please guide me on how to flatten the

JSON

data from the

MySQL

tap configuration before it’s loaded into

BigQuery

? I’ve already set

flattening_enabled: true

in the

BigQuery

target configuration, but I’m not achieving the desired result. Here's my current `meltano.yml`:

Copy code

plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: 
      git+<https://github.com/transferwise/pipelinewise.git#subdirectory=singer-connectors/tap-mysql>
    select:
    - '*.*'      # Select all tables first
    - '!*_audit*'      # Then exclude audit tables
    metadata:
      '*.*':    # Apply metadata to all non-excluded tables
        replication-method: INCREMENTAL
        replication_key: update_time
        key_properties:
        - id

  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    executable: meltano-map-transform
    mappings:
    - name: mysql_to_bq
      config:
        flattening_enabled: true
        flattening_max_depth: 3   # flatten only top-level properties
        stream_maps:
          "*":
            __alias: "{{ stream_name | regex_replace('^smartterm_', '') }}"

  loaders:
  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
    config:
      batch_size: 10000
      flattening_enabled: true
      flattening_max_depth: 3
      partition_granularity: day
      dedupe_before_upsert: true
      timeout: 3600
      upsert: true

Andy Carter

01/28/2025, 9:24 AM

Anyone considered or developed with

pytrends

to create a google trends tap? Google don't have a public API for the data. https://github.com/GeneralMills/pytrends

👀 1

Jun Pei Liang

01/29/2025, 10:57 PM

Copy code

got the following error with oracle tap. any idea?    


root@c713d163edac:/projects/my-meltano-project# meltano config tap-oracle test
2025-01-29T22:51:16.797625Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
AttributeError: 'NoneType' object has no attribute 'get'
root@c713d163edac:/projects/my-meltano-project# meltano --log-level=debug invoke tap-oracle
2025-01-29T22:52:29.015603Z [debug    ] Meltano 3.6.0, Python 3.12.3, Linux (x86_64)
2025-01-29T22:52:29.017052Z [debug    ] /etc/timezone found, contents:
 America/Los_Angeles

2025-01-29T22:52:29.017327Z [debug    ] /etc/localtime found
2025-01-29T22:52:29.018028Z [debug    ] 2 found:
 {'/etc/timezone': 'America/Los_Angeles', '/etc/localtime is a symlink to': 'America/Los_Angeles'}
2025-01-29T22:52:29.020152Z [info     ] Environment 'dev' is active
2025-01-29T22:52:29.030326Z [debug    ] Creating DB engine for project at '/projects/my-meltano-project' with DB URI '<sqlite://projects/my-meltano-project/.meltano/meltano.db>'
2025-01-29T22:52:29.054840Z [debug    ] Found plugin parent            parent=tap-oracle plugin=tap-oracle source=LOCKFILE
2025-01-29T22:52:29.146614Z [debug    ] Skipped installing extractor 'tap-oracle'
2025-01-29T22:52:29.146846Z [debug    ] Skipped installing 1/1 plugins
2025-01-29T22:52:29.267378Z [debug    ] Created configuration at /projects/my-meltano-project/.meltano/run/tap-oracle/tap.e8454e4e-b596-4b5a-a3e1-7072a967ecb4.config.json
2025-01-29T22:52:29.267662Z [debug    ] Could not find tap.properties.json in /projects/my-meltano-project/.meltano/extractors/tap-oracle/tap.properties.json, skipping.
2025-01-29T22:52:29.267820Z [debug    ] Could not find tap.properties.cache_key in /projects/my-meltano-project/.meltano/extractors/tap-oracle/tap.properties.cache_key, skipping.
2025-01-29T22:52:29.267948Z [debug    ] Could not find state.json in /projects/my-meltano-project/.meltano/extractors/tap-oracle/state.json, skipping.
2025-01-29T22:52:29.268222Z [debug    ] Using cached catalog file
2025-01-29T22:52:29.268718Z [debug    ] Visiting CatalogNode.STREAM at '.streams[0]'.
2025-01-29T22:52:29.269016Z [debug    ] Setting '.streams[0].selected' to 'False'
2025-01-29T22:52:29.269110Z [debug    ] Setting '.streams[0].selected' to 'True'
2025-01-29T22:52:29.269170Z [debug    ] Setting '.streams[0].replication_method' to 'INCREMENTAL'
2025-01-29T22:52:29.269242Z [debug    ] Setting '.streams[0].replication_key' to 'ROWVERSION'
2025-01-29T22:52:29.269325Z [debug    ] Skipping node at '.streams[0].tap_stream_id'
2025-01-29T22:52:29.269399Z [debug    ] Skipping node at '.streams[0].table_name'
2025-01-29T22:52:29.269483Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.ABC_CLASS'.
2025-01-29T22:52:29.269569Z [debug    ] Skipping node at '.streams[0].schema.properties.ABC_CLASS.maxLength'
2025-01-29T22:52:29.269636Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.ABC_PERCENT'.
2025-01-29T22:52:29.269711Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.OBJKEY'.
2025-01-29T22:52:29.269770Z [debug    ] Skipping node at '.streams[0].schema.properties.OBJKEY.maxLength'
2025-01-29T22:52:29.269829Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.OBJVERSION'.
2025-01-29T22:52:29.269911Z [debug    ] Skipping node at '.streams[0].schema.properties.OBJVERSION.maxLength'
2025-01-29T22:52:29.269983Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.OBJID'.
2025-01-29T22:52:29.270051Z [debug    ] Skipping node at '.streams[0].schema.type'
2025-01-29T22:52:29.270133Z [debug    ] Skipping node at '.streams[0].stream'
2025-01-29T22:52:29.270218Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[0]'.
2025-01-29T22:52:29.270265Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '[]'
2025-01-29T22:52:29.270322Z [debug    ] Setting '.streams[0].metadata[0].metadata.selected' to 'False'
2025-01-29T22:52:29.270373Z [debug    ] Setting '.streams[0].metadata[0].metadata.selected' to 'True'
2025-01-29T22:52:29.270433Z [debug    ] Setting '.streams[0].metadata[0].metadata.replication-method' to 'INCREMENTAL'
2025-01-29T22:52:29.270506Z [debug    ] Setting '.streams[0].metadata[0].metadata.replication-key' to 'ROWVERSION'
2025-01-29T22:52:29.270584Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.schema-name'
2025-01-29T22:52:29.270667Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.database-name'
2025-01-29T22:52:29.270760Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.is-view'
2025-01-29T22:52:29.270835Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.selected'
2025-01-29T22:52:29.270904Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.replication-method'
2025-01-29T22:52:29.270961Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.replication-key'
2025-01-29T22:52:29.271014Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[1]'.
2025-01-29T22:52:29.271067Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'ABC_CLASS']'
2025-01-29T22:52:29.271160Z [debug    ] Setting '.streams[0].metadata[1].metadata.selected' to 'False'
2025-01-29T22:52:29.271219Z [debug    ] Setting '.streams[0].metadata[1].metadata.selected' to 'True'
2025-01-29T22:52:29.271270Z [debug    ] Skipping node at '.streams[0].metadata[1].breadcrumb[0]'
2025-01-29T22:52:29.271309Z [debug    ] Skipping node at '.streams[0].metadata[1].breadcrumb[1]'
2025-01-29T22:52:29.271366Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.sql-datatype'
2025-01-29T22:52:29.271438Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.inclusion'
2025-01-29T22:52:29.271525Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.selected-by-default'
2025-01-29T22:52:29.271595Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.selected'
2025-01-29T22:52:29.271671Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[2]'.
2025-01-29T22:52:29.271753Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'ABC_PERCENT']'
2025-01-29T22:52:29.271834Z [debug    ] Setting '.streams[0].metadata[2].metadata.selected' to 'False'
2025-01-29T22:52:29.271883Z [debug    ] Setting '.streams[0].metadata[2].metadata.selected' to 'True'
2025-01-29T22:52:29.271952Z [debug    ] Skipping node at '.streams[0].metadata[2].breadcrumb[0]'
2025-01-29T22:52:29.272016Z [debug    ] Skipping node at '.streams[0].metadata[2].breadcrumb[1]'
2025-01-29T22:52:29.272082Z [debug    ] Skipping node at '.streams[0].metadata[2].metadata.sql-datatype'
2025-01-29T22:52:29.272161Z [debug    ] Skipping node at '.streams[0].metadata[2].metadata.inclusion'
2025-01-29T22:52:29.272237Z [debug    ] Skipping node at '.streams[0].metadata[2].metadata.selected-by-default'
2025-01-29T22:52:29.272301Z [debug    ] Skipping node at '.streams[0].metadata[2].metadata.selected'
2025-01-29T22:52:29.272382Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[3]'.
2025-01-29T22:52:29.272473Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'OBJKEY']'
2025-01-29T22:52:29.272549Z [debug    ] Setting '.streams[0].metadata[3].metadata.selected' to 'False'
2025-01-29T22:52:29.272598Z [debug    ] Setting '.streams[0].metadata[3].metadata.selected' to 'True'
2025-01-29T22:52:29.272642Z [debug    ] Skipping node at '.streams[0].metadata[3].breadcrumb[0]'
2025-01-29T22:52:29.272739Z [debug    ] Skipping node at '.streams[0].metadata[3].breadcrumb[1]'

Jun Pei Liang

01/29/2025, 10:57 PM

Copy code

2025-01-29T22:52:29.272860Z [debug    ] Skipping node at '.streams[0].metadata[3].metadata.sql-datatype'
2025-01-29T22:52:29.272993Z [debug    ] Skipping node at '.streams[0].metadata[3].metadata.inclusion'
2025-01-29T22:52:29.273059Z [debug    ] Skipping node at '.streams[0].metadata[3].metadata.selected-by-default'
2025-01-29T22:52:29.273121Z [debug    ] Skipping node at '.streams[0].metadata[3].metadata.selected'
2025-01-29T22:52:29.273186Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[4]'.
2025-01-29T22:52:29.273255Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'OBJVERSION']'
2025-01-29T22:52:29.273387Z [debug    ] Setting '.streams[0].metadata[4].metadata.selected' to 'False'
2025-01-29T22:52:29.273480Z [debug    ] Setting '.streams[0].metadata[4].metadata.selected' to 'True'
2025-01-29T22:52:29.273555Z [debug    ] Skipping node at '.streams[0].metadata[4].breadcrumb[0]'
2025-01-29T22:52:29.273627Z [debug    ] Skipping node at '.streams[0].metadata[4].breadcrumb[1]'
2025-01-29T22:52:29.273746Z [debug    ] Skipping node at '.streams[0].metadata[4].metadata.sql-datatype'
2025-01-29T22:52:29.273844Z [debug    ] Skipping node at '.streams[0].metadata[4].metadata.inclusion'
2025-01-29T22:52:29.273953Z [debug    ] Skipping node at '.streams[0].metadata[4].metadata.selected-by-default'
2025-01-29T22:52:29.274038Z [debug    ] Skipping node at '.streams[0].metadata[4].metadata.selected'
2025-01-29T22:52:29.274184Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[5]'.
2025-01-29T22:52:29.274307Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'OBJID']'
2025-01-29T22:52:29.274423Z [debug    ] Setting '.streams[0].metadata[5].metadata.selected' to 'False'
2025-01-29T22:52:29.274583Z [debug    ] Skipping node at '.streams[0].metadata[5].breadcrumb[0]'
2025-01-29T22:52:29.274689Z [debug    ] Skipping node at '.streams[0].metadata[5].breadcrumb[1]'
2025-01-29T22:52:29.274776Z [debug    ] Skipping node at '.streams[0].metadata[5].metadata.sql-datatype'
2025-01-29T22:52:29.274856Z [debug    ] Skipping node at '.streams[0].metadata[5].metadata.inclusion'
2025-01-29T22:52:29.274920Z [debug    ] Skipping node at '.streams[0].metadata[5].metadata.selected-by-default'
2025-01-29T22:52:29.274980Z [debug    ] Skipping node at '.streams[0].metadata[5].metadata.selected'
2025-01-29T22:52:29.275045Z [debug    ] Skipping node at '.streams[0].selected'
2025-01-29T22:52:29.275103Z [debug    ] Skipping node at '.streams[0].replication_method'
2025-01-29T22:52:29.275161Z [debug    ] Skipping node at '.streams[0].replication_key'
2025-01-29T22:52:29.276766Z [debug    ] Invoking: ['/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/bin/tap-oracle', '--config', '/projects/my-meltano-project/.meltano/run/tap-oracle/tap.e8454e4e-b596-4b5a-a3e1-7072a967ecb4.config.json', '--catalog', '/projects/my-meltano-project/.meltano/run/tap-oracle/tap.properties.json']
time=2025-01-29 14:52:29 name=singer level=INFO message=Running in cx mode
time=2025-01-29 14:52:29 name=singer level=WARNING message=cx_oracle is no longer maintained, use python-oracledb

To switch to python-oracledb set the environment variable ORA_PYTHON_DRIVER_TYPE=thin or thick.

Documentation for python-oracledb can be found here: <https://oracle.github.io/python-oracledb/>
time=2025-01-29 14:52:29 name=singer level=INFO message=Selected streams: ['IFSAPP-ABC_CLASS']
time=2025-01-29 14:52:29 name=singer level=INFO message=No currently_syncing found
time=2025-01-29 14:52:29 name=singer level=INFO message=Beginning sync of stream(IFSAPP-ABC_CLASS) with sync method(incremental)
time=2025-01-29 14:52:29 name=singer level=INFO message=Stream IFSAPP-ABC_CLASS is using incremental replication with replication key ROWVERSION
{"type":"SCHEMA","stream":"IFSAPP-ABC_CLASS","schema":{"properties":{"ABC_CLASS":{"maxLength":1,"type":["null","string"]},"ABC_PERCENT":{"type":["null","integer"]},"OBJKEY":{"maxLength":50,"type":["null","string"]},"OBJVERSION":{"maxLength":14,"type":["null","string"]},"OBJID":{}},"type":"object"},"key_properties":[],"bookmark_properties":["ROWVERSION"]}
time=2025-01-29 14:52:29 name=singer level=INFO message=dsn: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.10.131.67)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=<http://TEST10PDB.ifs10devdb.vcn10sj.oraclevcn.com|TEST10PDB.ifs10devdb.vcn10sj.oraclevcn.com>)))
{"type":"STATE","value":{"bookmarks":{"IFSAPP-ABC_CLASS":{"last_replication_method":"INCREMENTAL","replication_key":"ROWVERSION","version":1738191149591}},"currently_syncing":"IFSAPP-ABC_CLASS"}}
{"type":"ACTIVATE_VERSION","stream":"IFSAPP-ABC_CLASS","version":1738191149591}
time=2025-01-29 14:52:29 name=singer level=CRITICAL message='NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/bin/tap-oracle", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 686, in main
    raise exc
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 683, in main
    main_impl()
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 677, in main_impl
    do_sync(conn_config, args.catalog, args.config.get('default_replication_method'), state)
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 619, in do_sync
    state = sync_traditional_stream(conn_config, stream, state, sync_method_lookup[stream.tap_stream_id], end_scn)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 574, in sync_traditional_stream
    state = do_sync_incremental(conn_config, stream, state, desired_columns)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 460, in do_sync_incremental
    state = incremental.sync_table(conn_config, stream, state, desired_columns)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/sync_strategies/incremental.py", line 63, in sync_table
    replication_key_sql_datatype = md.get(('properties', replication_key)).get('sql-datatype')
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
2025-01-29T22:52:29.621253Z [debug    ] Deleted configuration at /projects/my-meltano-project/.meltano/run/tap-oracle/tap.e8454e4e-b596-4b5a-a3e1-7072a967ecb4.config.json

Jun Pei Liang

01/29/2025, 11:06 PM

please ignore

Jun Pei Liang

01/29/2025, 11:06 PM

it was using the view, that is why error occur

👍 1

Ismail Muallaoglu

02/10/2025, 7:53 AM

Hi there 🙂 I am kinda new in here and apologies if I shouldn't be posting any question in this channel. I started my Meltano experience with tap-taboola, but I get the same error as raised in here: https://github.com/singer-io/tap-taboola/issues/14

"float() argument must be a string or number, not 'NoneType'"

Probably related to lines like this in init.py

'cpc': float(campaign.get('cpc', 0.0))

Wondering if anyone managed to solve the issue yet?

👀 1

Reuben (Matatika)

02/12/2025, 7:50 PM

What's the best way to go about ensuring a tap emits state at regular intervals with a timestamp-based replication key? Is this already handled by the SDK based on record count?

Reuben (Matatika)

02/14/2025, 2:39 PM

Are there any example of REST taps that stream responses? I'm trying to retry the request on an exception that occurs as a result of the following

iter_lines

call, but not having much success...

Copy code

@override
    def _request(self, prepared_request, context):
        response = self.requests_session.send(
            prepared_request,
            stream=True,  # streaming request
            timeout=self.timeout,
            allow_redirects=self.allow_redirects,
        )
        self._write_request_duration_log(
            endpoint=self.path,
            response=response,
            context=context,
            extra_tags={"url": prepared_request.path_url}
            if self._LOG_REQUEST_METRIC_URLS
            else None,
        )
        self.validate_response(response)

        return response

    @override
    def parse_response(self, response):
        with response:  # ensure connection is eventually released
            yield from (
                json.loads(line, parse_float=decimal.Decimal)
                for line in response.iter_lines()
            )

Namely •

urllib3.exceptions.ProtocolError: Response ended prematurely

• (during handling of above)

requests.exceptions.ChunkedEncodingError: Response ended prematurely

and •

http.client.IncompleteRead: IncompleteRead(506 bytes read, 6 more expected)

• (directly caused by above)

urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(506 bytes read, 6 more expected)', IncompleteRead(506 bytes read, 6 more expected))

• (during handling of above)

requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(506 bytes read, 6 more expected)', IncompleteRead(506 bytes read, 6 more expected))

Malcolm Doepel

02/20/2025, 11:47 PM

Hello, I am trying to connect to a fairly simple OData service. Authentication is simple with a basic username and password, there is a service root URI address, and the data transfer format is atom/xml instead of json. Is this something that the tap-rest-api-msdk extractor should be able to handle? Or is a custom connector typically required for OData? I haven't been able to find much specifically on Meltano and the OData protocol. Any help is much appreciated!

Edgar Ramírez (Arch.dev)

02/27/2025, 4:14 PM

I think https://hub.meltano.com/extractors/tap-spreadsheets-anywhere can extract from a local JSON file. Are there specific requirements you have in mind?

Stéphane Burwash

03/04/2025, 8:28 PM

Hey everyone 👋 Question for yall -> Has anyone ever succesfully removed the prefix "METRIC" from a singer SDK metric? I'm trying to implement full json logging and it's a bit difficult 😆 https://sdk.meltano.com/en/v0.43.1/implementation/logging.html#custom-logging-configuration I'm looking into

custom_logging_configurations

, but the

METRIC:

seems to be part of the log -> I can't remove it without creating a filter. Any ideas?

Jessica Li

03/24/2025, 1:17 PM

I was looking at some issues about selecting nested properties and had a follow up question related to this: https://github.com/meltano/meltano/issues/3024 https://github.com/meltano/meltano/issues/8807 is it not possible to use

select

to specify only the desired nested fields? is it always required to explicitly deselect all other unwanted fields? for

tap-example

, if there were many unwanted attributes, they would all need to be listed as

!entity.attributes.a

entity.attributes.b

, etc.,? is there no way to ignore all other nested fields aside from the ones selected?

Copy code

plugins:
  extractors:
    - name: tap-example
      select:
      - entity.id
      # - entity.attributes             # without this, SDK will drop attributes
      # - entity.attributes.*           # without this, no nested properties will be selected
      - entity.attributes.name          # select wanted nested property
      # - "!entity.attributes.unwanted" # without this, all other nested properties will be selected

Chinmay

04/06/2025, 2:19 PM

Hi team, We're currently using

tap-quickbooks

and encountering an issue related to the

start_date

format. In our

.yml

config file, we’ve defined the

start_date

as:

Copy code

start_date: '2025-01-01T00:00:00.000Z'

However, we’re getting the following error when running the tap:

Copy code

CRITICAL time data '2025-01-01T00:00:00+00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '2025-01-01T00:00:00+00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'

Could you please help us resolve this or point us in the right direction? Thanks!

✅ 1

Lior Naim Alon

04/20/2025, 12:10 PM

I'm using the tap-hubspot, my goal is to extract all contacts data, run some transformations in my DWH, then write the enriched data back to hubspot for some unknown reason, every since many records were added in hubspot (it's a sandbox environment) - the tap is looping endlessly extracting data, with many duplicates being written over and over again to my target. the records don't seem to be different, so i suspect the tap is not using a proper paging mechanism to progress through the extraction? here's my

meltano.yml

Copy code

version: 1
default_environment: dev
environments:
- name: dev
- name: staging
- name: prod
state_backend:
  uri: <s3://dwh/meltano-states/>
s3:
    aws_access_key_id: ${AWS_ACCESS_KEY_ID}
    aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
plugins:
  extractors:
  - name: tap-hubspot
    # python: python
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-hubspot.git@v0.6.3>
    config:
      start_date: '2020-01-01'
    select:
    - contacts.*
  loaders:
  - name: target-s3
    variant: crowemi
    pip_url: git+<https://github.com/crowemi/target-s3.git>
    config:
      append_date_to_filename: true
      append_date_to_filename_grain: microsecond
      partition_name_enabled: true
  - name: target-s3--hubspot
    inherit_from: target-s3
    config:
      format:
        format_type: parquet
      prefix: dwh/hubspot
      flattening_enabled: false

Samuel Nogueira Farrus

04/30/2025, 11:34 AM

Greetings! I tried to use extractor

tap-db2

but it returned an pip/wheel error when attempting to install:

Copy code

(venv) PS C:\meltano\db2> meltano add extractor tap-db2
Cloning <https://github.com/mjsqu/tap-db2.git> to c:\temp\<user>\pip-req-build-km0p0dgy
  Running command git clone --filter=blob:none --quiet <https://github.com/mjsqu/tap-db2.git> 'C:\TEMP\<user>\pip-req-build-km0p0dgy'
  Resolved <https://github.com/mjsqu/tap-db2.git> to commit ea2cd49b9fcb4dd599e66249445d8c0d8b06d6d4
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting attrs==23.1.0 (from tap-db2==1.0.6)
  Using cached attrs-23.1.0-py3-none-any.whl.metadata (11 kB)
Collecting ibm-db-sa==0.4.0 (from tap-db2==1.0.6)
  Using cached ibm_db_sa-0.4.0-py3-none-any.whl.metadata (5.3 kB)
Collecting ibm-db==3.2.0 (from tap-db2==1.0.6)
  Using cached ibm_db-3.2.0.tar.gz (206 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting jinja2==3.1.2 (from tap-db2==1.0.6)
  Using cached Jinja2-3.1.2-py3-none-any.whl.metadata (3.5 kB)
Collecting markupsafe<2.2.0 (from tap-db2==1.0.6)
  Using cached markupsafe-2.1.5-py3-none-any.whl
Collecting pendulum==2.1.2 (from tap-db2==1.0.6)
  Using cached pendulum-2.1.2.tar.gz (81 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pyodbc==5.0.1 (from tap-db2==1.0.6)
  Using cached pyodbc-5.0.1.tar.gz (115 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pytz>=2018.1 (from tap-db2==1.0.6)
  Using cached pytz-2025.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting singer-python>=5.12.0 (from tap-db2==1.0.6)
  Using cached singer_python-6.1.1-py3-none-any.whl
Collecting sqlalchemy<3.0.0 (from tap-db2==1.0.6)
  Using cached sqlalchemy-2.0.40-cp313-cp313-win_amd64.whl.metadata (9.9 kB)
Collecting python-dateutil<3.0,>=2.6 (from pendulum==2.1.2->tap-db2==1.0.6)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytzdata>=2020.1 (from pendulum==2.1.2->tap-db2==1.0.6)
  Using cached pytzdata-2020.1-py2.py3-none-any.whl.metadata (2.3 kB)
Collecting six>=1.5 (from python-dateutil<3.0,>=2.6->pendulum==2.1.2->tap-db2==1.0.6)
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting greenlet>=1 (from sqlalchemy<3.0.0->tap-db2==1.0.6)
  Using cached greenlet-3.2.1-cp313-cp313-win_amd64.whl.metadata (4.2 kB)
Collecting typing-extensions>=4.6.0 (from sqlalchemy<3.0.0->tap-db2==1.0.6)
  Using cached typing_extensions-4.13.2-py3-none-any.whl.metadata (3.0 kB)
Collecting jsonschema==2.*,>=2.6.0 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached jsonschema-2.6.0-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting simplejson==3.*,>=3.13.2 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached simplejson-3.20.1-cp313-cp313-win_amd64.whl.metadata (3.4 kB)
Collecting backoff==2.*,>=2.2.1 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached backoff-2.2.1-py3-none-any.whl.metadata (14 kB)
Collecting ciso8601==2.*,>=2.3.1 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached ciso8601-2.3.2.tar.gz (28 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Using cached ibm_db_sa-0.4.0-py3-none-any.whl (31 kB)
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached sqlalchemy-2.0.40-cp313-cp313-win_amd64.whl (2.1 MB)
Using cached greenlet-3.2.1-cp313-cp313-win_amd64.whl (295 kB)
Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
Using cached backoff-2.2.1-py3-none-any.whl (15 kB)
Using cached jsonschema-2.6.0-py2.py3-none-any.whl (39 kB)
Using cached simplejson-3.20.1-cp313-cp313-win_amd64.whl (75 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Using cached typing_extensions-4.13.2-py3-none-any.whl (45 kB)
Building wheels for collected packages: tap-db2, ibm-db, pendulum, pyodbc, ciso8601
  Building wheel for tap-db2 (pyproject.toml): started
  Building wheel for tap-db2 (pyproject.toml): finished with status 'done'
  Created wheel for tap-db2: filename=tap_db2-1.0.6-py3-none-any.whl size=29948 sha256=ab8ca931a326cb0937229d903708ff208bdded393e366c8e6eb2d1833290179e
  Stored in directory: C:\TEMP\<user>\pip-ephem-wheel-cache-w9wzy1j_\wheels\67\43\15\a99e5c72b4b3dcd727d50dfa99a0647c44e30ae3cc0f543b84
  Building wheel for ibm-db (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for ibm-db (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for ibm-db (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for ibm-db
  Building wheel for pendulum (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for pendulum (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for pendulum (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for pendulum
  Building wheel for pyodbc (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for pyodbc (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for pyodbc (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for pyodbc
  Building wheel for ciso8601 (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for ciso8601 (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for ciso8601 (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for ciso8601
Successfully built tap-db2
Failed to build ibm-db pendulum pyodbc ciso8601
ERROR: Failed to build installable wheels for some pyproject.toml based projects (ibm-db, pendulum, pyodbc, ciso8601)

Can someone help me?

✅ 1

👀 1