https://linen.dev logo
Join Slack
Powered by
# singer-taps
  • s
    Hi! I am trying to move a lot of different ETL pipelines into meltano and one of our requirements is to tap specific files (json) from a github repo. Is there a tap that already allows you to tap specific files from a github repository? I did not find anything that meets my requirement, the closest was 
    tap-smoke-test
    but I am not sure if it fits our needs.
    e
    • 2
    • 2
  • p
    Hello, I now try to copy data from Postgres to any destination. I am using tap-postgres, default version. How can I filter the data? I don't want to copy all tables in the schema, this would be insane. I want to copy just some elected tables, and just some records. None of the known methods worked for me. This is the meltano.yml: 
    - name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
    config:
    sqlalchemy_url: ****
    filter_schemas:
    - myschema
    metadata:
    myschema-MYTABLE:
    replication-method: INCREMENTAL
    replication-key: KW_UPDATE_DATE
    I tried the following: • I tried to set filter_tables parameters but the tap does not allow it. • I tried replication_key_value (thinking I can block copying of particular tables by setting the value in the future) but it got ignored. • I tried to run invoke with parameter -s state.json but that parameter is not allowed • I looked at other variants of tap-postgres but they seem even more concise and not supportive of extra options My problem is that the current version of tap-postgres is copying the entire schema and all the data in all the tables in the schema. This is a no-go... Thank you.
    e
    • 2
    • 2
  • j
    does anyone have an update on https://github.com/singer-io/tap-dynamics-365?
    e
    • 2
    • 20
  • j
    much appreciated!
  • j
    @Edgar Ramírez (Arch.dev) Could you please guide me on how to flatten the 
    JSON
    data from the 
    MySQL
    tap configuration before it’s loaded into 
    BigQuery
    ? I’ve already set 
    flattening_enabled: true
    in the 
    BigQuery
    target configuration, but I’m not achieving the desired result. Here's my current `meltano.yml`:
    Copy code
    plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: 
      git+<https://github.com/transferwise/pipelinewise.git#subdirectory=singer-connectors/tap-mysql>
    select:
    - '*.*'      # Select all tables first
    - '!*_audit*'      # Then exclude audit tables
    metadata:
      '*.*':    # Apply metadata to all non-excluded tables
        replication-method: INCREMENTAL
        replication_key: update_time
        key_properties:
        - id

  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    executable: meltano-map-transform
    mappings:
    - name: mysql_to_bq
      config:
        flattening_enabled: true
        flattening_max_depth: 3   # flatten only top-level properties
        stream_maps:
          "*":
            __alias: "{{ stream_name | regex_replace('^smartterm_', '') }}"

  loaders:
  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
    config:
      batch_size: 10000
      flattening_enabled: true
      flattening_max_depth: 3
      partition_granularity: day
      dedupe_before_upsert: true
      timeout: 3600
      upsert: true
    e
    r
    • 3
    • 16
  • a
    Anyone considered or developed with 
    pytrends
    to create a google trends tap? Google don't have a public API for the data. https://github.com/GeneralMills/pytrends
    👀 1
  • j
    Copy code
    got the following error with oracle tap. any idea?    


root@c713d163edac:/projects/my-meltano-project# meltano config tap-oracle test
2025-01-29T22:51:16.797625Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
AttributeError: 'NoneType' object has no attribute 'get'
root@c713d163edac:/projects/my-meltano-project# meltano --log-level=debug invoke tap-oracle
2025-01-29T22:52:29.015603Z [debug    ] Meltano 3.6.0, Python 3.12.3, Linux (x86_64)
2025-01-29T22:52:29.017052Z [debug    ] /etc/timezone found, contents:
 America/Los_Angeles

2025-01-29T22:52:29.017327Z [debug    ] /etc/localtime found
2025-01-29T22:52:29.018028Z [debug    ] 2 found:
 {'/etc/timezone': 'America/Los_Angeles', '/etc/localtime is a symlink to': 'America/Los_Angeles'}
2025-01-29T22:52:29.020152Z [info     ] Environment 'dev' is active
2025-01-29T22:52:29.030326Z [debug    ] Creating DB engine for project at '/projects/my-meltano-project' with DB URI '<sqlite://projects/my-meltano-project/.meltano/meltano.db>'
2025-01-29T22:52:29.054840Z [debug    ] Found plugin parent            parent=tap-oracle plugin=tap-oracle source=LOCKFILE
2025-01-29T22:52:29.146614Z [debug    ] Skipped installing extractor 'tap-oracle'
2025-01-29T22:52:29.146846Z [debug    ] Skipped installing 1/1 plugins
2025-01-29T22:52:29.267378Z [debug    ] Created configuration at /projects/my-meltano-project/.meltano/run/tap-oracle/tap.e8454e4e-b596-4b5a-a3e1-7072a967ecb4.config.json
2025-01-29T22:52:29.267662Z [debug    ] Could not find tap.properties.json in /projects/my-meltano-project/.meltano/extractors/tap-oracle/tap.properties.json, skipping.
2025-01-29T22:52:29.267820Z [debug    ] Could not find tap.properties.cache_key in /projects/my-meltano-project/.meltano/extractors/tap-oracle/tap.properties.cache_key, skipping.
2025-01-29T22:52:29.267948Z [debug    ] Could not find state.json in /projects/my-meltano-project/.meltano/extractors/tap-oracle/state.json, skipping.
2025-01-29T22:52:29.268222Z [debug    ] Using cached catalog file
2025-01-29T22:52:29.268718Z [debug    ] Visiting CatalogNode.STREAM at '.streams[0]'.
2025-01-29T22:52:29.269016Z [debug    ] Setting '.streams[0].selected' to 'False'
2025-01-29T22:52:29.269110Z [debug    ] Setting '.streams[0].selected' to 'True'
2025-01-29T22:52:29.269170Z [debug    ] Setting '.streams[0].replication_method' to 'INCREMENTAL'
2025-01-29T22:52:29.269242Z [debug    ] Setting '.streams[0].replication_key' to 'ROWVERSION'
2025-01-29T22:52:29.269325Z [debug    ] Skipping node at '.streams[0].tap_stream_id'
2025-01-29T22:52:29.269399Z [debug    ] Skipping node at '.streams[0].table_name'
2025-01-29T22:52:29.269483Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.ABC_CLASS'.
2025-01-29T22:52:29.269569Z [debug    ] Skipping node at '.streams[0].schema.properties.ABC_CLASS.maxLength'
2025-01-29T22:52:29.269636Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.ABC_PERCENT'.
2025-01-29T22:52:29.269711Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.OBJKEY'.
2025-01-29T22:52:29.269770Z [debug    ] Skipping node at '.streams[0].schema.properties.OBJKEY.maxLength'
2025-01-29T22:52:29.269829Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.OBJVERSION'.
2025-01-29T22:52:29.269911Z [debug    ] Skipping node at '.streams[0].schema.properties.OBJVERSION.maxLength'
2025-01-29T22:52:29.269983Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.OBJID'.
2025-01-29T22:52:29.270051Z [debug    ] Skipping node at '.streams[0].schema.type'
2025-01-29T22:52:29.270133Z [debug    ] Skipping node at '.streams[0].stream'
2025-01-29T22:52:29.270218Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[0]'.
2025-01-29T22:52:29.270265Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '[]'
2025-01-29T22:52:29.270322Z [debug    ] Setting '.streams[0].metadata[0].metadata.selected' to 'False'
2025-01-29T22:52:29.270373Z [debug    ] Setting '.streams[0].metadata[0].metadata.selected' to 'True'
2025-01-29T22:52:29.270433Z [debug    ] Setting '.streams[0].metadata[0].metadata.replication-method' to 'INCREMENTAL'
2025-01-29T22:52:29.270506Z [debug    ] Setting '.streams[0].metadata[0].metadata.replication-key' to 'ROWVERSION'
2025-01-29T22:52:29.270584Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.schema-name'
2025-01-29T22:52:29.270667Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.database-name'
2025-01-29T22:52:29.270760Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.is-view'
2025-01-29T22:52:29.270835Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.selected'
2025-01-29T22:52:29.270904Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.replication-method'
2025-01-29T22:52:29.270961Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.replication-key'
2025-01-29T22:52:29.271014Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[1]'.
2025-01-29T22:52:29.271067Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'ABC_CLASS']'
2025-01-29T22:52:29.271160Z [debug    ] Setting '.streams[0].metadata[1].metadata.selected' to 'False'
2025-01-29T22:52:29.271219Z [debug    ] Setting '.streams[0].metadata[1].metadata.selected' to 'True'
2025-01-29T22:52:29.271270Z [debug    ] Skipping node at '.streams[0].metadata[1].breadcrumb[0]'
2025-01-29T22:52:29.271309Z [debug    ] Skipping node at '.streams[0].metadata[1].breadcrumb[1]'
2025-01-29T22:52:29.271366Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.sql-datatype'
2025-01-29T22:52:29.271438Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.inclusion'
2025-01-29T22:52:29.271525Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.selected-by-default'
2025-01-29T22:52:29.271595Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.selected'
2025-01-29T22:52:29.271671Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[2]'.
2025-01-29T22:52:29.271753Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'ABC_PERCENT']'
2025-01-29T22:52:29.271834Z [debug    ] Setting '.streams[0].metadata[2].metadata.selected' to 'False'
2025-01-29T22:52:29.271883Z [debug    ] Setting '.streams[0].metadata[2].metadata.selected' to 'True'
2025-01-29T22:52:29.271952Z [debug    ] Skipping node at '.streams[0].metadata[2].breadcrumb[0]'
2025-01-29T22:52:29.272016Z [debug    ] Skipping node at '.streams[0].metadata[2].breadcrumb[1]'
2025-01-29T22:52:29.272082Z [debug    ] Skipping node at '.streams[0].metadata[2].metadata.sql-datatype'
2025-01-29T22:52:29.272161Z [debug    ] Skipping node at '.streams[0].metadata[2].metadata.inclusion'
2025-01-29T22:52:29.272237Z [debug    ] Skipping node at '.streams[0].metadata[2].metadata.selected-by-default'
2025-01-29T22:52:29.272301Z [debug    ] Skipping node at '.streams[0].metadata[2].metadata.selected'
2025-01-29T22:52:29.272382Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[3]'.
2025-01-29T22:52:29.272473Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'OBJKEY']'
2025-01-29T22:52:29.272549Z [debug    ] Setting '.streams[0].metadata[3].metadata.selected' to 'False'
2025-01-29T22:52:29.272598Z [debug    ] Setting '.streams[0].metadata[3].metadata.selected' to 'True'
2025-01-29T22:52:29.272642Z [debug    ] Skipping node at '.streams[0].metadata[3].breadcrumb[0]'
2025-01-29T22:52:29.272739Z [debug    ] Skipping node at '.streams[0].metadata[3].breadcrumb[1]'
  • j
    Copy code
    2025-01-29T22:52:29.272860Z [debug    ] Skipping node at '.streams[0].metadata[3].metadata.sql-datatype'
2025-01-29T22:52:29.272993Z [debug    ] Skipping node at '.streams[0].metadata[3].metadata.inclusion'
2025-01-29T22:52:29.273059Z [debug    ] Skipping node at '.streams[0].metadata[3].metadata.selected-by-default'
2025-01-29T22:52:29.273121Z [debug    ] Skipping node at '.streams[0].metadata[3].metadata.selected'
2025-01-29T22:52:29.273186Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[4]'.
2025-01-29T22:52:29.273255Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'OBJVERSION']'
2025-01-29T22:52:29.273387Z [debug    ] Setting '.streams[0].metadata[4].metadata.selected' to 'False'
2025-01-29T22:52:29.273480Z [debug    ] Setting '.streams[0].metadata[4].metadata.selected' to 'True'
2025-01-29T22:52:29.273555Z [debug    ] Skipping node at '.streams[0].metadata[4].breadcrumb[0]'
2025-01-29T22:52:29.273627Z [debug    ] Skipping node at '.streams[0].metadata[4].breadcrumb[1]'
2025-01-29T22:52:29.273746Z [debug    ] Skipping node at '.streams[0].metadata[4].metadata.sql-datatype'
2025-01-29T22:52:29.273844Z [debug    ] Skipping node at '.streams[0].metadata[4].metadata.inclusion'
2025-01-29T22:52:29.273953Z [debug    ] Skipping node at '.streams[0].metadata[4].metadata.selected-by-default'
2025-01-29T22:52:29.274038Z [debug    ] Skipping node at '.streams[0].metadata[4].metadata.selected'
2025-01-29T22:52:29.274184Z [debug    ] Visiting CatalogNode.METADATA at '.streams[0].metadata[5]'.
2025-01-29T22:52:29.274307Z [debug    ] Visiting metadata node for tap_stream_id 'IFSAPP-ABC_CLASS', breadcrumb '['properties', 'OBJID']'
2025-01-29T22:52:29.274423Z [debug    ] Setting '.streams[0].metadata[5].metadata.selected' to 'False'
2025-01-29T22:52:29.274583Z [debug    ] Skipping node at '.streams[0].metadata[5].breadcrumb[0]'
2025-01-29T22:52:29.274689Z [debug    ] Skipping node at '.streams[0].metadata[5].breadcrumb[1]'
2025-01-29T22:52:29.274776Z [debug    ] Skipping node at '.streams[0].metadata[5].metadata.sql-datatype'
2025-01-29T22:52:29.274856Z [debug    ] Skipping node at '.streams[0].metadata[5].metadata.inclusion'
2025-01-29T22:52:29.274920Z [debug    ] Skipping node at '.streams[0].metadata[5].metadata.selected-by-default'
2025-01-29T22:52:29.274980Z [debug    ] Skipping node at '.streams[0].metadata[5].metadata.selected'
2025-01-29T22:52:29.275045Z [debug    ] Skipping node at '.streams[0].selected'
2025-01-29T22:52:29.275103Z [debug    ] Skipping node at '.streams[0].replication_method'
2025-01-29T22:52:29.275161Z [debug    ] Skipping node at '.streams[0].replication_key'
2025-01-29T22:52:29.276766Z [debug    ] Invoking: ['/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/bin/tap-oracle', '--config', '/projects/my-meltano-project/.meltano/run/tap-oracle/tap.e8454e4e-b596-4b5a-a3e1-7072a967ecb4.config.json', '--catalog', '/projects/my-meltano-project/.meltano/run/tap-oracle/tap.properties.json']
time=2025-01-29 14:52:29 name=singer level=INFO message=Running in cx mode
time=2025-01-29 14:52:29 name=singer level=WARNING message=cx_oracle is no longer maintained, use python-oracledb

To switch to python-oracledb set the environment variable ORA_PYTHON_DRIVER_TYPE=thin or thick.

Documentation for python-oracledb can be found here: <https://oracle.github.io/python-oracledb/>
time=2025-01-29 14:52:29 name=singer level=INFO message=Selected streams: ['IFSAPP-ABC_CLASS']
time=2025-01-29 14:52:29 name=singer level=INFO message=No currently_syncing found
time=2025-01-29 14:52:29 name=singer level=INFO message=Beginning sync of stream(IFSAPP-ABC_CLASS) with sync method(incremental)
time=2025-01-29 14:52:29 name=singer level=INFO message=Stream IFSAPP-ABC_CLASS is using incremental replication with replication key ROWVERSION
{"type":"SCHEMA","stream":"IFSAPP-ABC_CLASS","schema":{"properties":{"ABC_CLASS":{"maxLength":1,"type":["null","string"]},"ABC_PERCENT":{"type":["null","integer"]},"OBJKEY":{"maxLength":50,"type":["null","string"]},"OBJVERSION":{"maxLength":14,"type":["null","string"]},"OBJID":{}},"type":"object"},"key_properties":[],"bookmark_properties":["ROWVERSION"]}
time=2025-01-29 14:52:29 name=singer level=INFO message=dsn: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.10.131.67)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=<http://TEST10PDB.ifs10devdb.vcn10sj.oraclevcn.com|TEST10PDB.ifs10devdb.vcn10sj.oraclevcn.com>)))
{"type":"STATE","value":{"bookmarks":{"IFSAPP-ABC_CLASS":{"last_replication_method":"INCREMENTAL","replication_key":"ROWVERSION","version":1738191149591}},"currently_syncing":"IFSAPP-ABC_CLASS"}}
{"type":"ACTIVATE_VERSION","stream":"IFSAPP-ABC_CLASS","version":1738191149591}
time=2025-01-29 14:52:29 name=singer level=CRITICAL message='NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/bin/tap-oracle", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 686, in main
    raise exc
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 683, in main
    main_impl()
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 677, in main_impl
    do_sync(conn_config, args.catalog, args.config.get('default_replication_method'), state)
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 619, in do_sync
    state = sync_traditional_stream(conn_config, stream, state, sync_method_lookup[stream.tap_stream_id], end_scn)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 574, in sync_traditional_stream
    state = do_sync_incremental(conn_config, stream, state, desired_columns)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/__init__.py", line 460, in do_sync_incremental
    state = incremental.sync_table(conn_config, stream, state, desired_columns)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/my-meltano-project/.meltano/extractors/tap-oracle/venv/lib/python3.12/site-packages/tap_oracle/sync_strategies/incremental.py", line 63, in sync_table
    replication_key_sql_datatype = md.get(('properties', replication_key)).get('sql-datatype')
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
2025-01-29T22:52:29.621253Z [debug    ] Deleted configuration at /projects/my-meltano-project/.meltano/run/tap-oracle/tap.e8454e4e-b596-4b5a-a3e1-7072a967ecb4.config.json
  • j
    please ignore
  • j
    it was using the view, that is why error occur
    👍 1
  • i
    Hi there 🙂 I am kinda new in here and apologies if I shouldn't be posting any question in this channel. I started my Meltano experience with tap-taboola, but I get the same error as raised in here: https://github.com/singer-io/tap-taboola/issues/14 
    "float() argument must be a string or number, not 'NoneType'"
    Probably related to lines like this in init.py 
    'cpc': float(campaign.get('cpc', 0.0))
    Wondering if anyone managed to solve the issue yet?
    👀 1
    e
    • 2
    • 2
  • r
    What's the best way to go about ensuring a tap emits state at regular intervals with a timestamp-based replication key? Is this already handled by the SDK based on record count?
    e
    • 2
    • 7
  • r
    Are there any example of REST taps that stream responses? I'm trying to retry the request on an exception that occurs as a result of the following 
    iter_lines
    call, but not having much success...
    Copy code
    @override
    def _request(self, prepared_request, context):
        response = self.requests_session.send(
            prepared_request,
            stream=True,  # streaming request
            timeout=self.timeout,
            allow_redirects=self.allow_redirects,
        )
        self._write_request_duration_log(
            endpoint=self.path,
            response=response,
            context=context,
            extra_tags={"url": prepared_request.path_url}
            if self._LOG_REQUEST_METRIC_URLS
            else None,
        )
        self.validate_response(response)

        return response

    @override
    def parse_response(self, response):
        with response:  # ensure connection is eventually released
            yield from (
                json.loads(line, parse_float=decimal.Decimal)
                for line in response.iter_lines()
            )
    Namely • 
    urllib3.exceptions.ProtocolError: Response ended prematurely
    • (during handling of above) 
    requests.exceptions.ChunkedEncodingError: Response ended prematurely
    and • 
    http.client.IncompleteRead: IncompleteRead(506 bytes read, 6 more expected)
    • (directly caused by above) 
    urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(506 bytes read, 6 more expected)', IncompleteRead(506 bytes read, 6 more expected))
    • (during handling of above) 
    requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(506 bytes read, 6 more expected)', IncompleteRead(506 bytes read, 6 more expected))
    • 1
    • 3
  • m
    Hello, I am trying to connect to a fairly simple OData service. Authentication is simple with a basic username and password, there is a service root URI address, and the data transfer format is atom/xml instead of json. Is this something that the tap-rest-api-msdk extractor should be able to handle? Or is a custom connector typically required for OData? I haven't been able to find much specifically on Meltano and the OData protocol. Any help is much appreciated!
    e
    • 2
    • 2
  • e
    I think https://hub.meltano.com/extractors/tap-spreadsheets-anywhere can extract from a local JSON file. Are there specific requirements you have in mind?
  • s
    Hey everyone 👋 Question for yall -> Has anyone ever succesfully removed the prefix "METRIC" from a singer SDK metric? I'm trying to implement full json logging and it's a bit difficult 😆 https://sdk.meltano.com/en/v0.43.1/implementation/logging.html#custom-logging-configuration I'm looking into 
    custom_logging_configurations
    , but the 
    METRIC:
    seems to be part of the log -> I can't remove it without creating a filter. Any ideas?
    e
    • 2
    • 10
  • j
    I was looking at some issues about selecting nested properties and had a follow up question related to this: https://github.com/meltano/meltano/issues/3024 https://github.com/meltano/meltano/issues/8807 is it not possible to use 
    select
    to specify only the desired nested fields? is it always required to explicitly deselect all other unwanted fields? for 
    tap-example
    , if there were many unwanted attributes, they would all need to be listed as 
    !entity.attributes.a
    , 
    entity.attributes.b
    , etc.,? is there no way to ignore all other nested fields aside from the ones selected?
    Copy code
    plugins:
  extractors:
    - name: tap-example
      select:
      - entity.id
      # - entity.attributes             # without this, SDK will drop attributes
      # - entity.attributes.*           # without this, no nested properties will be selected
      - entity.attributes.name          # select wanted nested property
      # - "!entity.attributes.unwanted" # without this, all other nested properties will be selected
    e
    • 2
    • 1
  • c
    Hi team, We're currently using 
    tap-quickbooks
    and encountering an issue related to the 
    start_date
    format. In our 
    .yml
    config file, we’ve defined the 
    start_date
    as:
    Copy code
    start_date: '2025-01-01T00:00:00.000Z'
    However, we’re getting the following error when running the tap:
    Copy code
    CRITICAL time data '2025-01-01T00:00:00+00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '2025-01-01T00:00:00+00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'
    Could you please help us resolve this or point us in the right direction? Thanks!
    ✅ 1
    r
    • 2
    • 2
  • l
    I'm using the tap-hubspot, my goal is to extract all contacts data, run some transformations in my DWH, then write the enriched data back to hubspot for some unknown reason, every since many records were added in hubspot (it's a sandbox environment) - the tap is looping endlessly extracting data, with many duplicates being written over and over again to my target. the records don't seem to be different, so i suspect the tap is not using a proper paging mechanism to progress through the extraction? here's my 
    meltano.yml
    :
    Copy code
    version: 1
default_environment: dev
environments:
- name: dev
- name: staging
- name: prod
state_backend:
  uri: <s3://dwh/meltano-states/>
s3:
    aws_access_key_id: ${AWS_ACCESS_KEY_ID}
    aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
plugins:
  extractors:
  - name: tap-hubspot
    # python: python
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-hubspot.git@v0.6.3>
    config:
      start_date: '2020-01-01'
    select:
    - contacts.*
  loaders:
  - name: target-s3
    variant: crowemi
    pip_url: git+<https://github.com/crowemi/target-s3.git>
    config:
      append_date_to_filename: true
      append_date_to_filename_grain: microsecond
      partition_name_enabled: true
  - name: target-s3--hubspot
    inherit_from: target-s3
    config:
      format:
        format_type: parquet
      prefix: dwh/hubspot
      flattening_enabled: false
    e
    • 2
    • 2
  • s
    Greetings! I tried to use extractor 
    tap-db2
    but it returned an pip/wheel error when attempting to install:
    Copy code
    (venv) PS C:\meltano\db2> meltano add extractor tap-db2
Cloning <https://github.com/mjsqu/tap-db2.git> to c:\temp\<user>\pip-req-build-km0p0dgy
  Running command git clone --filter=blob:none --quiet <https://github.com/mjsqu/tap-db2.git> 'C:\TEMP\<user>\pip-req-build-km0p0dgy'
  Resolved <https://github.com/mjsqu/tap-db2.git> to commit ea2cd49b9fcb4dd599e66249445d8c0d8b06d6d4
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting attrs==23.1.0 (from tap-db2==1.0.6)
  Using cached attrs-23.1.0-py3-none-any.whl.metadata (11 kB)
Collecting ibm-db-sa==0.4.0 (from tap-db2==1.0.6)
  Using cached ibm_db_sa-0.4.0-py3-none-any.whl.metadata (5.3 kB)
Collecting ibm-db==3.2.0 (from tap-db2==1.0.6)
  Using cached ibm_db-3.2.0.tar.gz (206 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting jinja2==3.1.2 (from tap-db2==1.0.6)
  Using cached Jinja2-3.1.2-py3-none-any.whl.metadata (3.5 kB)
Collecting markupsafe<2.2.0 (from tap-db2==1.0.6)
  Using cached markupsafe-2.1.5-py3-none-any.whl
Collecting pendulum==2.1.2 (from tap-db2==1.0.6)
  Using cached pendulum-2.1.2.tar.gz (81 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pyodbc==5.0.1 (from tap-db2==1.0.6)
  Using cached pyodbc-5.0.1.tar.gz (115 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pytz>=2018.1 (from tap-db2==1.0.6)
  Using cached pytz-2025.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting singer-python>=5.12.0 (from tap-db2==1.0.6)
  Using cached singer_python-6.1.1-py3-none-any.whl
Collecting sqlalchemy<3.0.0 (from tap-db2==1.0.6)
  Using cached sqlalchemy-2.0.40-cp313-cp313-win_amd64.whl.metadata (9.9 kB)
Collecting python-dateutil<3.0,>=2.6 (from pendulum==2.1.2->tap-db2==1.0.6)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytzdata>=2020.1 (from pendulum==2.1.2->tap-db2==1.0.6)
  Using cached pytzdata-2020.1-py2.py3-none-any.whl.metadata (2.3 kB)
Collecting six>=1.5 (from python-dateutil<3.0,>=2.6->pendulum==2.1.2->tap-db2==1.0.6)
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting greenlet>=1 (from sqlalchemy<3.0.0->tap-db2==1.0.6)
  Using cached greenlet-3.2.1-cp313-cp313-win_amd64.whl.metadata (4.2 kB)
Collecting typing-extensions>=4.6.0 (from sqlalchemy<3.0.0->tap-db2==1.0.6)
  Using cached typing_extensions-4.13.2-py3-none-any.whl.metadata (3.0 kB)
Collecting jsonschema==2.*,>=2.6.0 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached jsonschema-2.6.0-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting simplejson==3.*,>=3.13.2 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached simplejson-3.20.1-cp313-cp313-win_amd64.whl.metadata (3.4 kB)
Collecting backoff==2.*,>=2.2.1 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached backoff-2.2.1-py3-none-any.whl.metadata (14 kB)
Collecting ciso8601==2.*,>=2.3.1 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached ciso8601-2.3.2.tar.gz (28 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Using cached ibm_db_sa-0.4.0-py3-none-any.whl (31 kB)
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached sqlalchemy-2.0.40-cp313-cp313-win_amd64.whl (2.1 MB)
Using cached greenlet-3.2.1-cp313-cp313-win_amd64.whl (295 kB)
Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
Using cached backoff-2.2.1-py3-none-any.whl (15 kB)
Using cached jsonschema-2.6.0-py2.py3-none-any.whl (39 kB)
Using cached simplejson-3.20.1-cp313-cp313-win_amd64.whl (75 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Using cached typing_extensions-4.13.2-py3-none-any.whl (45 kB)
Building wheels for collected packages: tap-db2, ibm-db, pendulum, pyodbc, ciso8601
  Building wheel for tap-db2 (pyproject.toml): started
  Building wheel for tap-db2 (pyproject.toml): finished with status 'done'
  Created wheel for tap-db2: filename=tap_db2-1.0.6-py3-none-any.whl size=29948 sha256=ab8ca931a326cb0937229d903708ff208bdded393e366c8e6eb2d1833290179e
  Stored in directory: C:\TEMP\<user>\pip-ephem-wheel-cache-w9wzy1j_\wheels\67\43\15\a99e5c72b4b3dcd727d50dfa99a0647c44e30ae3cc0f543b84
  Building wheel for ibm-db (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for ibm-db (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for ibm-db (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for ibm-db
  Building wheel for pendulum (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for pendulum (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for pendulum (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for pendulum
  Building wheel for pyodbc (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for pyodbc (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for pyodbc (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for pyodbc
  Building wheel for ciso8601 (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for ciso8601 (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for ciso8601 (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for ciso8601
Successfully built tap-db2
Failed to build ibm-db pendulum pyodbc ciso8601
ERROR: Failed to build installable wheels for some pyproject.toml based projects (ibm-db, pendulum, pyodbc, ciso8601)
    Can someone help me?
    ✅ 1
    👀 1
    r
    e
    • 3
    • 5
  • r
    How can I set the log level of an SDK tap? I've look through the code and it looks like 
    TAP_<NAME>_LOG_LEVEL=ERROR
    should work, but doesn't...
    e
    • 2
    • 12
  • s
    Hi Team, I am using the "tap-sftp" from singer https://github.com/singer-io/tap-sftp Getting one issue saying
    Copy code
    paramiko.ssh_exception.SSHException: Incompatible ssh peer (no acceptable host key)
    For this I made a change in the client.py
    Copy code
    self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')
    Copy code
    def __try_connect(self):
        if not self.__active_connection:
            try:
                self.transport = paramiko.Transport((self.host, self.port))
                self.transport.use_compression(True)
                self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')
                self.transport.connect(username = self.username, pkey = self.key)
                self.sftp = paramiko.SFTPClient.from_transport(self.transport)
            except (AuthenticationException, SSHException) as ex:
                self.transport.close()
                self.transport = paramiko.Transport((self.host, self.port))
                self.transport.use_compression(True)
                self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')
                self.transport.connect(username= self.username, pkey = None)
                self.sftp = paramiko.SFTPClient.from_transport(self.transport)
            self.__active_connection = True
            # get 'socket' to set the timeout
            socket = self.sftp.get_channel()
            # set request timeout
            socket.settimeout(self.request_timeout)
    Even after making this change, its not resolving the issue.
    ✅ 1
    r
    • 2
    • 3
  • r
    I writing a tap for reading OpenOffice Calc (ODS) files from a versioned S3 bucket. I know there's tap-spreadsheets-anywhere but it neither supports ODS nor versioned buckets, it's architecturally incompatible with pyexcel and is unmaintained. The ODS files have multiple sheets, but the same schema across files, so all files produce the same streams and all streams span across multiple files. Naturally the tap wouldn't produce records stream by stream, but file by file, with streams intertwined. That's not how the singer-sdk want's me to do things. Is this mode supported by the sdk (I'm assuming that Meltano will be fine with it)? Is there an example tap I could look at?
    a
    • 2
    • 16
  • r
    I find the role of 
    def discover_streams
    unclear, if it's called even if --catalog is passed. I'd expect it to be called only without the catalog or with --discover, when a discovery is actually needed
    e
    r
    • 3
    • 25
  • s
    Team, Do we have tap-onedrive from singer side ? Has anyone ever tried it ?
    e
    • 2
    • 2
  • a
    hello, i have a question not sure where to ask it: 1. I’m currently trying to extract (tap-salesforce) data from salesforce and load it onto a json file (target-jsonl). 2. Then I want to extract (tap-singer-jsonl) from the json and load it onto mongodb (target-mongodb). however, the tap-singer-jsonl does not successfully extract the data from the json file. I also tried using targer-singer-jsonl instead of target-jsonl (in step 1) to test the singer config, but it says “run invocation could not be completed as block failed: Loader failed. Seems like a singer issue since I was able to do step 1 with target-jsonl
    ✅ 1
    a
    m
    • 3
    • 4
  • a
    Does anyone know what’s going on?
  • c
    Hello team, we are using tap-quickbooks for fetching QBO data (https://github.com/hotgluexyz/tap-quickbooks), but it does not have RefundReceipt, Check & CreditCardCredit record types data given by this repo, so how to get these records? Can you guys help get these record types?.
    e
    • 2
    • 1
  • a
    Hello team, https://github.com/MeltanoLabs/tap-linkedin-ads --- We are using LinkedIn singer tap, from June 1, we started to get the 426 client error as it seems LinkedIn has deprecated old API endpoints. Also, noticed this tap is using LinkedIn-Version 2024 in the headers. ---
    Copy code
    error:          2025-06-10T02:55:07.873581Z [info     ] 2025-06-10 02:55:07,872 | ERROR    | tap-linkedin-ads.accounts | An unhandled error occurred while syncing 'accounts' cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap
-linkedin-ads                                                                                                                                                                                                                                                                                                                                                                      
2025-06-10T02:55:07.875835Z [info     ]     raise FatalAPIError(msg)   cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap-linkedin-ads                                                                                          
2025-06-10T02:55:07.875945Z [info     ] singer_sdk.exceptions.FatalAPIError: 426 Client Error: Upgrade Required for path: /rest/adAccounts cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap-linkedin-ads                      
2025-06-10T02:55:07.880461Z [info     ]     raise FatalAPIError(msg)   cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap-linkedin-ads                                                                                          
2025-06-10T02:55:07.880569Z [info     ] singer_sdk.exceptions.FatalAPIError: 426 Client Error: Upgrade Required for path: /rest/adAccounts cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap-linkedin-ads                      
2025-06-10T02:55:16.772779Z [error    ] Extractor failed                                                                                                                                                                                                                                                                                                                           
2025-06-10T02:55:16.772957Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Extractor failed') exit_codes={: 1} set_number=0 success=False
    e
    • 2
    • 1
  • h
    Hello team, Anyone is using snowflake tap https://github.com/MeltanoLabs/tap-snowflake? I noticed its not maintaining any bookmarks in state.json for the tables. Also I dont seem to find a setting for start_date. For instance: below state.json after successfully pulling first time
    Copy code
    {
  "completed": {
    "singer_state": {
      "bookmarks": {
        "dw_hs-dim_accounts": {},
        "dw_hs-dim_activities": {
          "starting_replication_value": null
        }
      }
    }
  },
  "partial": {}
}
    ✅ 1
    m
    • 2
    • 2