What's the best way to go about ensuring a tap emits state at regular intervals with a timestamp-based replication key? Is this already handled by the SDK based on record count?

Are there any example of REST taps that stream responses? I'm trying to retry the request on an exception that occurs as a result of the following

iter_lines

call, but not having much success...

@override
    def _request(self, prepared_request, context):
        response = self.requests_session.send(
            prepared_request,
            stream=True,  # streaming request
            timeout=self.timeout,
            allow_redirects=self.allow_redirects,
        )
        self._write_request_duration_log(
            endpoint=self.path,
            response=response,
            context=context,
            extra_tags={"url": prepared_request.path_url}
            if self._LOG_REQUEST_METRIC_URLS
            else None,
        )
        self.validate_response(response)

        return response

    @override
    def parse_response(self, response):
        with response:  # ensure connection is eventually released
            yield from (
                json.loads(line, parse_float=decimal.Decimal)
                for line in response.iter_lines()
            )

Namely •

urllib3.exceptions.ProtocolError: Response ended prematurely

• (during handling of above)

requests.exceptions.ChunkedEncodingError: Response ended prematurely

and •

http.client.IncompleteRead: IncompleteRead(506 bytes read, 6 more expected)

• (directly caused by above)

urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(506 bytes read, 6 more expected)', IncompleteRead(506 bytes read, 6 more expected))

• (during handling of above)

requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(506 bytes read, 6 more expected)', IncompleteRead(506 bytes read, 6 more expected))

Hello, I am trying to connect to a fairly simple OData service. Authentication is simple with a basic username and password, there is a service root URI address, and the data transfer format is atom/xml instead of json. Is this something that the tap-rest-api-msdk extractor should be able to handle? Or is a custom connector typically required for OData? I haven't been able to find much specifically on Meltano and the OData protocol. Any help is much appreciated!

I think https://hub.meltano.com/extractors/tap-spreadsheets-anywhere can extract from a local JSON file. Are there specific requirements you have in mind?

Hey everyone 👋 Question for yall -> Has anyone ever succesfully removed the prefix "METRIC" from a singer SDK metric? I'm trying to implement full json logging and it's a bit difficult 😆 https://sdk.meltano.com/en/v0.43.1/implementation/logging.html#custom-logging-configuration I'm looking into

custom_logging_configurations

, but the

METRIC:

seems to be part of the log -> I can't remove it without creating a filter. Any ideas?

I was looking at some issues about selecting nested properties and had a follow up question related to this: https://github.com/meltano/meltano/issues/3024 https://github.com/meltano/meltano/issues/8807 is it not possible to use

select

to specify only the desired nested fields? is it always required to explicitly deselect all other unwanted fields? for

tap-example

, if there were many unwanted attributes, they would all need to be listed as

!entity.attributes.a

,

entity.attributes.b

, etc.,? is there no way to ignore all other nested fields aside from the ones selected?

plugins:
  extractors:
    - name: tap-example
      select:
      - entity.id
      # - entity.attributes             # without this, SDK will drop attributes
      # - entity.attributes.*           # without this, no nested properties will be selected
      - entity.attributes.name          # select wanted nested property
      # - "!entity.attributes.unwanted" # without this, all other nested properties will be selected

Hi team, We're currently using

tap-quickbooks

and encountering an issue related to the

start_date

format. In our

.yml

config file, we’ve defined the

start_date

as:

start_date: '2025-01-01T00:00:00.000Z'

However, we’re getting the following error when running the tap:

CRITICAL time data '2025-01-01T00:00:00+00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '2025-01-01T00:00:00+00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'

Could you please help us resolve this or point us in the right direction? Thanks!

I'm using the tap-hubspot, my goal is to extract all contacts data, run some transformations in my DWH, then write the enriched data back to hubspot for some unknown reason, every since many records were added in hubspot (it's a sandbox environment) - the tap is looping endlessly extracting data, with many duplicates being written over and over again to my target. the records don't seem to be different, so i suspect the tap is not using a proper paging mechanism to progress through the extraction? here's my

meltano.yml

:

version: 1
default_environment: dev
environments:
- name: dev
- name: staging
- name: prod
state_backend:
  uri: <s3://dwh/meltano-states/>
s3:
    aws_access_key_id: ${AWS_ACCESS_KEY_ID}
    aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
plugins:
  extractors:
  - name: tap-hubspot
    # python: python
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-hubspot.git@v0.6.3>
    config:
      start_date: '2020-01-01'
    select:
    - contacts.*
  loaders:
  - name: target-s3
    variant: crowemi
    pip_url: git+<https://github.com/crowemi/target-s3.git>
    config:
      append_date_to_filename: true
      append_date_to_filename_grain: microsecond
      partition_name_enabled: true
  - name: target-s3--hubspot
    inherit_from: target-s3
    config:
      format:
        format_type: parquet
      prefix: dwh/hubspot
      flattening_enabled: false

Greetings! I tried to use extractor

tap-db2

but it returned an pip/wheel error when attempting to install:

(venv) PS C:\meltano\db2> meltano add extractor tap-db2
Cloning <https://github.com/mjsqu/tap-db2.git> to c:\temp\<user>\pip-req-build-km0p0dgy
  Running command git clone --filter=blob:none --quiet <https://github.com/mjsqu/tap-db2.git> 'C:\TEMP\<user>\pip-req-build-km0p0dgy'
  Resolved <https://github.com/mjsqu/tap-db2.git> to commit ea2cd49b9fcb4dd599e66249445d8c0d8b06d6d4
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting attrs==23.1.0 (from tap-db2==1.0.6)
  Using cached attrs-23.1.0-py3-none-any.whl.metadata (11 kB)
Collecting ibm-db-sa==0.4.0 (from tap-db2==1.0.6)
  Using cached ibm_db_sa-0.4.0-py3-none-any.whl.metadata (5.3 kB)
Collecting ibm-db==3.2.0 (from tap-db2==1.0.6)
  Using cached ibm_db-3.2.0.tar.gz (206 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting jinja2==3.1.2 (from tap-db2==1.0.6)
  Using cached Jinja2-3.1.2-py3-none-any.whl.metadata (3.5 kB)
Collecting markupsafe<2.2.0 (from tap-db2==1.0.6)
  Using cached markupsafe-2.1.5-py3-none-any.whl
Collecting pendulum==2.1.2 (from tap-db2==1.0.6)
  Using cached pendulum-2.1.2.tar.gz (81 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pyodbc==5.0.1 (from tap-db2==1.0.6)
  Using cached pyodbc-5.0.1.tar.gz (115 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pytz>=2018.1 (from tap-db2==1.0.6)
  Using cached pytz-2025.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting singer-python>=5.12.0 (from tap-db2==1.0.6)
  Using cached singer_python-6.1.1-py3-none-any.whl
Collecting sqlalchemy<3.0.0 (from tap-db2==1.0.6)
  Using cached sqlalchemy-2.0.40-cp313-cp313-win_amd64.whl.metadata (9.9 kB)
Collecting python-dateutil<3.0,>=2.6 (from pendulum==2.1.2->tap-db2==1.0.6)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytzdata>=2020.1 (from pendulum==2.1.2->tap-db2==1.0.6)
  Using cached pytzdata-2020.1-py2.py3-none-any.whl.metadata (2.3 kB)
Collecting six>=1.5 (from python-dateutil<3.0,>=2.6->pendulum==2.1.2->tap-db2==1.0.6)
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting greenlet>=1 (from sqlalchemy<3.0.0->tap-db2==1.0.6)
  Using cached greenlet-3.2.1-cp313-cp313-win_amd64.whl.metadata (4.2 kB)
Collecting typing-extensions>=4.6.0 (from sqlalchemy<3.0.0->tap-db2==1.0.6)
  Using cached typing_extensions-4.13.2-py3-none-any.whl.metadata (3.0 kB)
Collecting jsonschema==2.*,>=2.6.0 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached jsonschema-2.6.0-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting simplejson==3.*,>=3.13.2 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached simplejson-3.20.1-cp313-cp313-win_amd64.whl.metadata (3.4 kB)
Collecting backoff==2.*,>=2.2.1 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached backoff-2.2.1-py3-none-any.whl.metadata (14 kB)
Collecting ciso8601==2.*,>=2.3.1 (from singer-python>=5.12.0->tap-db2==1.0.6)
  Using cached ciso8601-2.3.2.tar.gz (28 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Using cached ibm_db_sa-0.4.0-py3-none-any.whl (31 kB)
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached sqlalchemy-2.0.40-cp313-cp313-win_amd64.whl (2.1 MB)
Using cached greenlet-3.2.1-cp313-cp313-win_amd64.whl (295 kB)
Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
Using cached backoff-2.2.1-py3-none-any.whl (15 kB)
Using cached jsonschema-2.6.0-py2.py3-none-any.whl (39 kB)
Using cached simplejson-3.20.1-cp313-cp313-win_amd64.whl (75 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Using cached typing_extensions-4.13.2-py3-none-any.whl (45 kB)
Building wheels for collected packages: tap-db2, ibm-db, pendulum, pyodbc, ciso8601
  Building wheel for tap-db2 (pyproject.toml): started
  Building wheel for tap-db2 (pyproject.toml): finished with status 'done'
  Created wheel for tap-db2: filename=tap_db2-1.0.6-py3-none-any.whl size=29948 sha256=ab8ca931a326cb0937229d903708ff208bdded393e366c8e6eb2d1833290179e
  Stored in directory: C:\TEMP\<user>\pip-ephem-wheel-cache-w9wzy1j_\wheels\67\43\15\a99e5c72b4b3dcd727d50dfa99a0647c44e30ae3cc0f543b84
  Building wheel for ibm-db (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for ibm-db (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for ibm-db (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for ibm-db
  Building wheel for pendulum (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for pendulum (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for pendulum (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for pendulum
  Building wheel for pyodbc (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for pyodbc (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for pyodbc (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for pyodbc
  Building wheel for ciso8601 (pyproject.toml): started
  error: subprocess-exited-with-error

  Building wheel for ciso8601 (pyproject.toml) did not run successfully.
  exit code: 1

  See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for ciso8601 (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for ciso8601
Successfully built tap-db2
Failed to build ibm-db pendulum pyodbc ciso8601
ERROR: Failed to build installable wheels for some pyproject.toml based projects (ibm-db, pendulum, pyodbc, ciso8601)

Can someone help me?

How can I set the log level of an SDK tap? I've look through the code and it looks like

TAP_<NAME>_LOG_LEVEL=ERROR

should work, but doesn't...

Hi Team, I am using the "tap-sftp" from singer https://github.com/singer-io/tap-sftp Getting one issue saying

paramiko.ssh_exception.SSHException: Incompatible ssh peer (no acceptable host key)

For this I made a change in the client.py

self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')

def __try_connect(self):
        if not self.__active_connection:
            try:
                self.transport = paramiko.Transport((self.host, self.port))
                self.transport.use_compression(True)
                self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')
                self.transport.connect(username = self.username, pkey = self.key)
                self.sftp = paramiko.SFTPClient.from_transport(self.transport)
            except (AuthenticationException, SSHException) as ex:
                self.transport.close()
                self.transport = paramiko.Transport((self.host, self.port))
                self.transport.use_compression(True)
                self.transport._preferred_keys = ('ssh-rsa', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-ed25519', 'ssh-dss')
                self.transport.connect(username= self.username, pkey = None)
                self.sftp = paramiko.SFTPClient.from_transport(self.transport)
            self.__active_connection = True
            # get 'socket' to set the timeout
            socket = self.sftp.get_channel()
            # set request timeout
            socket.settimeout(self.request_timeout)

Even after making this change, its not resolving the issue.

I writing a tap for reading OpenOffice Calc (ODS) files from a versioned S3 bucket. I know there's tap-spreadsheets-anywhere but it neither supports ODS nor versioned buckets, it's architecturally incompatible with pyexcel and is unmaintained. The ODS files have multiple sheets, but the same schema across files, so all files produce the same streams and all streams span across multiple files. Naturally the tap wouldn't produce records stream by stream, but file by file, with streams intertwined. That's not how the singer-sdk want's me to do things. Is this mode supported by the sdk (I'm assuming that Meltano will be fine with it)? Is there an example tap I could look at?

I find the role of

def discover_streams

unclear, if it's called even if --catalog is passed. I'd expect it to be called only without the catalog or with --discover, when a discovery is actually needed

Team, Do we have tap-onedrive from singer side ? Has anyone ever tried it ?

hello, i have a question not sure where to ask it: 1. I’m currently trying to extract (tap-salesforce) data from salesforce and load it onto a json file (target-jsonl). 2. Then I want to extract (tap-singer-jsonl) from the json and load it onto mongodb (target-mongodb). however, the tap-singer-jsonl does not successfully extract the data from the json file. I also tried using targer-singer-jsonl instead of target-jsonl (in step 1) to test the singer config, but it says “run invocation could not be completed as block failed: Loader failed. Seems like a singer issue since I was able to do step 1 with target-jsonl

Does anyone know what’s going on?

Hello team, we are using tap-quickbooks for fetching QBO data (https://github.com/hotgluexyz/tap-quickbooks), but it does not have RefundReceipt, Check & CreditCardCredit record types data given by this repo, so how to get these records? Can you guys help get these record types?.

Hello team, https://github.com/MeltanoLabs/tap-linkedin-ads --- We are using LinkedIn singer tap, from June 1, we started to get the 426 client error as it seems LinkedIn has deprecated old API endpoints. Also, noticed this tap is using LinkedIn-Version 2024 in the headers. ---

error:          2025-06-10T02:55:07.873581Z [info     ] 2025-06-10 02:55:07,872 | ERROR    | tap-linkedin-ads.accounts | An unhandled error occurred while syncing 'accounts' cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap
-linkedin-ads                                                                                                                                                                                                                                                                                                                                                                      
2025-06-10T02:55:07.875835Z [info     ]     raise FatalAPIError(msg)   cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap-linkedin-ads                                                                                          
2025-06-10T02:55:07.875945Z [info     ] singer_sdk.exceptions.FatalAPIError: 426 Client Error: Upgrade Required for path: /rest/adAccounts cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap-linkedin-ads                      
2025-06-10T02:55:07.880461Z [info     ]     raise FatalAPIError(msg)   cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap-linkedin-ads                                                                                          
2025-06-10T02:55:07.880569Z [info     ] singer_sdk.exceptions.FatalAPIError: 426 Client Error: Upgrade Required for path: /rest/adAccounts cmd_type=elb consumer=False job_name=prod:tap-linkedin-ads-to-target-clickhouse:UMOJn5gijo name=tap-linkedin-ads producer=True run_id=ab987cd0-89aa-4d5a-b179-8fb04e6d3f7d stdio=stderr string_id=tap-linkedin-ads                      
2025-06-10T02:55:16.772779Z [error    ] Extractor failed                                                                                                                                                                                                                                                                                                                           
2025-06-10T02:55:16.772957Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Extractor failed') exit_codes={: 1} set_number=0 success=False

Hello team, Anyone is using snowflake tap https://github.com/MeltanoLabs/tap-snowflake? I noticed its not maintaining any bookmarks in state.json for the tables. Also I dont seem to find a setting for start_date. For instance: below state.json after successfully pulling first time

{
  "completed": {
    "singer_state": {
      "bookmarks": {
        "dw_hs-dim_accounts": {},
        "dw_hs-dim_activities": {
          "starting_replication_value": null
        }
      }
    }
  },
  "partial": {}
}

I'm using

tap-salesforce

and am looking for the config to pass WHERE clauses into the SOQL that Meltano generates. I need to filter to particular values in a particular column in the Account object to make sure specific records aren't extracted. Chat gpt is leading me astray with configs that don't actually exist...does one exist?

Hi all, I got errors during my last run of extracting data using tap-oracle, variant s7clarke10, replication method log_based and I want to debug them. For that purpose, I have a rather basic question: How do I change the Log-level for tap-oracle? I tried running meltano with --log-level=debug or --log-level=info, but that has no effect on the output from tap-oracle. I figured out that tap-oracle uses singer.get_logger. So I suppose I have to adjust its settings. Any hints how to do so? I'd like to get those info messages either printed to terminal or in a log file like LOGGER.info("Running in thick mode")

Hello everyone, I’m experiencing an issue with 

tap-rest-api-msdk

when making POST requests. Has anyone resolved this or can suggest an alternative tap I could use?

Hi everyone, I have a use case where I need to extract the data from mysql and load it into bigquery. I want to use meltano's tap-mysql for this. One doubt I have is, the incremental load should happen through the updated_at column and according to standard ETL process, checking the destination before proceeding with the load is good. Is there a way, I can pass a WHERE condition or even a filter condition to the tap before I run the job?

Hey, I hope I'm reaching out in the right channel. I've created a custom catalog from a tap (based on the existing one) to add a new field. However, in some cases, this field is not returned at all by the REST API — not even as

null

, but completely missing — which leads to a Singer exception. I was wondering if there's a catalog's property designed to handle this kind of situation? I’ve already tried defining the field as nullable and using

additionalProperties

, but I’m still encountering the Singer error when the field is absent from the object. Here is the Singer error :

2025-07-25T10:06:57.048305Z [error  ] Loading failed        code=1 message="singer_sdk.exceptions.InvalidRecord: Record Message Validation Error: {'sub_prop_1': 'abc', 'sub_prop_2': 'def'} is not of type 'string'"

Here is what i have tried in the catalog :

{
  "streams": [
    {
      "tap_stream_id": "obj",
      ...,
      "schema": {
        "properties": {
          "prop_1": {
            "type": ["array", "null"],
            "items": {
              "type": "object",
              "properties": {
                "sub_prop_1": { "type": ["string", "null"] },
                "sub_prop_2": { "type": ["string", "null"] },
                "optional_sub_prop_3": { "type": ["string", "null"] }
              },
              "additionalProperties": true
            }
          }
        }
      }
    }
  ]
}

Thanks in advance to anyone who takes the time to help ☺️

What is the point of

select_filter

? Isn't

select

a kind of filtering mechanism by definition? Why would I need a filter for a filter? 😅

Hi everyone 👋 I'm working with the community-managed

tap-quickbooks

and noticed that some secrets (like API keys or tokens) seem to be logged in plain text during execution. From what I understand, there’s a

_make_request

method in the tap that logs the URL and the full body of the POST request used to request a token — which includes API secrets.

[...]

def _make_request(self, http_method, url, headers=None, body=None, stream=False, params=None, sink_name=None):
        if http_method == "GET":
            <http://LOGGER.info|LOGGER.info>("Making %s request to %s with params: %s", http_method, url, params)
            resp = self.session.get(url, headers=headers, stream=stream, params=params)
        elif http_method == "POST":
            <http://LOGGER.info|LOGGER.info>("Making %s request to %s with body %s", http_method, url, body)
            resp = <http://self.session.post|self.session.post>(url, headers=headers, data=body)
        else:
            raise TapQuickbooksException("Unsupported HTTP method")

[...]

Is there a way in Meltano to prevent secrets from being written to log files if the logging is done by the tap itself? Or is this considered a tap-specific issue that should be addressed on GitHub? 🤷‍♂️ Thanks in advance for any insights!

Hello everyone, I need some advice on the QuickBooks tap. QuickBooks uses OAuth2, where the refresh token gets updated roughly every day. Although this connector has a mechanism to capture the new refresh token when it’s updated, since there’s no write-back capability to the tap and target settings (as far as I understand – see issue #2660), the new refresh token value just gets lost. I wanted to ask: if someone has experience with this tap, how do you handle this? The only workaround I can think of is an additional helper script that runs right after the pipeline. This script would fetch the new token from the logs, where it’s stored as plain text (which isn’t ideal, but in this case, it’s useful). Currently, I’m running Meltano in a container, so what I’m trying now is to: 1. Pack the additional Python script in the same container. 2. Mount the

.env

file with the token. 3. Let the pipeline run, capturing the new token if there is one, and saving it to the log. 4. Have the Python script fetch it as soon as the pipeline is done. 5. Update the value in the

.env

file so the next sync uses the new valid token. I don’t have a better idea at the moment, apart from forking the connector and modifying the logic there, which I’d prefer to avoid. Has anyone faced a similar scenario? What do you think of this solution? Any advice or suggestions? Many thanks in advance!

I'm looking to sync data from MongoDB and noticed there are several MongoDB tap variants on Meltano Hub: https://hub.meltano.com/extractors/tap-mongodb/ Has anyone tried these and have opinions on which one to use? I noticed the default one hasn't been updated in 2 years and incremental replication is not working.

i want to use

tap-google-sheets

to extract data from multiple google sheets (with different sheet IDs). How do I configure meltano for this?

Hello, I am looking for guidance on incremental handling of taps. I have developed some code and would like to get incremental updates right. Right now I am using a plain fs directory over fsspec and planning to add S3-compatible storage. I have files like the following with monthly/daily additions 2025-01.csv 2025-02.csv I follwowed https://sdk.meltano.com/en/latest/incremental_replication.html and https://sdk.meltano.com/en/latest/implementation/state.html so far What I have been working on is here https://github.com/celine-eu/tap-spreadsheets/blob/main/tap_spreadsheets/stream.py#L33-L41 Basically using a custom ___updated_at field_ to track row level progress and tracking the reference file mtime. So, I suspect to have reinvented the wheel :) My question, what is already managed by the SDK and what should I do in my own code? Thank you