https://linen.dev logo
Join Slack
Powered by
# plugins-general
  • s
    Is it possible to use stream maps to set the replication key for a stream?
    ✅ 1
    e
    • 2
    • 5
  • k
    I have a feeling that many of the problems I'm having, such as state not persisting, or scheduler env vars not propagating etc. may be related to the fact that I'm using an airbyte variant of extractor, and maybe it's not set-up correctly. I just added an extractor and stated using it. I did not have docker installed but I was able to do full sync, etc. So, my question is, is there a detailed guide on what needs to be downloaded and ran in docker, and how the extractor needs to be set-up? I tried the configuration found on this page but now, I can't even pass in the required configurations (it throws an error). I may need a step-by-step guide on this. For example, the extractor page has a setting for 'Airbyte Spec Image' with default value 
    airbyte/source-hubspot
    . Where do I get this spec image and how do I set it up to be ran? Thank you.
    e
    c
    • 3
    • 3
  • r
    If Meltano is installed with 
    uv
    , does 
    uv
    become the default virtual environment backend for any newly added/installed plugins, or will I need to set venv.backend to 
    uv
    regardless?
    e
    • 2
    • 1
  • i
    `2024-09-24T163612.093898808Z 2024-09-24 163612 +0000 - dagster - WARNING - /project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dagster/_core/utils.py110 UserWarning: Found version mismatch between 
    dagster
    (1.8.8)and 
    dbt-core
    (1.8.0)` Is this safe to ignore for now? It seems like my jobs are still running in dagster. I'm getting the error with dbt-core v 1.8.6 as well
  • d
    Hi Everyone, I am using 
    tap-postgres
    and 
    target-snowflake
    to move data from postgres to Snowflake. In my 
    meltano.yaml
    I am providing the target_snowflake_schema but after execution I could see the data are getting loaded in 
    public
    schema and not the one I passed. Can anyone please help me if I am missing anything here
    e
    • 2
    • 4
  • m
    Hi all, with the help of the Slack channel I stumpled upon some multi purpose taps: https://hub.meltano.com/extractors/tap-rest-api-msdk https://hub.meltano.com/extractors/tap-spreadsheets-anywhere Outside of the Hub also exists this one: https://github.com/MeltanoLabs/tap-universal-file Are there any other multipurpose plugins one should know?
    v
    • 2
    • 5
  • s
    Hello, team! I want to use two different variant of target-csv. 
    name: target-csv
    variant: meltanolabs
    and 
    name: target-csv
    variant: hotgluexyz
    Is there anyway to not skip neither of them and install both of them when I do meltano install? this is my meltano.yml :
    Copy code
    loaders:
  - name: target-csv--hotglue
    inherit_from: target-csv
    variant: hotgluexyz
    pip_url: git+<https://github.com/hotgluexyz/target-csv.git>
  - name: target-csv--meltano
    inherit_from: target-csv
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-csv.git>
    I would be happy if someone can help me :)
    v
    • 2
    • 4
  • r
    Hi everyone, I've been bashing my head against this one for a while now and was hoping someone here could help Im trying to do 
    meltano run tap-jira target-bigquery
    but I've been getting this error: Does anyone have an idea what could be causing this ? For more context I used the pipx route to install meltano
    v
    r
    • 3
    • 8
  • g
    Do you know an extractor for performing GraphQL queries? I didn't found at Meltano Hub. I will try the 
    tap-rest-api-msdk
    v
    • 2
    • 2
  • a
    Hi everyone, I am trying to use tap-gmail with meltano but after giving the parameters its giving an error Plugin configuration is invalid singer_sdk.exceptions.FatalAPIError: 403 Client Error: Forbidden for path: /gmail/v1/users/<>/messages
    👀 1
    r
    v
    • 3
    • 5
  • v
    hey team, help needed!! 👋 I'm trying to ingest Google Ads data with meltano, but unfortunately facing weird issues on the most popular taps. Is this the right place to ask for help? • matatika variant: ◦ fails with: 
    400 Client Error: Bad Request for path: /v16/customers:listAccessibleCustomers
    • singer variant: ◦ fails with: 
    key error: tap-google-ads.api-field-names
    considerations: • We've recently received approval from Google for a Basic Acess developer_token (which can query production accounts) and I am able to use purely Google Ads python SDK with python and get campaigns with this sample. • in the thread, I will post details of my meltano configs for attempting to ingest data with both taps What am I doing wrong?
  • h
    Hi I am using batch strategy for a 
    tap-mssql:target-snowflake
    pipeline. I noticed the 
    _sdc
    metadata fields are empty, which makes sense for the same reason you can't use stream_maps on batch messages. But I am really interested in having metadata, especially for downstream dbt models. How might I achieve this? Or if it's currently impossible, what might I contribute to achieve this? Pondering the options, maybe allowing taps to add record metadata before batching?
    v
    e
    • 3
    • 5
  • m
    Hi everyone, is anybody know a 
    tap-spreadsheets-anywhere
    that can detect duplicate columns of an Excel file with the same name.
    👀 1
  • a
    Is there a sql-based tap that allows me to run a custom query as if it were a table? i.e. instead of replicating three tables, in 
    meltano.yml
    I can define a single query that joins the three tables together, and sync that with a 'full table' strategy?
    👀 1
    v
    • 2
    • 1
  • m
    Hi all, i'm using a tap-mssql extractor and i have table names with periods in them. after a lot of trial and error i've been unsuccessful in figuring out a working format for the select. An example would be something like "MA-Company.Advisor". My warning is always something like "stream MA... was not found in catalog". Is there a secret to doing this...?
    ✅ 1
    e
    • 2
    • 3
  • n
    Hey all, I'm looking for those of us that have experience with the meltano postgres connectors! Im running into an error with 
    target-postgres
    loader:
    Copy code
    2025-02-09T06:54:37.214383Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Loader failed
    What doesn't make sense to me is that the config for both 
    tap-postgres
    and 
    target-postgres
    are the same but the loader seems to be failing. Has anyone run into this issue before? Happy to share more details!
    r
    e
    • 3
    • 5
  • d
    Hi all, does anybody know if there is an existing tap that will bring in Google Reviews from GMB? The Google Search Console taps do not seem to do it. I've searched high and low.
    e
    a
    • 3
    • 6
  • m
    qq - is there a way to use multiple versions of the same plugin?
    ✅ 1
    e
    • 2
    • 6
  • t
    has anyone developed a tap for Google Ad Manager (not google ads not DV360)?
  • i
    dagster.daemon.SchedulerDaemon - INFO - Evaluating schedule
    I started getting this whenever I start my container with dagster, has anyone run into this? I never got it until now - but the daemon keeps seeing that my assets haven't executed and then attempts to run all of them. I found this: http://github.com/dagster-io/dagster/issues/18294 where someone ran into a similar issue, but was curious if anyone else has run into this
    👀 1
  • a
    Hey team! Im playing around with meltano for the first time and trying to set up a basic elt pipeline within a devcontainer on a local environment - I am planning to do this starting with installing ClickHouse. I have meltano as the base image on this devcontainer and i am running 
    meltano add utility clickhouse
    followed by 
    meltano install
    However, the output is as follows
    Copy code
    root@0153664ae564:/workspaces/sh-data-platform/src/hub# meltano add utility clickhouse
Added utility 'clickhouse' to your project
Variant:        clickhouse (default)
Repository:     <https://github.com/ClickHouse/ClickHouse/>
Documentation:  <https://hub.meltano.com/utilities/clickhouse--clickhouse>

2025-03-17T19:52:06.382434Z [info     ] Skipped installing utility 'clickhouse'
2025-03-17T19:52:06.382756Z [info     ] Skipped installing 1/1 plugins

To learn more about utility 'clickhouse', visit <https://hub.meltano.com/utilities/clickhouse--clickhouse>
root@0153664ae564:/workspaces/sh-data-platform/src/hub# meltano install
2025-03-17T19:52:11.621647Z [info     ] Installing 1 plugins          
2025-03-17T19:52:11.639125Z [info     ] Skipped installing utility 'clickhouse'
2025-03-17T19:52:11.639525Z [info     ] Skipped installing 1/1 plugins
    Any ideas why it keeps skipping the installation of the plugin? Is there a way for me to run these commands with a higher level of verbosity? This is on a fresh container so there shouldnt be any conflict issues.
    r
    • 2
    • 6
  • j
    hey team! I saw this PR for adding jql filtering to the 
    tap-jira
    plugin. I don't see those fields described on the meltano hub page and it hasn't seemed to be working when I tested it locally. has this feature been released yet?
    e
    • 2
    • 29
  • f
    Hello World! I am new to Meltano - it seems awesome so far! But I am trying to do something I don't know how. I am using 
    tap-klaviyo
     to fetch data and load into 
    target-postgres
    . It works fine, but we have many millions of rows in the "event" table, so I would like to run meltano in batches. Only fetching few thousands of records per run. How can a approach this problem? Thank you!!
    e
    • 2
    • 6
  • t
    Hi there everyone! I'm new to Meltano and was really impressed with it! But I've ran into a bit of an issue. So I'm trying to use Meltano to replicate the extractions that we use currently use Airbyte to make. I'm currently working with the tap-facebook and I've come to find that the extractions we make using Airbyte rely on the Ads Insights reports. Because of that I'm trying to figure out how to use the "insights_reports_list" on tap-facebook but I wasn't able to find any further documentation on how to use it. Can somebody help better understand how to use this functionality?
    ✅ 1
    v
    • 2
    • 3
  • t
    @Gary James I did a major update to your tap-monday for monday.com - I just put in a PR for it: https://github.com/GtheSheep/tap-monday/pull/2
    👌 1
  • t
    Is anyone else experiencing this? I get a 404 when trying to install airflow
    Copy code
    [tanner@sato ubb-meltano]$ meltano add utility airflow
Plugin definition is already locked at /home/tanner/Projects/ubb-meltano/plugins/utilities/airflow--apache.lock.
You can remove the file manually to avoid using a stale definition.

2025-04-16T23:12:33.046375Z [info     ] Installing utility 'airflow'  
2025-04-16T23:12:35.495813Z [info     ] Logged pip install output to /home/tanner/Projects/ubb-meltano/.meltano/logs/pip/utilities/airflow/pip.log
2025-04-16T23:12:35.496198Z [error    ] Utility 'airflow' could not be installed: Failed to install plugin 'airflow'.
2025-04-16T23:12:35.496306Z [info     ] ERROR: 404 Client Error: Not Found for url: <https://raw.githubusercontent.com/apache/airflow/constraints-2.10.5/constraints-no-providers-3.13.txt>

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)
    e
    • 2
    • 1
  • a
    Question for those of you using GA4 data - just discovering the pitfalls of trying to replicate 'users' metrics, deduplication in BI tools. Has anyone managed to get the raw Bigquery GA4 exports into another dwh, either with meltano or via some other way?
    r
    • 2
    • 4
  • m
    Hi guys, I am trying to get the 
    tap-magento
    hotglue variant to flatten the 
    invoices
    stream such that the 
    items
    array value within the stream is exploded either into additional rows within the 
    invoices
    table or within its own separate 
    invoices_items
    table. The data would then be fed into the 
    target-redshift
    ticketswap variant. I have the relevant parameters set below in the config file, which I believe should be sufficient to flatten anything that comes out of the Magento tap.
    Copy code
    flattening_enabled: true
flattening_max_depth: 20
    Unfortunately, it doesn't appear that the flattening is occurring as the value for 
    items
    is a stringified list of JSON objects. From the singer SDK, it looks like the source code requires the value in the key-value pair in question to be a dict, not a string. To deserialize the JSON list, I have tried applying the 
    json.loads(items)
    command to the 
    stream_maps
    parameter to both 
    tap-magento
    and 
    target-redshift
    to no avail as shown below:
    Copy code
    stream_maps:
   invoices:
      items: json.loads(items)
    ========== When 
    json.loads(items)
    is applied to 
    tap-magento
    stream_maps
    , the following error occurs:
    Copy code
    singer_sdk.exceptions.MapExpressionError: Failed to evaluate simpleeval expressions json.loads(items)
singer_sdk.helpers._simpleeval.NameNotDefined: 'json' is not defined for expression 'json.loads(items)'
    Which eventually terminates in the following schema validation error for 
    invoices.items
    as 
    json.loads
    never executes:
    Copy code
    singer_sdk.exceptions.InvalidJSONSchema: Schema Validation Error: [['array', 'string', 'null'], 'null'] is not valid under any of the given schemas
    Whereby the valid schema for 
    invoices.items
    should be 
    ['array', 'string', 'null']
    without the double nested extra null value. This error is strange as the docs here suggest that the 
    json.loads
    command should be supported within the 
    stream_maps
    parameter? Or is `json.loads`not specifically supported by the 
    tap-magento
    plugin (as it does appear to work for the 
    target-redshift
    plugin)? If so, how could we get the desired flattening to occur? Another question would be whether 
    json.loads
    would be executed before or after the flattening by the singer SDK (of course, we would want it to happen before so that 
    items
    is recognized as a dict. ========== When 
    json.loads(items)
    is applied to 
    target-redshift
    stream_maps
    , the 
    items
    field is converted to the Redshift 
    super
    data format with the fields containing JSON objects, but the flattening does not occur. I believe this is expected behavior as 
    tap-magento
    should be the plugin performing the flattening, and 
    target-redshift
    does not appear to have the flattening parameter as usable. cc: @Taylor Gerik
    e
    • 2
    • 2
  • s
    Having trouble finding currently maintained taps for S3 CSVs. Anyone have any recommendations for this? 
    tap-s3-csv
    seems to have been sunset. I am trying to setup a pipeline using a CSV, and I am also unsure if any of these taps supports incremental replication.
    e
    m
    a
    • 4
    • 23
  • a
    Hi, I have a tap developed for Google My Business, which can return reviews and search metrics for your locations on Google Maps. I am planning to open source to MeltanoLabs - if anyone is interested in participating in some testing, particularly around the authorization and inviting, I would appreciate it. A google service is required.
    ❤️ 1