Meltano #random

Love Eklund

09/11/2024, 5:31 PM

Hi, I've been getting some problems due to this function _handle_max_record_age https://github.com/meltano/sdk/blob/6708cb995c68ab6f74d4874dfc8f978c3b054ceb/singer_sdk/target_base.py#L284 So I thought it would be interesting to understand why it works like it does, if someone knows. My problem is cause by the fact that it causes targets to write out states in the middle of a run. Is it there to "checkpoint" runs in that way or what is the reason for this? and what is the reason drain_all() (https://github.com/meltano/sdk/blob/6708cb995c68ab6f74d4874dfc8f978c3b054ceb/singer_sdk/target_base.py#L477) writes out a state?

Matt Menzenski

09/11/2024, 7:03 PM

I was today years old when I realized the the meltano logo is an “M”

melty bouncy 1

😲 3

Love Eklund

09/16/2024, 12:14 PM

Hey, I have a question regarding inheritance. Sometimes on inheritance meltano skips installing a plugin, but sometimes it installs the inherited plugin. How is that determined ? also, the output from the install command doesn't seem to align with what I have in the .meltano/extractors folder, see details bellow. This is my meltano.yaml

Copy code

plugins:
  extractors:
  - name: tap-oracle
    variant: s7clarke10
    pip_url:
      git+<https://github.com/s7clarke10/pipelinewise-tap-oracle.git@53bb75ed27d7796d2f492e74cec87f10f1bce4d4>
    config:
      use_ora_rowscn: false
      ora_python_driver_type: thick
      use_singer_decimal: true

  - name: tap-oracle-demo_foo
    inherit_from: tap-oracle
    config:
      user: demo
      host: 123
      port: 420
      service_name: foo

  - name: tap-oracle-demo_foo-full_table
    inherit_from: tap-oracle-demo_foo
    config:
      default_replication_method: FULL_TABLE

  - name: tap-oracle-demo_foo-full_table-bar_baz
    inherit_from: tap-oracle-demo_foo-full_table
    select:
     - bar-baz.*

  - name: tap-oracle-demo_foo-full_table-asd_qwe
    inherit_from: tap-oracle-demo_foo-full_table
    select:
      - asd-qwe.*

when running

meltano install

I get the following output.

Copy code

2024-09-16T09:45:39.392382Z [info     ] Installing 8 plugins          
2024-09-16T09:45:39.392942Z [info     ] Skipped installing extractor 'tap-oracle-demo_foo'
2024-09-16T09:45:39.393006Z [info     ] Skipped installing extractor 'tap-oracle-demo_foo-full_table-asd_qwe'
2024-09-16T09:45:39.393708Z [info     ] Installing extractor 'tap-testsource'
2024-09-16T09:45:39.408495Z [info     ] Installing extractor 'tap-oracle'
2024-09-16T09:45:39.454512Z [info     ] Installing extractor 'tap-oracle-demo_foo-full_table'
2024-09-16T09:45:39.587498Z [info     ] Installing extractor 'tap-oracle-demo_foo-full_table-bar_baz'
2024-09-16T09:45:39.763693Z [info     ] Installing loader 'target-jsonl'
2024-09-16T09:45:39.776647Z [info     ] Installing loader 'target-bigquery'
2024-09-16T09:45:43.563063Z [info     ] Installed loader 'target-jsonl'
2024-09-16T09:45:46.557149Z [info     ] Installed extractor 'tap-oracle'
2024-09-16T09:45:46.708771Z [info     ] Installed extractor 'tap-oracle-demo_foo-full_table'
2024-09-16T09:45:46.938197Z [info     ] Installed extractor 'tap-oracle-demo_foo-full_table-bar_baz'
2024-09-16T09:45:48.596501Z [info     ] Installed extractor 'tap-testsource'
2024-09-16T09:45:55.892412Z [info     ] Installed loader 'target-bigquery'
2024-09-16T09:45:55.892868Z [info     ] Installed 6/8 plugins         
2024-09-16T09:45:55.892947Z [info     ] Skipped installing 2/8 plugins

it skips the

asd_qwe

but installs the

bar_baz

why? I assume here due to

Skipped installing extractor 'tap-oracle-demo_foo'

the that no venv is created for the

tap-oracle-demo_foo

but this is the result I get in the .meltano folder ( see attached screenshot).

Facundo Miño

09/16/2024, 3:22 PM

Hello there! I’m wondering if a large csv file can be extracted by chunks and what is the best way to do it. I tried with

tap-spreadsheets-anywhere

and it works perfect reading until 100k rows, after that it got freezed my pc (i have a macbook pro m2). I’m quite new here so, any suggestions are well received!

haleemur_ali

09/16/2024, 8:42 PM

I was wondering what arch provides to folks using meltano. Is arch a hosted solution comparable to stitch? or is it something entirely different.

Tyler P

09/19/2024, 10:00 PM

I am running a custom tap that extracts from a REST API, in which all records have a primary key, so incremental syncs are working, and everything is great until the source/data producer deletes a record, and we are left with orphaned records that skew the data. Previously, we were using a custom Python script to compare source data to the destination table on every upload. Any idea on how to handle hard deletes from the source within Meltano and dbt? We are using a custom extractor and loading it to Postgres.

Facundo Miño

09/20/2024, 1:06 PM

Hello! I’m wondering if there is a way to know when a job ends inside a scheduler. I have set a scheduled job to run once a day and i want to trigger a message to api/email or somewhere if it failed or finish successfully (I’m guessing to add a custom script at the end of the job but if it fails for some reason it won’t be executed).

fraser__dagster_labs_

10/02/2024, 9:10 PM

OSA CON, the free, online open-source conference, is now open for registration.

3 days of events, 40+ sessions, and we expect 2,500 participants.

Join us November 19-21, 2024. The prizes for early registration are still up for grabs, I believe.

https://osacon.io/

Andrew Gelinas

10/30/2024, 2:08 AM

hey - im helping and trying to update the meltano tool ranking in stack wizard (https://stackwizard.com). we got a note that a few features/integrations CDC, upserts, etc. have become available since we added meltano to the ETL category. I made a spreadsheet showing what we have and don't have as available. is there someone who could confirm which features/integrations are live now?

meltano etl stack wizard.xlsx

👀 1

Reuben (Matatika)

11/07/2024, 10:54 AM

I think the discuss.meltano.com search is broken - from browser network inspector:

Copy code

{
  "message": "Forbidden - a valid `x-typesense-api-key` header must be sent."
}

👀 1

Cristina Munteanu

11/13/2024, 9:43 PM

📣 Hello all! The Open Source Analytics Conference 2024 is next week! 🚀 Some really cool talks on databases, orchestration, Bi/visualization tools! • When: Nov 19-21 • Where: Online! • More info: osacon.io Hope to see you there!

👀 1

Reuben (Matatika)

11/28/2024, 11:03 AM

Love it when companies just decide to pull the plug on certain REST API endpoints without any warning... ✨ (

tap-spotify

is now ~~partially~~ mostly broken) https://developer.spotify.com/blog/2024-11-27-changes-to-the-web-api

😬 1

Reuben (Matatika)

12/17/2024, 5:13 PM

Help, I've been banned 😢 I'm not doing anything suspicious, I promise

👀 1

Adam Wegscheid

01/09/2025, 1:59 PM

Please excuse my inexperience with Docker and the terminology used, but is there a repository available that has the meltano image build files? My team would like to adopt Meltano and move it into production but the provided images have critical vulnerabilities so we are lacking security approval. I would like to build the images myself to resolve them. Image: https://hub.docker.com/layers/meltano/meltano/v3.6.0-python3.11/images/sha256-a93fdc71cd79e793453fb7b94d119cd8c4374cb6d571a9c5e858e50dd9bca9b3

MONIKA RAJPUT

01/13/2025, 6:03 PM

Hey everyone! 👋 Excited to share Tirith, an open-source policy framework by StackGuardian. 🚀 Tirith scans Infrastructure as Code (IaC) configurations like Terraform against JSON-defined policies, making it easier to secure and comply with standards right from the start. Whether you're into building features, fixing bugs, or reviewing pull requests, we’d love to have your support and contributions! Check it out here: https://github.com/StackGuardian/tirith Give it a ⭐ if you find it useful, and feel free to drop in your thoughts or ideas. Let's build something awesome together! 😊 Also please join tirith community

Ian OLeary

01/14/2025, 7:22 PM

Is there anywhere to check my replication key values stored locally? Need to make sure my replication keys are being set properly for my custom tap streams EDIT: found it

meltano state list

meltano state get <stateid>

👍 1

ayoade_abel_adegbite

01/16/2025, 4:53 PM

Hi everyone, How do I get my tutorial featured on the Meltano YouTube Channel? Who should I be talking to ? Thank you

Ian OLeary

01/27/2025, 5:04 PM

So I'm struggling on the best way to integrate a new API endpoint. Most of my endpoints just require a "fromdate" and "todate" and return records created or updated in that range. I have one of these that returns a bunch of "customer_id"s for instance and updates associated with those customer records - this is how I'm creating my dimensions. However, if I want the customer detail from the API I need to pass in a list of all the customer id's (as a parameter) I want detail for. I'm wondering the best way to do this extract. Right now I think I would either have to create a new stream in my custom tap (new class and new logic) and make the connection to snowflake in that new stream logic to get all the rows and customer_id's (from my customer dim table or maybe the stg table) that I need to pass through as a parameter to the API and do it that way (query them, store in memory, loop through and pass them as params), OR I could create a python dbt model that runs on snowflake so any queries to my tables would probably be faster - but then if I'm doing the extract outside of my tap, I'd have to recreate the auth/token logic. Any ideas?

Adam Wegscheid

01/27/2025, 8:31 PM

I want to assign specific environment variables but I am getting a bit confused with the documentation. > • `MELTANO_EXTRACT_SETTING_NAME`: one environment variable for each of the extractor's settings and extras, e.g.

MELTANO_EXTRACT_PRIVATE_TOKEN

for the

private_token

setting, and

MELTANO_EXTRACT__LOAD_SCHEMA

for the

load_schema

extra > • `SETTING_ENV`: all of the extractor's regular configuration environment variables, as listed by

meltano config <plugin> list

, e.g.

TAP_GITLAB_API_URL

for the

api_url

setting 1. When do you use one underscore before the setting name and when do you use two? a. MELTANO_EXTRACT_PRIVATE_TOKEN b. MELTANO_EXTRACT__LOAD_SCHEMA 2. Is MELTANO_EXTRACT and TAP_GITLAB interchangeable? For example, will MELTANO_EXTRACT_API_URL have the same impact on the run as TAP_GITLAB_API_URL?

Kartik Shah

01/28/2025, 8:52 AM

Has the meltano team considered upgrading their slack subscription or moving to an open source alternative like mattermost (FOSS slack clone)? With slack's free plan we can only have messages from the last 90 days and everything older gets auto deleted, and that's not ideal imho since there's so much troubleshooting advise that gets lost. We had a similar problem in a company I worked for and we switched to mattermost to not have to pay the exorbitant slack fees and still keep our older messages.

✅ 1

Samaksh Agarwal

01/29/2025, 7:32 AM

Hey Team, I'm looking to discuss a potential partnership with Meltano for HackByte, Central India's biggest Hackathon in Jabalpur, India. We're expecting over 500 developers and tech enthusiasts from across country, and we believe Meltano could be a fantastic partner for our hackathon. Could someone point me to the best person to contact for partnership opportunities? I'd really appreciate it! Thanks in advance for your help! Samaksh Agarwal Organiser, HackByte 3.0 hackbyte.in

Nick Mitchum

02/13/2025, 4:00 PM

I have a question regarding the Cloud version of Meltano.. Arch Do you maintain the community versions with the same bug fixes that are applied in the Cloud version? Does the community fixes get integrated into the cloud version? From my experiences so far with Meltano Community Taps there are quite a few deal breaking bugs if I were paying for the service.

chiara

03/04/2025, 4:45 PM

Hi all 👋! Do you have an open source project that you think could turn into a business? You might like this talk that is talking place tomorrow, Wednesday 5 March. It takes examples from three companies (Percona, DBeaver, and Altinity) that built profitable businesses selling, supporting, and running open source software. Register here: https://altinity.com/events/build-a-great-business-on-open-source-without-selling-your-soul

👀 1

Taylor Murphy (Arch)

03/09/2025, 12:40 AM

Hack day project for Meltano??? https://www.dbreunig.com/2025/02/24/building-an-easier-to-use-ffmpeg-with-llm.html?_bhlid=3403a29bbf667ccd8aa6422cc2c8e57e8a61d610

👌 1

Ian OLeary

03/10/2025, 8:44 PM

Does anyone know if meltano dbt-snowflake supports key-pair auth? Is it just a matter of setting your private key path in profiles.yml? Are there any required env vars to set? Could I just reference the env vars I plan on using to my TARGET_SNOWFLAKE_PRIVATE_KEY_PATH or something like that?

✅ 1

Ian OLeary

03/12/2025, 11:51 PM

Dagster/docker users - do you guys dockerignore these folders which are gitignored in your root meltano project dockerfiles, or do they have any use in the container?

Samuel Nogueira Farrus

04/09/2025, 11:32 AM

Greetings, everyone! Not sure if this is the right place, but a suggestion for Meltano logging: show total elapsed time at the end of the run. Would be nice to have this information after an execution (it is true that it can easily be obtained when running on Airflow, Dagster or some orchestration tool, however when running locally for development purposes it would be a nice metric to have).

visch

04/13/2025, 8:14 PM

Nice usecase for tap-gmail

Copy code

WITH json_messages AS (
  SELECT payload::jsonb AS payload_json
  FROM tap_gmail.messages
), parsed as (
SELECT 
  header ->> 'value' AS sender
FROM json_messages,
     jsonb_array_elements(payload_json -> 'headers') AS header
WHERE header ->> 'name' = 'From'
)
select sender, count(*)
from parsed
group by sender 
order by count(*) desc

Copy code

- name: tap-gmail
    namespace: tap_gmail
    pip_url: git+<https://github.com/AutoIDM/tap-gmail.git>
    executable: tap-gmail
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: oauth_credentials.client_id
    - name: oauth_credentials.client_secret
    - name: oauth_credentials.refresh_token
    - name: user_id
    - name: messages.q
    config:
      messages.q: label:inbox
    select:
    - message_list.*
    - messages.*

No easy way to group Google Inboxes by Sender Here's some output as I was trying to 0 out my inbox that I was behind on

Copy code

Google Voice <voice-noreply@google.com>	51
Smart Plugin Manager <smart.plugin.manager@wpengine.com>	24
The Google Workspace Team <workspace-noreply@google.com>	20
HubSpot <noreply@notifications.hubspot.com>	19
Stripe <notifications@stripe.com>	18

🙌 3

Samuel Nogueira Farrus

04/30/2025, 2:11 PM

Greetings! Does anyone know if there is a way to dynamically use

tap-rest-api-msdk

?? My case: I want to extract info from Jenkins REST API, for each pipeline, so the URL paths would depend on their names, which can be changed, deleted or new ones can be created.

Samuel Nogueira Farrus

04/30/2025, 7:25 PM

Greetings (again)! Is there a way to set different

.env

variables according to environments?? My case: I have two environments named

prod1

and

prod2

and my

target-postgres

credentials are different for each, so I would like to have them both at the

.env

file and switch according to the environment, if possible (also, I do accept workarounds and bypasses).

✅ 1