Hi we have an existing meltano instance on version

v1.105.0

that we're preparing to migrate to

v3.0.0

for. v1 is currently using a postgres database as its state backend. I've gone ahead and just created a new project running v3, rather than modifying the existing one. My question is would we be able to simply point the new project that's running v3 to the same backend uri while maintaining the existing bookmarks? i've been running v3 locally and inspecting the tables using the local file system in

.meltano/meltano.db

vs the tables in our postgres instance and it looks like v3 has some new job related tables

state

and

runs

vs v1 which only has the

job

table. Would someone be able to give me more info on how state was maintained in v1 vs any differences in v3?

Anyone have success using the Oracle target with tns and wallet authentication?

Hey, I want to create an alias of a stream in my target and I though of using

stream_maps

. My tap is tap-saasoptics and target is target-mysql When I do this config on the target:

..
      stream_maps:
        transactions:
          __alias__: my_prefix_transactions

I get:

singer_sdk.exceptions.RecordsWithoutSchemaException: A record for stream 'transactions' was encountered before a corresponding schema. Check that the Tap correctly implements the Singer spec.

When I do that config on the target:

..
      stream_maps:
        my_prefix_transactions:
          __source__: transactions

I get two tables being created in MySQL - my_prefix_transactions and transactions Any advice how to achieve that? I mean get only a single table created in MySQL db called my_prefix_transactions.

hello all, I was getting some data from

tap-google-search-console

but I'm getting duplicate data . I know meltano says that there could be duplicate data because of

atleast-once

guarantee between tap and target but not sure that is the case. Is this a recurring theme with users of

tap-google-search-console

is there any way to prevent Meltano from logging the intermediate results of a pipeline when executing

meltano run

? we have a pipeline of this form:

meltano run tap mapper target

where

mapper

is responsible for obfuscating data from one of the fields that's being emitted by

tap

. however the output of

tap

is leaking sensitive information into logs which has to be avoided

Hey guys, I’m trying to have meltano.yml read from environment variable as a way to only read specific configs depending on the dev environment. Currently I have

version: 1
project_id: <project_id>
default_environment: ${MELTANO_ENVIRONMENT}  
include_paths:
- ./environments/environments.meltano.yml
- ./extract/*.meltano.${MELTANO_ENVIRONMENT}.yml
- ./load/*.meltano.${MELTANO_ENVIRONMENT}.yml
- ./orchestrate/*.meltano.${MELTANO_ENVIRONMENT}.yml

but it doesn’t read the environment variable. I tried setting it also in environments.meltano.yml but it doesnt also work

👋 I'm using the wonderful z3z1ma target-bigquery package. I've loved setting up stream_maps for our data, however I am a bit stumped as to whether you can rename the json column (default set to 'data') using stream maps when not normalising. I've dug through the docs and it's not clear from what I've read. I also dug into the source code, and it looks like it may not be possible without denormalising the data. This is to bring consistency across two projects in our stack. Any ideas greatly appreciated 👌

Getting a 404 on your banner link to blog posting about Meltano Cloud in Public Beta: https://meltano.com/blog/introducing-meltano-cloud-you-build-the-pipelines-we-manage-the-infrastructure/

Hello 🤝 i get following error The default environment 'dev' will be ignored for

meltano config

. To configure a specific environment, please use the option

--environment=<environment name>

"meltano --environment=dev" is no good ? Trying to understand this environment difference issue, i already had this working one dev environment ,

Hello, Peeps! Having hard time fixing this error. I am running 'tap-mysql' 'target-s3' pipeline with default variants and am getting:

jsonschema.exceptions.ValidationError: '"{\\"uuid\\":\\"b0f01821-5ced-4ad1-a430-9c3765c1e777\\",\\"email\\":\\"<mailto:xyz@email.com|xyz@email.com>\"}"' is not of type 'null', 'object'

Hi, we are running into a customer issue for tap-snowflake where we are unable to validate configurations nor do imports. However, I am able to test select using meltano config custom-snowflake-tap test. Anyone have any ideas?

meltano select custom-snowflake-tap --list

is returning customer_schema-customer_table.* selected attributes However, when I run

meltano config custom-snowflake-tap test

, I get the error

Plugin configuration is invalid
No RECORD message received

the configurations for your reference.

- config:
      account: <redacted>
      database: CUSTOMER_DB
      password: <redacted>
      schema: customer_schema
      user: <redacted>
      warehouse: CUSTOMER_WH
      tables:
       - customer_schema.table_name
    inherit_from: tap-snowflake
    metadata: {}
    name: custom-snowflake-tap
    schema: {}
    select:
    - customer_schema-table_name.*
    select_filter: []

Hi awesome Meltano team! I'm unable to set Airflow's

executor

to

LocalExecutor

. Setup info: • Meltano version:

3.1.0

• Airflow (in

meltano.yaml

)

utilities:
- name: airflow
    variant: apache
    pip_url: psycopg2 apache-airflow-providers-postgres git+<https://github.com/meltano/airflow-ext.git@main>
      apache-airflow==2.3.3 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt>
    config:
      core:
        executor: LocalExecutor
      webserver:
        web_server_port: '8973'
      database:
        sql_alchemy_conn: <postgresql://jobyatp:xM2jaGOzfsOTe3o71TPBU1H8IzHvT6@172.29.1.3:5432/airflow>

• Double checked that there is no orchestrator version of airflow present. • I ran

meltano config airflow set core.executor LocalExecutor

However, this is what I get when I run `meltano invoke airflow config list`:

[core]
...
executor = SequentialExecutor

I just switched from using the Orchestrator version because it looks like the utilities version is preferred. Any help is appreciated!

HI Team, i am trying "meltano elt tap-postgres target-bigquery" to el a table from postgress to bigquery but cant find what causes this [error ] *Loading failed '*2023-11-03T050208.916482Z [error ] Loading failed code=1 message=google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/----tables?prettyPrint=false: Invalid table ID: "information_schema__pg_foreign_data_wrappers" uses a reserved prefix. name=meltano run_i". any hint / help is appreciated. thank you

Hey guys! I have a quick question concerning the Meltano State in S3: I have done the configuration to be able to store my state, and installed the necessary dependencies (i.e. boto3 s3) - the job goes through and is green, though nothing is written in the bucket - am I missing something? Thank you

Hi all, I'm having some issues that I cannot troubleshoot by myself any further... I created a new meltano project with a dedicated venv (meltano 3.1.0 installed and nothing else). After initializing a new project i get the scaffold, so I changed working directory into the newly created directory. after trying to install the extractor tap-parquet I get the following error: Extractor 'tap-parquet' could not be installed: Failed to install plugin 'tap-parquet'. Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community. Failed to install plugin(s) The logs didn't tell me particularly much either other than 'error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/', however after an installation of this software and verification of said software and a restart of my windows machine, the issue persists... Does anyone know how to resolve? Much appreciated, - Andreas

Hi team, is there a way to override stream attribute type without changing source code of the tap? In my case, I'm using

tap-braintree

through Airbyte, and getting the following error when trying to load payment transactions:

ValueError: No format in ['%Y-%m-%d %H:%M:%S'] matching 2023-10-03

hi all, I have been working to get a survey monkey tap stood up and am getting turned around on the config I was able to get the tap installed, and performed a test and was returned this error:

CRITICAL Config is missing required keys: ['start_date', 'access_token']

I tried troubleshooting a couple of ways, not sure either correct. 1. I modified the plugins>extractors>tap-surveymonkey--singer.io.lock to include the above (access token is aliased) 2. The other is I added the above as config items in our environments config file for the extractor environments>environment.yml a.

environments:

b.

- name: env

c.

config:

d.

plugins:

e.

extractors:

f.

- name: tap-surveymonkey

g.

config:

h.

start_date: '2023-01-01T00:00:00Z'

i.

access_token: $(alias)

I am now met with:

Plugin configuration is invalid
Exception: key_properties must be a string or list of strings

any thoughts are appreciated

Hi! Question around

tap-oracle

usage. Is this still the recommended go to? I see it was updated in august, but its dependency on a pipeline wise fork is from March. Is this generally pretty normal for metano taps? I’m looking for some basic replication from oracle and hoping that meltano fits the bill here, but a bit concerned thinking of a production pipeline relying on stale packages.

Hi, I'm trying to run

$ meltano install extractor tap-singer-jsonl

but am getting the error message

Failed to parse JSON array from string: 'None'

. Python:

Python 3.10.10

Meltano:

meltano, version 3.1.0

How can I get around this issue? EDIT: Seems to originate here:

/Users/user/data-project/meltano_pipeline/venv/lib/python3.10/site-packag │
│ es/meltano/core/settings_service.py:238 in config_with_metadata                                  │
│                                                                                                  │
│   235 │   │   │   if prefix and not setting_def.name.startswith(prefix):                         │
│   236 │   │   │   │   continue                                                                   │
│   237 │   │   │                                                                                  │
│ ❱ 238 │   │   │   value, metadata = self.get_with_metadata(                                      │
│   239 │   │   │   │   setting_def.name,                                                          │
│   240 │   │   │   │   setting_def=setting_def,                                                   │
│   241 │   │   │   │   source=source,                                                             │
│                                                                                                  │
│ /Users/user/data-project/meltano_pipeline/venv/lib/python3.10/site-packag │
│ es/meltano/core/settings_service.py:408 in get_with_metadata                                     │
│                                                                                                  │
│   405 │   │   │   │   │   value = object_value                                                   │
│   406 │   │   │   │   │   metadata["source"] = object_source                                     │
│   407 │   │   │                                                                                  │
│ ❱ 408 │   │   │   cast_value = setting_def.cast_value(value)                                     │
│   409 │   │   │   if cast_value != value:                                                        │
│   410 │   │   │   │   metadata["uncast_value"] = value                                           │
│   411 │   │   │   │   value = cast_value                                                         │
│                                                                                                  │
│ /Users/user/data-project/meltano_pipeline/venv/lib/python3.10/site-packag │
│ es/meltano/core/setting_definition.py:441 in cast_value                                          │
│                                                                                                  │
│   438 │   │   │   │   )                                                                          │
│   439 │   │   │   elif self.kind == SettingKind.ARRAY:                                           │
│   440 │   │   │   │   value = list(                                                              │
│ ❱ 441 │   │   │   │   │   self._parse_value(value, "array", Sequence),  # type: ignore           │
│   442 │   │   │   │   )                                                                          │
│   443 │   │                                                                                      │
│   444 │   │   processor = self.value_processor                                                   │
│                                                                                                  │
│ /Users/user/data-project/meltano_pipeline/venv/lib/python3.10/site-packag │
│ es/meltano/core/setting_definition.py:416 in _parse_value                                        │
│                                                                                                  │
│   413 │   │   │   ) as ex:                                                                       │
│   414 │   │   │   │   raise parse_error from ex                                                  │
│   415 │   │   if not isinstance(parsed, expected_type):                                          │
│ ❱ 416 │   │   │   raise parse_error                                                              │
│   417 │   │   return parsed                                                                      │
│   418 │                                                                                          │
│   419 │   def cast_value(self, value: t.Any) -> t.Any:  # noqa: C901

Is anyone using tap-spreadsheets-anywhere in an AWS environment using IAM roles? We developed an ELT pipeline locally to read from an s3 bucket and write the files to a target. In local development things work great and Meltano/the tap find the aws credentials we are using in our local credentials file to authenticate and read from the s3 bucket. However, when we deploy and run this with an AWS batch job with an IAM role with the same permissions, the tap shows that 0 streams can be found.

Hello everybody! I work with a pipeline with a custom tap-intacct and target-bigquery (isn't custom). It's works, but have some problems with some tables (doesn't load the data in bigquery) and get this error (image 1). But for tables with the same schema load the data. Also I have the issue to for some tables doesn't load all the data, only like the middle of the data, but don't get any error in the terminal and watch the Block Run Completed. When I ran my tap-intacct with target-csv, I could to get all the data for the tables. But when I tried with target-bigquery have these errors. For other hand, for some reason the last week, when I ran the pipeline for some tables that loaded the data in bigquery stopped doing it. The last friday, I used meltano invoke tap-intacct > output.json and cat output.json | meltano invoke target-bigquery, to know where is the problem, and get this (image 2) from the output.json. So someone have idea to what it is? What could be the error or the problem?

Hi all, I'm currently trying to upgrade my meltano-managed dbt project from v1.6 => v1.7, and have run into an apparent breaking change: dbt no longer allows cleaning paths outside the project directory by default, and meltano's default target path (

.meltano/transformers/dbt/target/

) is outside the dbt project directory (

transformer/

). As a result, any run gives an error:

2023-11-06T18:57:57.133172Z [info     ] Environment 'dev' is active
Extension executing `dbt clean`...
18:57:58  Running with dbt=1.7.0
18:57:58  Encountered an error:
Runtime Error
  dbt will not clean the following directories outside the project: ['.meltano/transformers/dbt/target']

This is similar to but (afaict) not the same as this open issue from last year. Just wondering if anyone else has had this issue and/or if meltano folks have specific recommendations about how to handle this change in dbt's functionality. Poking around dbt's code suggests that there may be a straightforward workaround -- adding a new envvar / cli flag -- but maybe there's a better or less clunky way to address this. Thanks in advance for your help!

Hi Team, New to Meltano, Requesting for a help to understand this error "[error ] Block run completed. block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False" and how do we troubleshoot with Meltano error's ?

Hello everyone, I'm relatively new to Meltano, and I'm encountering an issue in my environment. I'm utilizing Meltano to replicate data into a data warehouse, and this particular database requires the use of

FULL_TABLE

replication. Problem: I'm dealing with a table that has a size of 2.1 GB and contains over 11,000 rows, with two fields populated with extensive JSON data. Whenever Meltano processes this table, I encounter the following error message:

[error    ] Extraction failed              code=-9 message=            
ELT could not be completed: Extractor failed.

Curiously, it starts the loading of some records into my data warehouse but subsequently crashes with the above error. To provide context, this pipeline is orchestrated by Airflow and is built from an image featuring Meltano v:2.12.0. • Database: PostgresSQL • Target: Snowflake • Tap: tap-postgres-transferwise Has someone encountered something similar while loading large JSONs fields?

I am using tap-s3-csv plugin to load the data. for some reason, I got the following error when I tried to load 1.5G of CSV file. it was working fine to load 2.7G of CSV file before. does anyone know how to fix this issue?

2023-11-08T03:22:48.638466Z [info     ] time=2023-11-08 03:22:48 name=tap_s3_csv level=CRITICAL message=("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')) cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.638900Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.639177Z [info     ]   File "/opt/dagster/dagster_home/meltano/.meltano/extractors/tap-s3-csv/venv/lib/python3.9/site-packages/urllib3/response.py", line 444, in _error_catcher cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.639491Z [info     ]     yield                      cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.639744Z [info     ]   File "/opt/dagster/dagster_home/meltano/.meltano/extractors/tap-s3-csv/venv/lib/python3.9/site-packages/urllib3/response.py", line 567, in read cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.639976Z [info     ]     data = self._fp_read(amt) if not fp_closed else b"" cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.640201Z [info     ]   File "/opt/dagster/dagster_home/meltano/.meltano/extractors/tap-s3-csv/venv/lib/python3.9/site-packages/urllib3/response.py", line 533, in _fp_read cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.640433Z [info     ]     return self._fp.read(amt) if amt is not None else self._fp.read() cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.640650Z [info     ]   File "/usr/local/lib/python3.9/http/client.py", line 463, in read cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.640875Z [info     ]     n = self.readinto(b)       cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.641086Z [info     ]   File "/usr/local/lib/python3.9/http/client.py", line 507, in readinto cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.641397Z [info     ]     n = self.fp.readinto(b)    cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.641677Z [info     ]   File "/usr/local/lib/python3.9/socket.py", line 704, in readinto cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.641922Z [info     ]     return self._sock.recv_into(b) cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.642150Z [info     ]   File "/usr/local/lib/python3.9/ssl.py", line 1275, in recv_into cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.642369Z [info     ]     return self.read(nbytes, buffer) cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.642587Z [info     ]   File "/usr/local/lib/python3.9/ssl.py", line 1133, in read cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.642805Z [info     ]     return self._sslobj.read(len, buffer) cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.643025Z [info     ] ConnectionResetError: [Errno 104] Connection reset by peer cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.643242Z [info     ]                                cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.643466Z [info     ] During handling of the above exception, another exception occurred: cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv
2023-11-08T03:22:48.643682Z [info     ]                                cmd_type=elb consumer=False name=tap-s3-csv producer=True stdio=stderr string_id=tap-s3-csv

Good evening/morning, I'm sending some data from tap-mongodb to target-snowflake. This seems to be accepting a smaller set OK, but when I send the whole set I have (3k records) I get the following error (attached). There's no indication what might be the issue. My config looks something like that:

environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-mongodb
        config:
          strategy: envelope
          mongo:
            host: <mongodb://localhost:27017/>
        select:
        - t_ent_393_entity_safety-report.*
      loaders:
      - name: target-snowflake
        config:
          account: XXX.eu-west-2.aws
          database: DEV
          schema: NET
          user: MELTANO
          warehouse: ELT
          role: DATALOADER
          default_target_schema: NET
          file_format: DEV.NET.MELTANO_CSV
          hard_delete: false
      - name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mongodb
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/tap-mongodb.git>
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  - name: target-snowflake
    variant: meltanolabs
    pip_url: meltanolabs-target-snowflake

Can I set a plugins pip_url with a

meltano config <plugin> set

command? Or some other way?

I'm running a pipeline with

tap-singer-jsonl -> target-redshift

and although I have the S3 State Backend configured it doesn't seem to be used. Every run of the pipeline results in all of the files in the S3 bucket being read. Anyone else encountered this?

I want to replicate our backend database to Snowflake. We have several country-specific schemas (

us

,

gb

,

fr

) and other miscellaneous schemas. I want to replicate all country schemas to Snowflake as-is (

us

->

us

,

gb

->

gb

) but all other schemas should go to

misc

. How can I achieve this?

Welcome!

agreeable-fall-83767

joined #troubleshooting. Also,

ripe-window-43112

,

adorable-photographer-18422

and

rapid-answer-47056

joined.