I implemented a solution for issue https://github.com/meltano/meltano/issues/8256. I executed tests locally:

poetry run pytest

It got stuck forever at 99%. Advice? My local setup is: Latest Ubuntu, Python 3.10.

does meltano have a guest author program?

When forking a branch, what's the best way to test in a Meltano pipeline before raising a PR?

Hey,

meltano compile

creates manifest.json. The problem is that it does not expand attributes in streams. Example: using tap-github, I declare

workflows.*

. When I run the compile, manifest.json contains

workflows.*

, the wildcard is not expanded. Current workaround solution: Run

meltano select --list --all

and extract selected (and expanded) attributes for the workflows stream. It is not an ideal workaround because

meltano select

command much slower than

meltano compile

. Questions: • Would it feasible to extend the

meltano compile

command accordingly? • If yes, I can contribute, if anyone is so kind to guide me 😉 cc: @Justin Cohen

Could anyone review/merge my PR: https://github.com/transferwise/pipelinewise-tap-s3-csv/pull/249

Not sure if I shared the message in the right channel, so duplicating here for visibility - https://meltano.slack.com/archives/C069RGRJYD7/p1706593447732259

Hi, follow up on ⬆️ and updating hub - https://github.com/meltano/hub/pull/1659. Not sure what to put into maintainers. Is it the one who maintain https://github.com/dbt-athena/dbt-athena? Or then it should be me, who created this utility plugin?

Hi there, I need to optimise for storage on my target-postgres. Current implementation is optimized to not loose any data, which is nice but isn't convenient for my use case. Example: Some of my text fields would benefit from a pg domain/type and some other text data would be more efficiently stored as bytea. Even better if I could manipulate the tablespace of the schema/table or set a table to be an hydra columnar table, a timescaledb table, or set pg_partman config somehow. I see multiple proposals in GitHub but idk if there is a consensus on a target implementation to force a column type to be something else or to hook into table dll. If there's no consensus I'll just fork target-postgres and do my own thing but just wanted to ask in case that could be contributed back somehow

Hey all, I'd like to update the tap-salesforce API version (and possibly make it configurable). I'm not sure where to turn or how to do it, any advice?

Hey team, I have a PR open to update the tap-salesforce API version.

Big tap-mongodb PR https://github.com/MeltanoLabs/tap-mongodb/pull/31 The log-based replication in AWS DocumentDB has worked super well, but the log-based replication for MongoDB has been buggy. This PR fixes that, and fixes three open issues 💪

Hello folks! New to contributing to Meltano, quite excited! We are currently using several taps and targets from Transferwise and we would like to migrate to the ones that use the Meltano SDK, which we are more familiar with and we feel more comfortable to contribute to. We are starting the migration with Target Postgres. 1. As somebody pointed out, it seems that this target was heavily customized (overrode several methods of the Meltano SDK), and I am not sure to what extent this is still necessary. And we would need this (

load_method

=

overwrite

, which we would get out of the box with the current version of the SDK). I posted a comment here and I am willing to take over the refactor. 2. The column caching seems to throw wrong results on schema changes. I also posted my opinion here Looking forward to contributing!

Anyone knows why there is no streamlit utility to add? I am thinking of trying to add it my self 😄

hi folks, I have a small PR that adds a retry logic for an intermittent salesforce failure that we've encountered. I was not sure what the best practice is to bring attention to it - apologies in advance if this is not the right place to highlight it. issue describing the problem: HTTPError 406 CustomNotAcceptable encountered intermittently · Issue #58 · MeltanoLabs/tap-salesforce · GitHub PR to address the issue: fix: Handle 406 http errors by haleemur · Pull Request #57 · MeltanoLabs/tap-salesforce · GitHub

👋 Hello folks! First time posting in Meltano slack so please redirect if another channel is more appropriate! I'm looking for reviews on a PR to fix a bug in the Survey Monkey Tap. I've had the PR open for two weeks now but have been unable to get any reviews from folks with Write Access. Would highly appreciate guidance on how to best get this PR reviewed as it's blocking my organization's usage of the tap. Thanks in advance! 🙏

Hello, I am not sure if this is the right place but I have a PR up for the Meltano EDK repository that is meant to try to handle the issue I ran into a few weeks ago: https://meltano.slack.com/archives/C069CQNHDNF/p1716463234362109. It's not exactly what we discussed in the thread as a solution but my knowledge around Python asyncio/event loops/subprocesses is quite limited so this is the best alternative I could come up with right now. PR: https://github.com/meltano/edk/pull/274 Resolves issue: https://github.com/meltano/edk/issues/264

Hi, I would like to add custom properties stream into tap-github. Can someone advice the better way to do that?

https://github.com/meltano/hub/pull/1808/files Hi folks, I just opened a PR for an Apache Iceberg tap I've started to work on, currently passing the generic extractor test suite so I think ready to be added into the catalog.

I found a bug in the latest version of

tap-airbyte-wrapper

which breaks Docker functionality. I created a PR to fix it: https://github.com/MeltanoLabs/tap-airbyte-wrapper/pull/20

Anyway we could get a release, we want https://github.com/meltano/meltano/issues/6763 😄

👋 I'm working towards a PR to change tap-github's authenticator to enable refreshing for github app tokens, and as far as I can tell there are no tests for anything in the authenticator file. Is that fairly standard practice, or should I be trying to add tests for the changes I'm making (and if so, is there an example of authenticator tests in another tap that would be good to study up on)?

Hello folks, I'm happy to announce that we've added allcontributors to the Meltano repo: https://github.com/meltano/meltano/blob/main/CONTRIBUTORS.md I'm planning to add all folks that contribute to Meltano, the SDK, any MeltanoLabs repos, and MeltanoHub. Do comment here or in the issue if you notice you or someone else is missing.

I created a small Pull Request to improve column quoting in the meltano SDK. https://github.com/meltano/sdk/pull/2582 I appreciate your reviews on it. In general object quoting feels a bit brittle, because in some methods custom queries are constructed using parts generated by sqlalchemy, while other methods, sqlalchemy is used entirely. I'll try to make some improvements in that regard as well. Slack Conversation

Hey team, the Pulumi Data team is contributing a more complete tap for the Pulumi Cloud API, feel free to check it out here: https://github.com/MeltanoLabs/tap-pulumi-cloud/pull/184

Hi folks, I have some availability for the rest of the month to contribute code outside of work. I'm wondering if there is a prioritized backlog of issues I can pick from that would help the meltano community out. Maybe some of the more involved devs know of some important issues that you'd want help on?

Should

invoke

have

--refresh-catalog

(.venv) visch@visch-work-2:~/git/ironman_meltano$ meltano invoke --help
Usage: meltano invoke [OPTIONS] PLUGIN_NAME[:COMMAND_NAME] [PLUGIN_ARGS]...

  Invoke a plugin's executable with specified arguments.

  Read more at <https://docs.meltano.com/reference/command-line-interface#invoke>

Options:
  --print-var TEXT                Print to stdout the values for the provided
                                  environment variables, as passed to the
                                  plugininvoker context. Useful for debugging.
  --plugin-type [extractor|extractors|loader|loaders|transform|transforms|orchestrator|orchestrators|transformer|transformers|file|files|utility|utilities|mapper|mappers|mapping|mappings]
  --dump [catalog|config]         Dump content of specified file to disk.
  --list-commands                 List the commands supported by the plugin.
  --containers                    Execute plugins using containers where
                                  possible.
  --install                       Install the subject plugin(s) automatically.
  --no-install                    Do not install the subject plugin(s)
                                  automatically.
  --only-install                  Only install the subject plugin(s).
  --database-uri TEXT             System database URI.
  --help                          Show this message and exit.

Trying to give steps for someone to dump the catalog they have without telling them to delete

.meltano/

or a clean install

Is

v3.6.0

due for release soon? I'd like to upgrade our platform to leverage https://github.com/meltano/meltano/pull/8815.

Hi Everyone, I’ve forked the default tap-mssql and added a new feature: CDC on Views. You can find the details and implementation in the repository below: GitHub Repo: pipelinewise-tap-mssql - CDC on Views Why This Feature Was Created: In our organization, we stream data to BigQuery using views that include joins and then some calculated columns based on the secondary table. This streaming is currently handled by a custom .NET application using change tracking, which has a delay of 15-20 minutes. The process involves generating CSV files and performing merges, adding to the latency. While working on a proof of concept (POC), I found Meltano to be much faster. By running sync operations immediately after the previous run completes, we can achieve near real-time data updates. The Challenge: CDC (Change Data Capture) in MSSQL does not natively work with views, which created a roadblock for our use case. The Solution: To overcome this, I forked the tap-mssql repository and implemented custom logic to enable CDC on views. The approach involves: • Extracting CDC data from the primary table associated with the view. • Joining this CDC data with the view to fetch only the latest changes. • Delivering data directly from the view while ensuring that calculated columns and transformations are preserved. This enhancement allows me to leverage CDC from the Primary Table and Actual Data from views and achieve near real-time data streaming without the delay associated with the previous solution. Feedback and suggestions are welcome! Do let me know if any thing can be done better.

Have we ever thought about notifying about backpressure issues like the target getting behind? Here's what I did a while back when I was messing with all of this (snippet of some of my own tap->target magic)

async def _outstream_handler_tap(self, handle, name, procname, queue):
        data = "initial"
        i = 0
        queue_max_size = queue.maxsize

        while data:
            data = await handle.readline()
            line = data.decode()
            if data: 
                await queue.put(line)
            i = i + 1
            if (i >= (queue_max_size*.1)):
                i = 0
                if (queue.qsize() >= queue_max_size*.1):
                    <http://logging.info|logging.info>(f"Queue Size is >10% currently at: {queue.qsize()}. Max Queue size is: {queue_max_size}. Target is behind")

I'm having some back pressure issues and haven't dove in, but it'd be nice to know if the target wasn't keeping up with the tap

Hi, I have a tap developed for Google My Business, which can return reviews and search metrics for your locations on Google Maps. I am planning to open source to MeltanoLabs - if anyone is interested in participating in some testing, particularly around the authorization and inviting, I would appreciate it. A google service is required. Slack Conversation