Hello, I tried to develop my first Connector (Python) and I cannot see it in the interface. I read this : https://docs.airbyte.com/connector-development/tutorials/building-a-python-source#step-11-add-the-connector-to-the-apiui 1- build docker image docker build . -t airbyte/source-custom:dev 2- Add this to source_definitions.yaml - name: Custom sourceDefinitionId: 1cdad10e-1cc3-11ed-861d-0242ac120002 dockerRepository: airbyte/source-custom dockerImageTag: dev documentationUrl: https://docs.com icon: airtable.svg sourceType: api releaseStage: alpha 3 - With or without remvongin the volume created by docker-compose up. Not any success. Any ideas ?

This message was deleted.

Hello all, do we have any Airbyte connector to get Facebook Leads?

Hello all, Could anybody tell me if the token is refreshed during exponential back off or how to gracefully handle expired tokens / 401 HTTP errors?

I'm running Airbyte 0.32.10-alpha on a GCP server. The newest Salesforce source cannot be set up. "Fetching connector failed." Is the error message. I also am unable to edit my existing Salesforce sources, the Settings tab just reverts to Overview. My plan is to upgrade the Airbyte image to the latest 0.40.0-alpha. Has anyone encountered this before, is there something else I should look into? Thanks!

Is airbyte cloud available in Europe?

Hello, Aribyte noob here. is it possible to make a duplicate of an existing source, destination or the connection? This would save a lot of dev time :P

@Airbyte team.. We are facing schema issues in Oracle Connector for APPS schema. Its running for more than 1 hour and failed not able to show the tables. already submitted the ticket.. Can we get paid consultant support?

Hey there Airbyte team, thanks for making this awesome tool. I'm using the MySQL source and Postgres destination connectors that work great for

Full refresh | Overwrite

. However, the moment I try

Incremental | Deduped + history

I get an error. What am I missing?

Hi Airbyte team , I want to add 3PL central connector but not able to find 1. Docker repository name 2. Docker image tag 3. Connector Documentation URL above fields on https://hub.docker.com/u/airbyte Is docker didn't provided it's service for 3PL central Can you please help me on this?

👋 Hello, team!

Hey Team! I am new to Airbyte . I came across as error while running my airbyte source i.e. extracting data from my custom source and syncing it to PostgreSQL destination.My connection is getting cancelled without my input.Logs are:-

2022-08-15 10:33:37 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):325 - Records read: 225000 (203 MB) 2022-08-15 10:42:39 destination > 2022-08-15 10:42:39 INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(lambda$flushAll$1):86 - Flushing <stream_name>: 7435 records (24 MB)

2022-08-15 10:52:16 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$3):191 - Running sync worker cancellation...

2022-08-15 10:52:16 INFO i.a.w.g.DefaultReplicationWorker(cancel):444 - Cancelling replication worker...

2022-08-15 10:52:26 INFO i.a.w.g.DefaultReplicationWorker(cancel):452 - Cancelling destination...

2022-08-15 10:52:26 INFO i.a.w.i.DefaultAirbyteDestination(cancel):125 - Attempting to cancel destination process...

2022-08-15 10:52:26 INFO i.a.w.i.DefaultAirbyteDestination(cancel):130 - Destination process exists, cancelling...

2022-08-15 10:52:26 INFO i.a.w.g.DefaultReplicationWorker(run):175 - One of source or destination thread complete. Waiting on the other.

2022-08-15 10:52:26 WARN i.a.c.i.LineGobbler(voidCall):86 - airbyte-destination gobbler IOException: Stream closed. Typically happens when cancelling a job.

2022-08-15 10:52:26 INFO i.a.w.i.DefaultAirbyteDestination(cancel):132 - Cancelled destination process!

2022-08-15 10:52:26 INFO i.a.w.g.DefaultReplicationWorker(cancel):459 - Cancelling source...

2022-08-15 10:52:26 INFO i.a.w.i.DefaultAirbyteSource(cancel):142 - Attempting to cancel source process...

2022-08-15 10:52:26 INFO i.a.w.i.DefaultAirbyteSource(cancel):147 - Source process exists, cancelling...

2022-08-15 10:52:26 INFO i.a.w.g.DefaultReplicationWorker(run):177 - Source and destination threads complete.

2022-08-15 10:52:26 INFO i.a.w.i.DefaultAirbyteSource(cancel):149 - Cancelled source process!

2022-08-15 10:52:26 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$3):195 - Interrupting worker thread...

2022-08-15 10:52:26 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$3):198 - Cancelling completable future...

2022-08-15 10:52:26 WARN i.a.w.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):53 - Job either timed out or was cancelled.

2022-08-15 10:52:26 WARN i.a.w.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):53 - Job either timed out or was cancelled.

What could be the issue here? I am assuming it is because my data is more than 200MB.If this is how could I resolve it?

In Airbyte workspace it is showing the Negative Credits. Can anyone help me with this?

Hello everyone, I'm looking to modify the s3-source connector to output multiple streams for each selected file. Is that even possible? If yes, where should I start?

Hello, I'm getting this error when trying to replicate Cockroachdb to BigQuery

FailureReason@4d70ea94[failureOrigin=<null>,failureType=system_error,internalMessage=java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: unimplemented: multiple active portals not supported
airbyte-server   | Detail: cannot perform operation sql.PrepareStmt while a different portal is open
airbyte-server   | Hint: You have attempted to use a feature that is not yet implemented.
airbyte-server   | See: <https://go.crdb.dev/issue-v/40195/v22.1,externalMessage=Something> went wrong in the connector. See the logs for more details.,metadata=io.airbyte.config.Metadata@7d4ee3a3[additionalProperties={attemptNumber=null, jobId=null, from_trace_message=true}],stacktrace=java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: unimplemented: multiple active portals not supported
airbyte-server   | Detail: cannot perform operation sql.PrepareStmt while a different portal is open
airbyte-server   | Hint: You have attempted to use a feature that is not yet implemented.
airbyte-server   | See: <https://go.crdb.dev/issue-v/40195/v22.1>
airbyte-server   | at io.airbyte.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:100)
airbyte-server   | at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:332)
airbyte-server   | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
airbyte-server   | at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
airbyte-server   | at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
airbyte-server   | at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
airbyte-server   | at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
airbyte-server   | at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
airbyte-server   | at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
airbyte-server   | at io.airbyte.db.jdbc.JdbcDatabase.queryJsons(JdbcDatabase.java:170)
airbyte-server   | at io.airbyte.integrations.source.postgres.PostgresSource.getPrivilegesTableForCurrentUser(PostgresSource.java:383)
airbyte-server   | at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.discoverInternal(AbstractJdbcSource.java:124)
airbyte-server   | at io.airbyte.integrations.source.postgres.PostgresSource.discoverRawTables(PostgresSource.java:201)
airbyte-server   | at io.airbyte.integrations.source.postgres.PostgresSource.discoverInternal(PostgresSource.java:183)
airbyte-server   | at io.airbyte.integrations.source.postgres.PostgresSource.discoverInternal(PostgresSource.java:73)
airbyte-server   | at io.airbyte.integrations.source.relationaldb.AbstractDbSource.discoverWithoutSystemTables(AbstractDbSource.java:147)
airbyte-server   | at io.airbyte.integrations.source.relationaldb.AbstractDbSource.getTables(AbstractDbSource.java:337)
airbyte-server   | at io.airbyte.integrations.source.relationaldb.AbstractDbSource.discover(AbstractDbSource.java:97)
airbyte-server   | at io.airbyte.integrations.source.postgres.PostgresSource.discover(PostgresSource.java:166)
airbyte-server   | at io.airbyte.integrations.base.ssh.SshTunnel.sshWrap(SshTunnel.java:205)
airbyte-server   | at io.airbyte.integrations.base.ssh.SshWrappedSource.discover(SshWrappedSource.java:45)
airbyte-server   | at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:127)
airbyte-server   | at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:97)
airbyte-server   | at io.airbyte.integrations.base.adaptive.AdaptiveSourceRunner$Runner.run(AdaptiveSourceRunner.java:86)
airbyte-server   | at io.airbyte.integrations.source.postgres.PostgresSourceRunner.main(PostgresSourceRunner.java:15)
airbyte-server   | Caused by: org.postgresql.util.PSQLException: ERROR: unimplemented: multiple active portals not supported
airbyte-server   | Detail: cannot perform operation sql.PrepareStmt while a different portal is open
airbyte-server   | Hint: You have attempted to use a feature that is not yet implemented.
airbyte-server   | See: <https://go.crdb.dev/issue-v/40195/v22.1>

Hey Team! I am deleting a connection, and it disappears from the UI, but it reappears after a few seconds as if I had never deleted it. Is this expected?

Hello! I have one question: Can I config Airbyte to run it's worker or tasks on different AWS server? I use command

docker-compose up

to start a Airbyte Server on AWS server And then I use Airbyte Web to create and start a connection, A docker container started to execute some tasks. Now I want to know, Can I config Airbyte to run it's worker(or connection tasks) on specific AWS server ? Because some API be called just on specific public IP address.

Does Airbyte has the feature that run task on specific Server?

Hello Airbyte! We are running Airbyte OSS with a mongo database as source and postgresql as destination. The collections from our mongo database are being synced daily with "Full Refresh | Append" strategy. We notice when adding new collections to the mongo database, that these are not discovered by Airbyte, also not when invoking "Refresh source schema" in the connection replication settings. How can we make sure that Airbyte sees the new collections and start syncing them without impact on the existing collections synchronisations. Many thanks!

Hi everyone. I've successfully set up a connection using the YouTube (alpha) connector but require the Content Owner Reports. Does the team have a rough ETA on when this feature will be released? Thanks.

Hi everyone. We are testing airbyte changes on snowflake key-pair authentication. Unfortunately by generating the keys according to the docs we have an issue :

Could not connect with provided configuration. Private key provided is invalid or not supported: rsa_key.p8: Cannot invoke "<http://net.snowflake.client.jdbc.internal.org.bouncycastle.util.io|net.snowflake.client.jdbc.internal.org.bouncycastle.util.io>.pem.PemObject.getContent()" because the return value of "<http://net.snowflake.client.jdbc.internal.org.bouncycastle.util.io|net.snowflake.client.jdbc.internal.org.bouncycastle.util.io>.pem.PemReader.readPemObject()" is null

When connecting to snowflake using snowsql the key/pair auth is working with the same keys. What are we missing?

Hi .. I hope everyone doing good. If anyone available for 2 to 3 hour session(as a paid consultant support) to help us to fix some of the Airbyte challenges we have. it would be much appreciated. Thanks for your support

Hello! octavia wave Just as a reminder about this channel is designed to function. This channel is for basic questions related to Airbyte’s functionality. Some examples are: Does the Salesforce connector support the user region field in the User endpoint? or Does Airbyte deploy in an AWS Container service? If you are facing a deployment or connection issue, or are encountering errors when trying to deploy or use Airbyte, we ask you to please use our Discussion Forum (https://discuss.airbyte.io/). In our Forum you can often find answers to common issues, talk with other teams having the same issue, or communicate directly with the Community Assistance Team by opening a new Forum Topic. This also allows other community members who might encounter your issue later on to search for and find the answer. In addition, we’ve created an integration between our Forum and the Github Issues for Airbyte, so if the problem encountered needs to be escalated you will be apprised of the progress on that issue automatically. We thank you for your time and attention, and we hope you will continue to participate in the community and keep using Airbyte!

Hi! I just set up the connection between App Store and Big query. The sync was successful but no data was transferred. Any idea what is going wrong?

Hi, the channel #C030LRCU7NV has been archived so I am unsure where to put this. I have setup Airbyte OSS on a VM in Google cloud a bit over a week ago. I made a few connection and so far it was working mostly fine. Yesterday I had to reset my VM, since my connections got stuck in sync and could not be cancelled. Afterwards it worked normally again. Now today I see the message

Cannot reach the server. The server may still be starting up.

for over 1 hour now. I looked at the recommendations in https://docs.airbyte.com/troubleshooting/on-deploying/ and I do get error messages in both docker logs

airbyte-server

and

airbyte-worker

. Since I am fairly new to airbyte I am not sure what those messages mean. Can anyone help me with this? Thanks

Hello Team, Upon setting up self-hosted Airbyte locally our infra team wrote this comment to us, During i want to install airbyte i found out that airbyte only have versions which are marked as

alpha

. From this point i can absolutely not recommend to use an alpha verions in production environments. Was it carefully evaluated, tested, decided to really use it for a production environment? Here you can find what is an alpha versions and what are the risks: Software release life cycle Could someone helps us with the above? Is that still in alpha phase or we shall try with “stable” release? We have passed setup documentation page as : https://docs.airbyte.com/quickstart/deploy-airbyte/?_ga=2.89522395.1160840054.1659428690-1169156739.1659428688

Hi team, when you first deploy a connector for incremental sync can you set the state for the first run? or does it have to run once first without a state in order to save a state to use for next time?

We want to build a custom Airbyte source connector which needs pull data from Apple Search Ads and we want to reuse a python library which is already there to pull this data. So we would like to wrap the library calls in Airbyte source connector. We generated a python HTTP API source connector using generator script and noticed the interface is based on requests library, but we want to use our existing custom module for this. We don't want to rely on these methods which uses requests module. Since out custom python module already does this

def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
        return None
 
    def request_params(
        self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, any] = None, next_page_token: Mapping[str, Any] = None
    ) -> MutableMapping[str, Any]:
        return {}
 
    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
        yield {}

What type of source connector will be the right choice to use here? Should we use Python HTTP connector, Python Source connector or Python Generic Source connector ?

HI I just want to ask if there is a way in airbyte whereby we only replicate the no. of records we want. Let's say I've 100000 records in my source but I want only 10k records in my destination. So just curious if there's any workaround.

Hello everyone, I see in your docs you can deploy on Kubernetes (which is Beta) or on Plural (which aids in deployment to Kubernetes) I was wondering what are some pros and cons of using Kubernetes on your own rather than Plural? Is it just convenience or are there other pros/cons Im missing? Thanks.