hallo all o/ Just deployed airbyte onsite and it’s working great - except i can’t seem to get the google analytics connector to connect properly, i’m not sure what i’m doing wrong

Hi! I have added a custom connector (source) to my local installation and I think I have added it twice. At one time it wasn’t there anymore so I hade to add it again. Now when I start up the docker containers and point my browser to localhost:8000 I get in to an infinite loop in the browser. This is the server log:

irbyte-server      | 2021-10-06 18:49:50 INFO i.a.s.e.KnownExceptionMapper(toResponse):21 - {workspace_app_root=/tmp/workspace/server/logs} - Known exception
airbyte-server      | 2021-10-06 18:49:50 ERROR i.a.s.RequestLogger(filter):87 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.19.0.3 POST 404 /api/v1/web_backend/connections/list - {"workspaceId":"a8d6c28e-ab6d-443d-ba5d-4f7568474ecf"}

And this is what I managed to grab from the browser:

Is this your first time deploying Airbyte: yes OS Version / Instance:  MacOS 11.6 Deployment: docker Airbyte Version: 0.30.13-alpha Source name: postgres 9.5 destination: Snowflake Description: hi all, just got airbyte up and running locally (macbook). postgres rds > snowflake (through s3). is there a way to keep the s3 objects that airbyte creates? i'm hoping i can keep it for records. i could also just build another 'connection' and dump data into s3 but since airbyte is already doing that, thought it would be nice to keep it there. i'm not sure how it's using the buckets

Hi everybody, I have a quick question for you. I am evaluating Airbyte and I need to know/understand if the tool supports sourcing data from REST or SOAP APIs? can anybody help me please?

Hey team! I’ve just started having a go at messing around with Airbyte. I’m wondering whether there’s currently any support for configuration of sources/destinations/connectors as code? Something along the lines of having a

.yaml

file to define the things and then having some capacity to run these definitions against Airbyte? Any thoughts are much appreciated ❤️

If I'm using the Singer Intercom connector and want to migrate to the native Airbyte connector, how do I do so without having to re-importing all the data from the beginning?

Hi all, Airbyte looks like a really promising product but I’m having a huge headache. I have a craftcms nitro mysql database and a craftcms nitro postgres database. I want to copy the mysql database to the postgres one. I’ve installed airtable, ran

docker-compose up

and visited localhost:8000. I’ve entered the MySQL credentials but I can’t connect to it. I was wondering if anyone else has managed to get this working? I see in the docs that using

host.docker.internal

as the host is recommended but this doesn’t seem to work either. Any help would be MASSIVELY appreciated 🙏

Question: If you were about to start designing a new service now that should have API and UI and you would like the approach "API and UI are equal, because the UI uses the API", • How far can one get with this approach in practice? • Would you consider something else than OpenAPI? • Why do you think for example GraphQL is not a fit for Airbyte? Is it because GraphQL is data-oriented and Airbyte needed something more RPC? • Would you start the UI by bootstrapping from https://github.com/facebook/create-react-app like Airbyte did? What alternatives would you consider? • What reading would you recommend about APIs? I found https://cloud.google.com/blog/products/api-management - but there are too many articles to grasp its usefulness right away. Thank you.

Hi there, tagging #C01MFR03D5W as well (in case resolution there) I'm having an issue when running this command in my terminal

gcloud --project=$PROJECT_ID beta compute ssh airbyte -- -L 8000:localhost:8000 -N -f

I receive this error

ERROR: (gcloud.beta.compute.ssh) unrecognized arguments:
  -L
  8000:localhost:8000
  -N
  -f
  To search the help text of gcloud commands, run:
  gcloud help -- SEARCH_TERMS

I am able to ssh thorugh using

gcloud --project=$PROJECT_ID --zone=$ZONE beta compute ssh $INSTANCE_NAME

Hi!👋 I have questions regarding the API: • Are you able to specify DBT transformations via the API? (Commands, git repo etc). Couldn't see it in the docs. • If not, Are we able to get the logs for the sources to then trigger a run of the DBT transformations upon syncing complete • Lastly, What happens in the case that multiple clients connect Google Ads at the same time on our instance? Will it execute all clients in parallel or buffer them? Will Googles API limits come into play?

Hello, How is it possible to change timezone for Airbyte? should I compose down change the system's timezone and then compose-up? thanks

hi, is there a way to configure airbyte using yml/json instead of thru the UI?

Hi folks, while setting up a source where do i declare the source schema name? so that Airbyte can read/replicate table from non-public schema? All my tables are stored in a different schema from Public

Hi! I have a question about the BigQuery-connector. When checking the BQ-logs I saw that jsons where temporary uploaded to a Google storage bucket that we do not own. Is it possible to set a folder that's within our ownership instead? Since we are in eu we need to make sure our pii data isn't in places that we do not control.

Hi all, I have a question regarding Postgres to S3 connection as JSONL. I'm facing an issue where the postgres timestamp fields are not coming to S3 with the same precision. It's taking only upto seconds and ignoring the microseconds that's in the system. Is there any way to fix this?

hey, I am using a airbyte connector from postgres to postgres, it is creating an additional copy of the tables that I am replicating every time I sync and the db is running out of space, is there a way I can stop airbyte from making multiple copies of the same table?

Hi! How can I add custom python script to transform data before loading to destination?

A general question about good practices in airbyte, I am replicating multiple tables from a psql db to another, however one of the tables is extremely large and takes around 12 hours to sync after a reset, would it be a good practice to make different connectors for different tables if it works ?

hey all, on the Airbyte website is mentioned the REST:API source, but when deployed locally there doesnt look to be one. Is this only available in the Cloud version? 90% of the sources I work with are REST and before I go and create new connectors for this I wanted to test it out. Am I missing something to get this added?

Can someone share an example CURL on how to trigger a sync via API call?

Hi all.. What’s the best way to trigger running a dbt cloud project once airbyte finishes the extract+load? Do we need airflow or similar orchestrator tool or can this be done without one?

How do we set the schema that a table lands in when connecting an MSSQL Server source to a Snowflake or Redshift destination? I have normalization turned on and it is currently ignoring the schema that is set in the destination for each of the above listed Destinations and instead mirroring the schema assignments from the Source. i.e. if there is database.analytics.communication in the source it will land in <Destination database>.analytics.communication so it keeps the database, but ignores the schema attribute

Hey all, So my organization really likes Airbyte and we are looking to build a CICD pipeline to deploy our fork of Airbyte onto an EC2 instance anytime we contribute to our fork. However, we also have some custom connectors we would love to contribute back to the broader Airbyte repo. Are there recommendations on a git workflow pattern to support this? Because right now, if we were to create a PR back to the main Airbyte repo we would inadvertently be contributing back the custom connector code plus all of our CICD process code of Terraform and GitHub Actions.

hey all, wanted to know how mongo connector works. Is it pull based or CDC based(using oplog or changestream). Also does Airbyte has operational dashboard for launching new connectors. Also what about monitoring. Does it provides end to end monitoring and alerting dashboards.

Hi all, wanted to know if there is a way to filter out duplicate records (row) in AirByte?

reposting this here since it might have gotten lost in the noise in the other channel: Basically, the issue is around importing and getting NaN vs “” or NULL https://airbytehq.slack.com/archives/C01MFR03D5W/p1634053236402900

Hey 🙂 does the Mailchimp connector also support transactional mail reporting (Mandrill Add-on)?

Hello! Quick question, how I can connect to Ms SQL with one IP with multiple instances/workspaces of databases?

Hello team!! I’d like to know how to delete

_airbyte_ab_id

and

_airbyte_emitted_at

columns by default when to move a csv file from local File(CSV) to S3(MinIO). And tell me why these columns exist.

Hi all, I'm transfering data from mySql to Postgres with the Sync mode INCREMENTAL / Decuped + History. I used this mode because I just want to have unique data in my destination-table. In this sync mode airbyte creates a XXX_scd and _airbyte_raw_XXX table for each table which are getting bigger and bigger now. Is it possible to clean the tables (truncate?) them?