Nice demo, even though I only caught the last part 😳 How many levels does the basic normalization do? e.g. in Snowflake if I have a VARIANT col that then has nested attributes that also may have nested attributes etc. At some point these need to go into separate tables, so curious what you guys chose to do here

Sorry if this isn't the right place for these reports, but getting another Docs links error. Feels like "Feedback" more than "Troubleshooting". But anyway, 404 error in Docs: https://docs.airbyte.io/tutorials/upgrading-airbyte

Hi Guys, I have updated the Airbyte's version and get this error - Could not find configuration for STANDARD_WORKSPACE: 6cd99b4e-7407-4831-999f-c22c6a79344d.

Hi everyone nice to meet you. I came here wondering if anyone has tried to build a connector with blockchain data, and if it’s a use case for airbyte

Hi Airbyte team. Have enjoyed testing with your product over this past week, starting with local deployment and then migrating it to GCP. I’ve been testing by connecting to an on-prem MSSQL server database hosting a transactional system. A couple notes of initial feedback / future requests: • After setting up a source connector to sync, in order to add new tables to the integration you have to reset the whole thing and reselect. This is quite a pain when the source database contains hundreds of tables. Would be nice to be able to just select the additional tables without impacting the current table configuration. • Be able to select columns and limit the data integrated in a source table (For instance there may be a table that has 200 columns, but I only want 20 of them loaded into the datalake) ◦ I will be testing using the DBT transformation to accomplish this, but would be nice if I could select/deselect in the UI

Hi! I just caught up the community call - great to hear that SSL for DBs is coming soon. I was hoping to hear about the ability to select fields from a source, but nothing on that front. It seems to be a pretty highly requested feature (even just above this message simple smile) and it's quite critical for my use-case, so I was wondering whether you have any kind of timeline for it? Cheers.

Feature request for the work being done on outreach: They hard delete items in their system (ie they have a workflow and they'll create all the tasks upfront, but if the workflow completes after task 1, all downstream tasks are hard deleted). They have a webhook available for you to respond to platform events (https://api.outreach.io/api/v2/docs#respond-to-platform-events), but the way I actually handle this is I don't use that and just maintain a list of all items that get returned in the get request hourly and then I add a soft delete flag for all items not in that list.

I am trying to connect using MySql source and getting error Could not connect with provided configuration. Error: Cannot create PoolableConnectionFactory (Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)

Hi! I am trying to build monitoring & alerting for Airbyte (on Kubernetes). does Airbyte expose (or plan to expose) job-related metrics? like sync duration, failures etc by source/destination/connection

Hi! I have been experimenting with Airbyte for the past few days to see if it can replace the python scripts I have written. I am very impressed with the tool, and I am all in on the open source community driven mission, but I think I need to wait a little while as there are some blockers to adoption in our case. 1. Changing table selections in the ui causes a full re sync of data. This is a pain, because on connectors like intercom/marketo, the initial sync can take 8+ hours. 2. Not being able to de select or hash specific columns is a concern for us because PII and other sensitive data in our analytics database means one more database with security / audit concerns. 3. Pre-defined schemas means that we will miss a lot of data as new columns are added to custom fields in apps like SFDC or Zendesk. It would be nice where possible to be able to opt out of column selection rather than opt in. 4. The salesforce connector doesn't support custom objects, which is a blocker. 5. The zoom connector is missing the phone/call endpoints 6. Connectors for SalesLoft / SaasOptics None of these issue on their own would stop me from adopting the tool, but all of them together severely limit the scope for which we can adopt it.

I see there's a Redshift destination connector update, but nothing in the connector catalog changelog, or the other connector changelog location - Any ideas?

I am curious, will Airbyte eventually move into the reverse ETL space as well. Right now tools like FiveTran exist for ETL and tools like Census exist for reverse ETL. Im hoping that Airbyte would evolve into a single tool for supporting both similar to what Meltano is stating. Obviously a big feature like reverse ETL is something that doesnt appear over night, but im interested in know if the team believes that is an important direction to head in time. Thoughts??

I have a piece of feedback on the new Github Native connector. It would be nice to include the rest of the user information returned by the GitHub API beyond just the user_id. On all streams, but particularly pull requests and commits. The API endpoint docs are here: https://docs.github.com/en/rest/reference/pulls And here is an example of the raw data returned by the connector:

{
  "url": "<https://api.github.com/repos/airbytehq/airbyte/pulls/4646>",
  "id": 686552355,
  "node_id": "MDExOlB1bGxSZXF1ZXN0Njg2NTUyMzU1",
  "html_url": "<https://github.com/airbytehq/airbyte/pull/4646>",
  "diff_url": "<https://github.com/airbytehq/airbyte/pull/4646.diff>",
  "patch_url": "<https://github.com/airbytehq/airbyte/pull/4646.patch>",
  "issue_url": "<https://api.github.com/repos/airbytehq/airbyte/issues/4646>",
  "number": 4646,
  "state": "closed",
  "locked": false,
  "title": "0.27.1 Connector Patch Notes",
  "body": "## Main Changes\r\n- Adds Connector changelog for the 0.27.1 patch\r\n\r\n## Misc Changes\r\n- Fixes naming for SurveyMonkey and CockroachDB (capitalization matters!)\r\n- Reorganized CockroachDB in the integrations list to be in alphabetical order... very important.",
  "created_at": "2021-07-09T07:14:39Z",
  "updated_at": "2021-07-09T07:15:29Z",
  "closed_at": "2021-07-09T07:15:28Z",
  "merged_at": "2021-07-09T07:15:28Z",
  "merge_commit_sha": "15971e89b1fb623a006a98489d8aa48cb2de2956",
  "draft": false,
  "commits_url": "<https://api.github.com/repos/airbytehq/airbyte/pulls/4646/commits>",
  "review_comments_url": "<https://api.github.com/repos/airbytehq/airbyte/pulls/4646/comments>",
  "review_comment_url": "<https://api.github.com/repos/airbytehq/airbyte/pulls/comments{/number}>",
  "comments_url": "<https://api.github.com/repos/airbytehq/airbyte/issues/4646/comments>",
  "statuses_url": "<https://api.github.com/repos/airbytehq/airbyte/statuses/0becef350a3da69215644fe38fb2bcd17a32d738>",
  "head": {
    "label": "airbytehq:abhi/indras-net",
    "ref": "abhi/indras-net",
    "sha": "0becef350a3da69215644fe38fb2bcd17a32d738",
    "user_id": 59758427,
    "repo_id": 283046497
  },
  "base": {
    "label": "airbytehq:master",
    "ref": "master",
    "sha": "db223a4d068b793d0cb054b7fc671b9dc108bfe0",
    "user_id": 59758427,
    "repo_id": 283046497
  },
  "_links": {
    "self": {
      "href": "<https://api.github.com/repos/airbytehq/airbyte/pulls/4646>"
    },
    "html": {
      "href": "<https://github.com/airbytehq/airbyte/pull/4646>"
    },
    "issue": {
      "href": "<https://api.github.com/repos/airbytehq/airbyte/issues/4646>"
    },
    "comments": {
      "href": "<https://api.github.com/repos/airbytehq/airbyte/issues/4646/comments>"
    },
    "review_comments": {
      "href": "<https://api.github.com/repos/airbytehq/airbyte/pulls/4646/comments>"
    },
    "review_comment": {
      "href": "<https://api.github.com/repos/airbytehq/airbyte/pulls/comments{/number}>"
    },
    "commits": {
      "href": "<https://api.github.com/repos/airbytehq/airbyte/pulls/4646/commits>"
    },
    "statuses": {
      "href": "<https://api.github.com/repos/airbytehq/airbyte/statuses/0becef350a3da69215644fe38fb2bcd17a32d738>"
    }
  },
  "author_association": "CONTRIBUTOR",
  "auto_merge": null,
  "active_lock_reason": null,
  "user_id": 33042053,
  "milestone": null,
  "assignee": null,
  "labels": [
    2235194062
  ],
  "assignees": [],
  "requested_reviewers": [],
  "requested_teams": [],
  "_ab_github_repository": "airbytehq/airbyte"
}

Moving thread to the proper channel

Hey guys. Looks like migrating from Ec2 to k8s is not straight forward there are some huddles which we faced 1. Need to have minimum compute engine we use t3.xlarge with 3 nodes anything less than this is not working for PROD. 2. For the very first time when we do

kubectl apply

server fails with error pvc

airbyte-volume-workspace

not found. This can be resolved by doing

kubectl delete all --all

and do

kubectl apply

again. 3. We can't migrate if there are more jobs and attempts. Suggestion from @Chris (deprecated profile) is to delete old data and it worked like champ. 4. We will have to upgrade both sources, destinations to version where there is

ENV AIRBYTE_ENTRYPOINT

as this is added for k8s. After this 🎉 things are smooth. Thanks team for k8s support.

I'm trying to connect to a MySQL 5.7.25 database at PureCloud Genesys. I can successfully connect with my general purpose database tool, but I get the following error when I try to use the Airbyte MySQL connector.

Could not connect with provided configuration. Error: Could not query the variable binlog_row_image

Any help would be appreciated as I have zero experience with MySQL.

hi . I'm using airbyte on k8s and its amazing,,but its create multiple pods (around 300 completed pods) in day!! how can i handle this issue.. i deleted completed pods manually?is it possible to prevent create this pods or deleted completed automatically?

🎉 - Testing now and resyncing history!

Hello Airbyte team! I started exploring Airbyte; my first connector is Postgres. Couple features i thought would be really useful as we sync the data using Airbyte. • Ability to select the required columns to sync to destination( example - scenarios where we have to exclude columns containing PII). • Ability to choose/overwrite the datatype for the destination. (I experienced a situation where Airbyte was reading integer column from source and creating float column at the destination. It's not a big deal unless that integer column is epoch timestamp, which is in my case) Is Airbyte planning to implement those features?

For instance, is there any connector or feature that we're missing and that you would need?

You have a way I can be notified on Slack if AirByte has a problem with a particular sync job. That's nice, but it's not enough to tell me whether my jobs are running. If AirByte just stops running, or the server goes down or loses connectivity, I won't get any notification. We use Datadog for monitoring/alerting. So, for my Singer jobs: • I use dogwrap to send an event to Datadog • I configure some threshold (e.g. 'less than 5 successful within 24 hours' so I can get an alert if there's any problem at all (even something like the server going down) If I could send an event to Datadog for any successful jobs (not just for failed jobs), that would allow me to set up alert rules the same way we do for the rest of our infrastructure.

Hi Airbyte team! Can you please point me to the best resource to understand timing of and follow progress toward release of your managed offering? Thanks! airbyte rocket

Hello, is there a way to disable the 3 retries when a connection fails? I have a connection that often fails because of the API rate limits and retrying is only making this worse.

destination-postgres: v3.7 doesn’s support basic_normalization through configuration API. When I upgraded to v 3.7 from v.3.6 both: • /v1/web_backend/destinations/recreate • /v1/destinations/create generate error for basic_normalization not defined in the schema.

{"message": "The provided configuration does not fulfill the specification. Errors: json schema validation failed. \nerrors: $.basic_normalization: is not defined in the schema and the schema does not allow additional properties \nschema:

Switching back to v3.6 solves the problem.

per debezium docs,

DATE

 , 

DATETIME

 types are converted into epoch milliseconds or microseconds based on the column’s precision. Is there already a request to convert these (now BIGINT) columns back to timestamp by default in the normalization step when using CDC?

Hey team, have a couple questions: • The docs indicate that the Google Ads connector does not support incremental syncs, but when running locally it looks like I can still replicate data under the “Incremental Sync - Deduped History” setting. Do you know if there’s something missing in the current implementation (or who I can connect with) to get more info on using that connector? • Do you know if there’s a timeline around when ECS will be supported?

Hey folks, I can’t seem to find an appropriate channel for this. Is there a (recommended) way how to maintain the used connectors using VCS? For instance, we want to be able to recreate all the connectors from the CI/CD pipeline, even in new installations of Airbyte. Thanks

Hello all, is there any possibility of supporting a “split-worker” mode where an Airbyte connection launches multiple workers, with each worker being responsible for a certain subcomponent of the source, e.g. one worker per repository for a Github source?

As of today I have made the decision (and started) to use Airbyte in production for replicating (and hopefully soon, Transforming) our backend data in our warehouse. Still our biggest wishlist and things that would definitely keep us in the ecosystem long term: • CDC for Mongo • User Management Thanks to John, Marcos, Shrif, Abhi and more who have spoken/engaged with me in the past months. Is there a separate channel for those of us who have made the big choice to use in production? :)

Is there a destination sftp connector? I have some vendors that require their files to be delivered via sftp

hey all, I'm I was going through the for for Airbyte Cloud and when it asks if I've ever used airbyte before, does it mean knowing how the tool works or does it mean used in a production environment so connectors won't extract all the historic data again? because also the expected monthly volume will change