Hi, I was wondering if anyone has looked into integrating the data from Heap into Airbyte (so that it could be synced with an external platform like marketo for instance)?

Question: How to solve the Notification error in the Airbyte API configuration? While creating workspace using POST /v1/workspaces/create in Airbyte API Documentation with example as per the schema given below, I am getting Response Status: 422. Please find the reference : https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#servers The example that I have used to generate POST request: { "email": "user@example.com", "anonymousDataCollection": false, "name": "string", "news": false, "securityUpdates": false, "notifications": [ { "notificationType": "slack", "slackConfiguration": { "webhook": "string" } } ] } The response that I have got:

{
  "message": "Invalid JSON",
  "details": "Unrecognized field \"notifications\" (class io.airbyte.api.model.WorkspaceCreate), not marked as ignorable"
}

<!channel> Hey everyone 👋! here are the updates for the past week: • Welcome to @Mason Wheeler who's joining our team as senior software engineer 🎉 🎉 ! We're so excited to have him on board! • Airbyte has become an official package provider of Airflow! • Progress on connectors: ◦ Oracle DB is now supported as a source connector, thanks to @[DEPRECATED] Marcos Marx ◦ Some fixes • Progress on Core (0.20.0): ◦ Change Data Capture (CDC) is now supported for Postgres, thanks to @Jared Rhizor (Airbyte) and @charles. We will now expand it to MySQL and MSSQL in the coming weeks. ◦ When displaying the schema for a source, you can now search for table names, thanks to @Artem Astapenko ◦ Better feedback UX when manually triggering a sync with “Sync now” • 3 months ago, we published our OKRs for Q1 2021. Here’s an article about how we performed on those OKRs, and also about our goals for Q2. You will see how much the community has grown, which is super exciting! • Don’t forget to register to our 2nd community call on Wednesday 05/05 at 3pm PST with @Jins Kadwood, who will show us how AgriDigital is leveraging Airbyte for their caching strategy.

Welcome @Mason Wheeler

Hi everyone! I've been trying to install airbyte on GCP but I keep getting errors. I'm new to GCP and Airbyte, so I don't really know what's going on. I would really appreciate some help. Thank you in advance. 😁

Hi all, our use-case with Airbyte is to embed it in our SaaS product, and use it’s API internally to create connections, but also load runtime status of connections and jobs to visualize it in our own UI. I don’t think we need source and destination be in namespace, but just connections (in our use case). I heard about the workspace/namespace concept to separate our tenants/customers connections natively, so that loading data is always tenant-separated. I saw activities around creating/listing/deleting workspaces (https://github.com/airbytehq/airbyte/issues/1735) but could not see (on a first glimpse)

wordpaceId

params in all other API endpoints. Is this currently a feature we should rely on, or should we (for now) develop a naming scheme for connections (can connections even have a name today?)?

Hi, can anyone help me on how to build a connector?

I was just testing out the oracle source connector and it works great. Will Oracle be added as a destination too?

Hi all. Probably it was answered many times already. What would be the best way to combine frequent key based replication (PostgreSQL in GCP thus no CDC, and Mongo say every hour) with nightly full syncs? It’s possible of course to setup 2 different sources and destinations for the same database, configure them differently and combine them with DBT. I am wondering if there is a better way out of the box?

Hello everyone! Is it possible to only fetch those fields which are needed, let say if a source-connector provides 30 fields but I find only 5 fields useful my use case. Can I fetch only those fields?

<!channel> Hey everyone 👋! here are the updates for the past week: • 3 new source connectors! ◦ Zendesk Talk ◦ Iterable  ◦ Quickbooks • Other progress on connectors: ◦ Postgres source/destination: add SSL option, thanks to @[DEPRECATED] Marcos Marx (#2757) ◦ Google sheets bugfix: handle duplicate sheet headers, thanks to @Aneesh Makala (#2905) ◦ Source Google Adwords: support specifying the lookback window for conversions, thanks to @Harshith (Airbyte) (#2918) ◦ MongoDB improvement: speed up mongodb schema discovery, thanks to Yury Koleda (#2851) ◦ MySQL bugfix: parsing Mysql jdbc params, thanks to @Vasily Safronov (#2891) ◦ CSV bugfix: discovery takes too much memory, thanks to @Oleh Zorenko (#2089) ◦ A lot of work was done on improving the standard tests for the connectors, for better standardization and maintenance! • Progress on Core (0.21.0):  ◦ Namespace support: supported source-destination pairs will now sync data into the same namespace as the source (#2862), thanks to @Davin Chia (Airbyte) ◦ Add “*Refresh Schema*” button (#2943), thanks to @Artem Astapenko ◦ In the Settings, you can now add a webhook to get notified when a sync fails, thanks to @Chris (deprecated profile) (#2938) ◦ Add destinationSyncModes to connection form ◦ Add tooltips for connection status icons • We will soon release a framework for rapidly developing and testing high quality connectors, and would love some beta testers! If you want to try out the new framework to implement a connector (especially a REST-API based one) respond to this message and we’ll reach out. • Misc: ◦ Don’t miss our CEO, Michel, talking on the podcast “Making Data Simple” here ◦ Our 2nd connector contest will end on 04/30. Feel free to open your PRs for your connectors to have a (very high) chance of winning $500 worth of prizes! ◦ Don’t forget to register to our 2nd community call on Wednesday 05/05 at 3pm PST with @Jins Kadwood Kadwood, who will show us how AgriDigital is leveraging Airbyte for their caching strategy.

Random question - we’re looking at Precog, and wanted to understand how you’re different https://precog.com/ We’re testing out both, but just curious in case someone is able to shed light

👋 Hi everyone! So i have been exploring airbyte for a week now and i must say job well done guys.

i am trying to build a destination connector for S3 but could not find java connector setup option in when i do npm run generate

Hey Everyone! 1. I'm new to Airbyte, and want to install it on testing server(VM). 2. After installation how am i suppose to run airbyte UI on local browser? Can you please help me with this?

We did a POC with Airbyte (and Salesforce) and team came up with following questions. Can someone help me on these questions • We noticed duplication of data into 3 different tables for the same object with syncMode=’incremental’ and destinationSyncMode=’append_dedup’ configuration. How can we archive the unwanted data from those tables? • Despite providing the syncCatalog configuration to fetch specific fields from the tables, airbyte actually syncs the entire object on

_airbyte_raw_<object>

table as a json object and then it is duplicated in

_scd

and actual object tables. Overall, the data is duplicated thrice in our datastore. How can we reduce data duplication? • How can we monitor if the data is getting synced properly? Is there any way from the logs to determine if the sync failed for a specific workspace (like having some workspace identifier in logs) ? • What is the difference between a free and paid plan. What additional features do we get apart from support?

Can I ask what the constant @ Worknet in this workspace is about?

Hi there, I was not sure where this should go as it’s not directly an issue. I wondered why airbyte maps date / datetype / nurmerics to strings. Is the expectation then to remap it to another format via dbt to similar?

I’d appreciate what the expected best practice is please?

Hi, I was setting up Airbyte on AWS  by following the same instructions described here: https://docs.airbyte.io/deploying-airbyte/on-aws-ec2 But I wanted to set up on different port than default 8000. In the doc its mentioned to update API_URL in env file - So can you please help me what exactly needs to be added in docker-compose.yaml file? webapp:     image: airbyte/webapp:${VERSION}     container_name: airbyte-webapp     ports:       - 5000:80     environment:       - AIRBYTE_ROLE=${AIRBYTE_ROLE:-}       - AIRBYTE_VERSION=${VERSION}       - API_URL=${API_URL:-} I did tried to change ports section in webapp ports but when i tried to run UI on browser. Its opening Airbyte but giving error-

Hi everyone, is there a way to schedule a connector to run at or soon after a certain time each day?

hi everyone, I am finding it difficult to connect to MongoDB. I will sincerely appreciate any form of help..

Anyone have experience putting Airbyte behind a proxy? I added http_proxy/https_proxy to the yaml env vars, but the containers don't seem to be respecting them. The page doesn't load but whatever request is breaking it in the background isn't hitting the proxy.

Hello. I got an instance of airbyte running locally (with some changes to the docker-compose method), and while everything seems to be running, I'm failing to setup Salesforce as a source. I created an oauth client, followed through the guide to get a refresh token and yet the connection tests fail. Is there any way to troubleshoot?

Hey Guys, I'm amazed by the app and using demo. Just started demo on localhost with intrudaction, but don't know where in config files change host to my IP. (I hooked app on VPS)

So i've been looking around in the documentation to figure out where the

state

is stored? I mean is it possible to change that through the UI for a specific source? Lets say i have a source and i want to manually set the

state

back due to, some manual manipulation in my destination (datawarehouse).

<!channel> Hey everyone 👋 ! Here are the updates for the past week:  • We’re super thrilled to welcome @abhi as our new senior developer advocate 🎉. Abhi will help bring you the best developer experience. If you have any suggestions on documentation or tutorial, he’s your go-to person ;) • We will also soon have a user success engineer, whose entire focus is to help you become successful in your data pipelining efforts! We will announce more on that matter tomorrow [spoiler 😉] • We have renamed #general into #C01A4CAP81L to be more explicit about its purpose. For any technical issues, please post on #C01MFR03D5W For any questions, on #C019WJFR1JT . For anything related to connectors, on #C01AHCDHCKE • Progress on Core: ◦ Replace timeout for sources (#3031), thanks to @charles ◦ Fix UI issue where tables with the same name are selected together (#3032), thanks to @Artem Astapenko ◦ Fix feed handling when feeds are unavailable (#2964), thanks to @charles ◦ Export whitelisted tables (#3055), thanks to @Jared Rhizor (Airbyte) ◦ Create a contributor bootstrap script (#3028) (#3054), thanks to nclsbayona   • Progress on Connectors ◦ Zendesk Talk: fix normalization failure (#3022), thanks to yevhenii-ldv ◦ Github: pull_requests stream only incremental syncs (#2886) (#3009), thanks to Zirochkaa ◦ Create streaming writes to a file and manage the issuance of copy commands for the destination (#2921), thanks to @Jared Rhizor (Airbyte) ◦ Redshift: make Redshift part size configurable. (#3053), thanks to @Davin Chia (Airbyte) ◦ Hubspot: fix argument error in log call (#3087) (#3091) , thanks to Nathan Yergler

<!channel> Hey everyone 👋! here are the updates for the past week: • We’re stoked to welcome @Subodh (Airbyte) as our new senior software engineer 🎉. Subodh was Fivetran’s first engineer in India, and therefore knows a great deal about ELT at scale! Subodh will also be able to help on APAC timezones! • 2 new sources:  ◦ Smartsheets, thanks to Nathan Nowack ◦ Zendesk Chat, thanks to Yevhenii • Progress on connectors: ◦ Appstore: bugfix private key handling in the UI (#3201), thanks to @s ◦ Facebook marketing: Wait longer (5 min) for async jobs to start (#3116), thanks to @Max Krog ◦ Stripe: support reading data from connected accounts (#3121), and 2 new streams with Refunds & Bank Accounts (#3030) (#3086), thanks to Oleh Zorenko ◦ Redshift destination: Ignore records that are too big (instead of failing) (#2988), thanks to Chris Duong ◦ MongoDB: add supporting TLS and Replica Sets (#3111) ◦ HTTP sources: bugfix on handling array responses gracefully (#3008) • Progress on Core (0.22.2):  ◦ Split replication and normalization into separate temporal activities (#3136), thanks to @charles ◦ Fix normalization Nesting bug (#3110), thanks to @Chris (deprecated profile) • Misc:  ◦ Just one week after joining the team, @abhi already gave a

talk at KubeCon about how Airbyte is using Kubernetes▾

(13min video). That's what we call a supersonic ramp-up!! ◦ Don't forget our 2nd community call on this Wednesday 1pm PST with @Jins Kadwood as our guest to present their use case of Airbyte on their caching strategy ◦ We restructured the documentation to make it easier for new users to find whatever they’re looking for. Let us know how you feel about it!

Reminder: It's office hours time!! Join now: https://officehours.airbyte.io

Hey everyone 👋 ! We just posted on HackerNews about our new Connector Development Kit that lets you build data connectors in hours, instead of days! Look for "data connector" on https://news.ycombinator.com/newest! airbyte rocket airbyte rocket