Now that the databricks connector also supports Az...
# replication-troubleshootingh
Huib
10/20/2022, 6:47 AMNow that the databricks connector also supports Azure I decided to give her a spin, with mixed results. The cluster Airbyte is going to use is not on 24/7, and this causes timeouts in various parts of the sync (either at the start or at the end). Is there anything I can do to prevent this? Increase the time Airbyte waits for the cluster to come online (in Azure this takes 5min easily, sometimes 10)
✍️ 1
Huib
10/20/2022, 6:50 AMWhat’s worse: the databricks connector uses a two-step approach where it first writes to a staging table before involving databricks and creating a delta table. When the second part fails, it also does the first part again - which is a complete waste of time and resources
e
Eli Sigal
10/20/2022, 7:51 AMHi.
I would open an issue on this since this requires code changes.
it is possible to do so though requires a bit logic.
1. if the runs state is new i.e. didnt have any retry then create staging table - verify data integrity and duplicates - try write to table.
2. if failed at step 2 and step 1 was ok fetch that state of the staging table and try the run again by increasing the timeout (dont create default high timeout since it really depends on the destination - only if fails increate by X amount)
3. if step 1 fails try to think of the exceptions and raises needed as to why it failed and work around this.
4. in general if step 1 fails i would return an error log with full details as to why - could be permissions, memory etc..
u
user
10/20/2022, 11:57 AM@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2966 to ensure timely resolution!
u
user
10/20/2022, 3:42 PMHey Huib, thanks for bringing this up! Could you please make a feature request issue for this on GitHub? Feel free to tag me there with @natalyjazzviolin, I'll make sure it gets triaged quickly.
Meanwhile I'll look into if there's a workaround for your use case!
h
Huib
10/20/2022, 5:04 PMThanks! As a workaround I’ve turned off auto-shutoff for the cluster so that the first (big) sync worked - the incremental syncs afterwards don’t suffer from the problem as much, as the cluster is still on from the initial “test connection” step - so for now I can sync my databases 🙂
Huib
10/20/2022, 5:11 PMTo clarify: the first incremental sync fails quickly (on checking the connection it times out) before any data has been transferred. The time it takes to try again (kill containers, boot new ones) is enough for the databricks cluster to come online - so eventually it works.
Huib
10/20/2022, 5:15 PMBy the way, if it would be possible to use a job cluster instead of a general purpose cluster (and an option to skip “waking up” the cluster at the beginning of the sync), that would be very helpful - It feels like a waste to leave a cluster running 24/7 for a handful of syncs that take maybe 1hr processing time total on the databricks end
Huib
10/20/2022, 5:21 PMthe way it’s currently implemented will quickly do that; assume you have 10 syncs, each taking 30min to put the files in blob/s3 and 15mins to start + insert into the delta tables. This should be 150mins of databricks time, but because it’s a general purpose cluster it doesn’t turn off immediately (15mins idle is what I use). So in practice it’ll be 50mins per sync (5 boot/check + 15mins idle , 5min boot + 10mins work, 15mins idle), or 500mins total - more than tripling the cost.
If you synced more than once a day, it would be very easy to have the cluster running 24/7, but having it idle most of the time
u
user
10/20/2022, 7:31 PMI'm so glad you found a workaround for this! Thanks for all the info, I'm sure other community members will find this useful as well :) I'm marking this as resolved and looking forward to seeing that feature request on GitHub!
6 Views