Hey :wave: How does one go about estimating EC2 in...
# announcementsc
Charles Lariviere
11/19/2020, 6:27 PMHey 👋 How does one go about estimating EC2 instance size needed to run Airbyte? We set it up on t3.micro, though our syncs fail and the logs include java error messages around running out of memory — assuming this is because the instance does not have enough memory?
Charles Lariviere
11/19/2020, 6:27 PMThe sync continues after these logs, but ends if an error after creating the table but not loading anything.
Copy code
DefaultAirbyteStreamFactory(lambda$create$0):73 - # There is insufficient memory for the Java Runtime Environment to continue.
2020-11-19 17:49:42 INFO (/tmp/workspace/19/4) DefaultAirbyteStreamFactory(lambda$create$0):73 - # Native memory allocation (mmap) failed to map 50806784 bytes for committing reserved memory.
2020-11-19 17:49:42 INFO (/tmp/workspace/19/4) DefaultAirbyteStreamFactory(lambda$create$0):73 - # An error report file with more information is saved as:
2020-11-19 17:49:42 INFO (/tmp/workspace/19/4) DefaultAirbyteStreamFactory(lambda$create$0):73 - # /data/19/4/hs_err_pid7.log
2020-11-19 17:49:42 ERROR (/tmp/workspace/19/4) LineGobbler(voidCall):69 - OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000fa8ad000, 50806784, 0) failed; error='Not enough space' (errno=12)s
s
11/19/2020, 7:50 PM
Hi Charles! the main bottleneck is going to be the machine size since docker takes up as much as space as it needs. t3.micro provides 1GiB memory which is a bit low. I’d start with a t3.medium which provies 4GiB. I created an issue to provide clearer guidelines here: https://github.com/airbytehq/airbyte/issues/1025
c
Charles Lariviere
11/19/2020, 7:58 PMThanks @s! I ended up going with t3.large which worked 👍 On another subject, but since I got this to work I was able to test Airbyte for the first time! I set up Postgres->Snowflake, and realized that Airbyte sends an
INSERT()j
Jared Rhizor (Airbyte)
11/19/2020, 9:08 PM
This is unexpected since we try to batch 500 records into a single statement at a time, except for the final insertion with the remaining records that may be <500.
Jared Rhizor (Airbyte)
11/19/2020, 9:08 PM
If you have the time, would you mind sharing the logs for the sync?
m
Michel
11/24/2020, 3:50 AM
@Charles Lariviere we indeed have an issue with the snowflake implementation and an invalid assumption about how the jdbc driver is working. We are investigating how to fix it. FYI: https://github.com/airbytehq/airbyte/issues/1054
c
Charles Lariviere
11/24/2020, 2:35 PMThanks for the update! Sorry I missed the earlier messages.
m
Michel
11/25/2020, 10:56 PM
Hey @Charles Lariviere, we actually resolved the issue. It is on master now. @charles since we haven't create a release yet, can you just explain how Charles can use the latest version of snowflake?
c
charles
11/25/2020, 10:57 PM
Of course! We are also planning to do a release this afternoon to include this fix.
c
Charles Lariviere
11/26/2020, 12:22 AMSounds great! I’ll give that a spin once it’s released — no rush on my end. Appreciate the update 🙏
c
charles
11/26/2020, 2:53 AM
Hi @Charles Lariviere I have released our newest version v0.6.0-alpha which includes the fix for snowflake.
charles
11/26/2020, 2:54 AM
all you should need to do is pull down the most recent version of master to take advantage of this.
charles
11/26/2020, 2:54 AM
lmk if you need any help!
3 Views