Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hey all, I am trying to setup airbyte on AWS EKS using manifests files
I read the files and I see that there is a deployment of airbyte-cron
Can anyone please help me to understand why we need airbyte-cron ?

Airbyte cron is need to schedule and run cron syncs. Airbyte uses quartz scheduler to do that (<http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html|http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html>), so it is separated from main Airbyte.

<@U01MMSDJGC9> turned this thread into Zendesk ticket <https://airbyte7538.zendesk.com/agent/tickets/2898 | 2898> to ensure timely resolution!

Hey <@U03PWCEG39S> I know what cron job does in unix but I wanted to understand what it is doing in airbyte

Airbyte recently introduced cron to run its syncs, so you can schedule a sync in a cron fashion. You can still run basic syncs, just setting the frequency.

<https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection|https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection>

&gt; If you need to use cron scheduling:
&gt; 
&gt; In the Replication Frequency dropdown, click Cron.
&gt; Enter a cron expression and choose a time zone to create a sync schedule.
&gt; NOTE
&gt; Only one sync per connection can run at a time.
&gt; If cron schedules a sync to run before the last one finishes, the scheduled sync will start after the last sync completes.
&gt; Cloud does not allow schedules that sync more than once per hour

<@U046ZJADCNN>, let us know if Marcelo answered your question here! If you have more questions feel free to post in the channel :slightly_smiling_face:

hey <@U03PWCEG39S> I wanted to know if i can run the cron every minute in the airbyte open source deployment

Source :- Mysql
Target :- Mysql

Abhishek it is recommended to run minimum 5min, but you can test running every minute. Airbyte doesn't limit your cron frequency