Paweł Wilczyński
05/22/2023, 3:27 PM0.45.17 on GKE (using helm).
Not sure why syncing fails after around 27m 30s with errors like Failure during reporting of activity result to the server. ActivityId = .... , ActivityType = RunWithJobOutput, WorkflowId=connection_manager_7c51cd55-1d52-4402-a457-4ea954a2274d, WorkflowType=ConnectionManagerWorkflow, RunId=... io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
Any way to increase the timeout?
I also can see many pods being in Init:Error or Evicted state... So it looks like syncs are competing for CPU requests. Any way to limit the number of parallel jobs?kapa.ai
05/22/2023, 3:27 PMkapa.ai
05/22/2023, 3:28 PM.env file and change the JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT variables. Here's an example of how a user set them: [link](https://discuss.airbyte.io/t/increase-sync-worker-resources/1714/3). You can find more information on configuring Airbyte [here](https://docs.airbyte.com/operator-guides/configuring-airbyte). To limit the number of parallel jobs, you can set the following environment variables: - `MAX_SPEC_WORKERS`: Maximum number of Spec workers allowed to run in parallel. - `MAX_CHECK_WORKERS`: Maximum number of Check connection workers allowed to run in parallel. - `MAX_DISCOVERY_WORKERS`: Maximum number of Discovery workers allowed to run in parallel. - `MAX_SYNC_WORKERS`: Maximum number of Sync workers allowed to run in parallel. The default value for these environment variables is currently set to 5. You can find more information on this in the [Airbyte documentation](https://docs.airbyte.com/understanding-airbyte/jobs). Additionally, you can increase the number of replicas for the airbyte-worker deployment to allow more jobs to be processed. You can find examples of increasing worker pods in a Kustomization patch in airbyte/kube/overlays/dev-integration-test/kustomization.yaml and airbyte/kube/overlays/dev-integration-test/parallelize-worker.yaml as mentioned in the [Airbyte documentation](https://docs.airbyte.com/deploying-airbyte/on-kubernetes#increase-job-parallelism). Remember to also set the value for TEMPORAL_WORKER_PORTS in the .env file to limit the exposed ports, as not having enough ports to communicate might cause jobs to not complete or halt until ports become available.Paweł Wilczyński
05/23/2023, 12:19 PM