@kapa.ai why airbyte always pull image when they make a job?

Based on the knowledge sources provided, Airbyte doesn't always pull images for every job. However, there are situations where image pulling issues can occur, leading to the perception that Airbyte is constantly pulling images. Let me explain: 1. Image Caching: Normally, Airbyte should cache images locally after the first pull. This means that subsequent jobs should use the cached image rather than pulling it again. 2. Image Not Found Locally: If an image is not found locally, Airbyte will attempt to pull it. This is evident from the log messages in several issues, such as:

airbyte/source-postgres:3.3.18 not found locally. Attempting to pull the image...

[Github Issue: DOCKER COMPOSE : Could not find image: airbyte/](https://github.com/airbytehq/airbyte/issues/37433) 3. Docker Agent Issues: Sometimes, the problem may be related to the local Docker agent rather than Airbyte itself. As mentioned in one of the comments: "This often happens due to the local docker agent. This is not something Airbyte is responsible for." [Github Issue: Docker - Can't find/pull source-image during Sync](https://github.com/airbytehq/airbyte/issues/40642) 4. Network or Registry Issues: There might be problems with accessing the Docker registry or network issues preventing successful image pulls. 5. Rate Limiting: In some cases, rate limiting from Docker Hub can cause issues with pulling images, especially for users on shared IP ranges (like cloud providers). To mitigate these issues and reduce the frequency of image pulls, you can consider: 1. Using a custom image registry to keep images in your network, which can improve security and reduce external dependencies. 2. Configuring Airbyte to use a custom image registry by modifying the

values.yaml

file in your Helm chart:

yaml    global:      image:        registry: your-custom-registry-url

[Documentation: Custom image registry](https://docs.airbyte.com/deploying-airbyte/integrations/custom-image-registries) 3. Ensuring your Docker daemon has enough resources and is configured correctly. 4. If you're experiencing rate limiting issues, consider authenticating with Docker Hub or using a pull-through cache. If you're still experiencing frequent image pulls after implementing these solutions, it may be worth investigating further as it could indicate a configuration issue or a bug in your specific Airbyte setup.