https://linen.dev logo
Join Slack
Powered by
Hi <@U04SE163WC8> We have a custom connector built...
# replication-ask-ai
h
Hi @kapa.ai We have a custom connector built using the UI builder running daily pushing data to Postgres. The job used to take 1 hour, but more recently it's taking 5 hours. We have a T2.Large instance (airbyte OSS version 0.50.33). I can see that CPU usage is 99% during all that time. checking tasks, I could see 2 "java" tasks(commands) by the root user were the cause of CPU usage being maxed out. No other connector is set to sync. The job extracts a total of approx. 200K rows which is not that much. We didn't do any configuration, we just installed Airbyte using docker compose
k
Based on the information provided in the extracts, there could be several reasons for the increased sync time. Here are a few possibilities: 1. Resource Limitations: The high CPU usage indicates that your instance might be under-resourced for the task at hand. As per the [extract from this forum post](https://discuss.airbyte.io/t/3715), it's recommended to have a larger disk space (around 100GB) to avoid issues. Also, the CPU and memory resources allocated to the source and destination containers might not be sufficient. You might want to consider upgrading your instance or optimizing resource allocation. 2. Connector Performance: The performance of the connector can also impact the sync time. As seen in [this forum post](https://discuss.airbyte.io/t/3284), the user experienced poor sync performance with high volume tables. They tried several optimizations like increasing connection's CPUs and memory, changing output formats, and using ThreadPools in the custom connector, but the improvements were not significant. 3. Logging Impact: As per the [same forum post](https://discuss.airbyte.io/t/3284), excessive logging can also impact the sync performance. For a table containing a large number of rows, a large number of log messages are printed, which can potentially slow down the process. 4. Database Indexing: In [another forum post](https://discuss.airbyte.io/t/4007), a user found that the cursor_field was not indexed in the production database, leading to infinite response time. To troubleshoot, you could try the following: - Check if the issue is specific to the custom connector by testing with other connectors. - Monitor the resource usage during the sync process and adjust the resources accordingly. - Check the logs for any errors or warnings that might indicate the cause of the issue. - If the custom connector is querying a database, ensure that the fields used in the queries are properly indexed. Please note that these are general suggestions and the actual solution might vary depending on the specifics of your setup and the custom connector. If the issue persists, consider reaching out to the Airbyte community or support for more specific guidance.
A message from kapa.ai
Based on the information provided in the extracts, there could be several reasons for the increased sync time. Here are a few possibilities: 1. Resource Limitations: The high CPU usage indicates that your instance might be under-resourced for the task at hand. As per the [extract from this forum post](https://discuss.airbyte.io/t/3715), it's recommended to have a larger disk space (around 100GB) to avoid issues. Also, the CPU and memory resources allocated to the source and destination containers might not be sufficient. You might want to consider upgrading your instance or optimizing resource allocation. 2. Connector Performance: The performance of the connector can also impact the sync time. As seen in [this forum post](https://discuss.airbyte.io/t/3284), the user experienced poor sync performance with high volume tables. They tried several optimizations like increasing connection's CPUs and memory, changing output formats, and using ThreadPools in the custom connector, but the improvements were not significant. 3. Logging Impact: As per the [same forum post](https://discuss.airbyte.io/t/3284), excessive logging can also impact the sync performance. For a table containing a large number of rows, a large number of log messages are printed, which can potentially slow down the process. 4. Database Indexing: In [another forum post](https://discuss.airbyte.io/t/4007), a user found that the cursor_field was not indexed in the production database, leading to infinite response time. To troubleshoot, you could try the following: - Check if the issue is specific to the custom connector by testing with other connectors. - Monitor the resource usage during the sync process and adjust the resources accordingly. - Check the logs for any errors or warnings that might indicate the cause of the issue. - If the custom connector is querying a database, ensure that the fields used in the queries are properly indexed. Please note that these are general suggestions and the actual solution might vary depending on the specifics of your setup and the custom connector. If the issue persists, consider reaching out to the Airbyte community or support for more specific guidance.
h
@kapa.ai I get the same extraction time regardless of the destination (including redshift). the EC2 has a 20GB disk. How can I chck if the disk is the issue?
k
A message from kapa.ai
A message from kapa.ai
To check if the disk is the issue, you can monitor the disk usage during the sync process. Here are the steps you can follow: 1. Check the current disk usage: You can use the 
df -h
command in the terminal to check the current disk usage. This will show you how much disk space is used and how much is available. 2. Monitor the disk usage during the sync process: You can use the 
iostat
command to monitor the disk usage in real-time. This will show you the read/write speed and other disk-related metrics. If you see that the disk is constantly at high usage during the sync process, it might be the bottleneck. 3. Check the Airbyte logs: Airbyte stores logs in the disk. If the disk space is not sufficient, it might cause issues. You can check the Airbyte logs for any errors or warnings related to disk space. You can get the logs by running 
docker logs airbyte-server > server.log
as mentioned in [this forum post](https://discuss.airbyte.io/t/3715). If you find that the disk is indeed the issue, you might want to consider increasing the disk space. As per the [extract from this forum post](https://discuss.airbyte.io/t/3715), it's recommended to have a larger disk space (around 100GB) to avoid issues. Also, as per the [extract from this documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte/), Airbyte recommends allocating a minimum of 30GBs of disk space per node. Since storage is on the cheaper side, it's better to err on the side of over-provisioning.
You can check the disk usage on your EC2 instance by using the 
df
command, which reports the amount of disk space used and available on filesystems. Here's how you can do it: 1. SSH into your EC2 instance. 2. Run the 
df -h
command. This will display the disk usage in a human-readable format. The output will look something like this:
Copy code
Filesystem      Size  Used Avail Use% Mounted on udev            7.9G     0  7.9G   0% /dev tmpfs           1.6G  8.8M  1.6G   1% /run /dev/xvda1      7.7G  7.7G     0 100% / tmpfs           7.9G     0  7.9G   0% /dev/shm tmpfs           5.0M     0  5.0M   0% /run/lock tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup /dev/loop0       98M   98M     0 100% /snap/core/10185 /dev/loop1       98M   98M     0 100% /snap/core/10126 tmpfs           1.6G     0  1.6G   0% /run/user/1000
In this example, you can see that the 
/dev/xvda1
filesystem (which is likely the filesystem your EC2 instance is using) is 100% full. If your disk is indeed full or nearly full, this could be the cause of the issue. As per the [extract from this forum post](https://discuss.airbyte.io/t/3715), it's recommended to have a larger disk space (around 100GB) to avoid issues. You might want to consider increasing the disk size of your EC2 instance. Also, as per the [extract from this documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte), Airbyte recommends allocating a minimum of 30GBs of disk space per node. If storage is on the cheaper side, it's better to err on the side of over-provisioning.
h
@kapa.ai I have a 20GB disk, it's been used up by var/lib/docker. here are all the images. why are there multiple with the same name?
Copy code
REPOSITORY                             TAG       IMAGE ID       CREATED        SIZE
airbyte/source-google-search-console   1.3.6     493fff94430e   5 days ago     434MB
airbyte/source-postgres                3.2.14    d5e6f3abfb7c   5 days ago     975MB
airbyte/airbyte-api-server             0.50.33   dbc6e2af771e   6 days ago     1.01GB
airbyte/worker                         0.50.33   41a9b1f50a82   6 days ago     1.85GB
airbyte/webapp                         0.50.33   f64b37609a59   6 days ago     52.4MB
airbyte/temporal                       0.50.33   5a438137a90b   6 days ago     344MB
airbyte/server                         0.50.33   9eb5a24d517c   6 days ago     1.06GB
airbyte/proxy                          0.50.33   f8357d70f4d1   6 days ago     188MB
airbyte/db                             0.50.33   779ca2de2c01   6 days ago     232MB
airbyte/connector-builder-server       0.50.33   a425a4cc33ca   6 days ago     2.24GB
airbyte/cron                           0.50.33   c79d5004b80a   6 days ago     1.06GB
airbyte/init                           0.50.33   615df4bc0782   6 days ago     5.61MB
airbyte/bootloader                     0.50.33   e0c478094a33   6 days ago     1.04GB
airbyte/source-hubspot                 1.6.1     9b34c1de1a11   11 days ago    333MB
airbyte/source-declarative-manifest    0.51.41   2eb57edba49d   12 days ago    74.6MB
airbyte/source-google-analytics-v4     0.2.2     b914ad5b2a85   2 weeks ago    333MB
airbyte/destination-redshift           0.6.8     c41561615a4e   3 weeks ago    1.31GB
airbyte/source-hubspot                 1.4.1     8ef89c4ee4b3   2 months ago   74.5MB
airbyte/airbyte-api-server             0.50.11   ac89cee4815e   3 months ago   891MB
airbyte/worker                         0.50.11   0a8f0466df4e   3 months ago   1.86GB
airbyte/webapp                         0.50.11   4d1a9bb4c728   3 months ago   163MB
airbyte/temporal                       0.50.11   90c33adf4e32   3 months ago   344MB
airbyte/server                         0.50.11   4ab8adb8ad9d   3 months ago   1.06GB
airbyte/proxy                          0.50.11   15f0c98c3849   3 months ago   188MB
airbyte/db                             0.50.11   90299b0b7a38   3 months ago   232MB
airbyte/connector-builder-server       0.50.11   d98dba11ac8d   3 months ago   2.28GB
airbyte/cron                           0.50.11   5af2b0590ebe   3 months ago   1.06GB
airbyte/init                           0.50.11   bc63494182d7   3 months ago   5.61MB
airbyte/bootloader                     0.50.11   779ad5c33bbd   3 months ago   1.03GB
airbyte/source-declarative-manifest    0.47.1    5bacde8fea74   3 months ago   114MB
airbyte/source-hubspot                 1.1.0     eea05e810e07   3 months ago   116MB
airbyte/source-google-analytics-v4     0.2.1     0e0e3322f237   3 months ago   252MB
airbyte/source-postgres                3.0.2     e608ed5d9ccb   3 months ago   1.03GB
airbyte/destination-redshift           0.6.1     600b448d5cf0   3 months ago   2.08GB
airbyte/source-google-search-console   1.2.1     d9093c7f8cca   3 months ago   331MB
airbyte/destination-postgres           0.4.0     29374b330953   4 months ago   1.28GB
airbyte/destination-s3                 0.5.1     f2facfcbe901   4 months ago   1.27GB
airbyte/source-xero                    0.2.3     8b2bae5de5cc   4 months ago   115MB
airbyte/normalization-redshift         0.4.3     a2102c72e15f   5 months ago   784MB
k
A message from kapa.ai
The multiple images with the same name but different tags represent different versions of the same Docker image. Each tag usually represents a different version of the application contained in the image. In your case, you have multiple versions of the same Airbyte component. For example, 
airbyte/source-google-search-console
has versions 
1.3.6
and 
1.2.1
. This is likely because when you updated Airbyte, the older images were not removed. These images can take up a significant amount of disk space. If you're not planning to use the older versions, you can remove them to free up disk space. Here's how you can do it: 1. List all Docker images:
Copy code
bash docker images
2. Remove a specific Docker image:
Copy code
bash docker rmi <IMAGE ID>
Replace 
<IMAGE ID>
with the ID of the image you want to remove. You can get the image ID from the output of the 
docker images
command. Please note that you should be careful when removing Docker images. Make sure that the image is not being used by any running container. If you try to remove an image that's being used by a running container, Docker will prevent you from doing so. Also, remember to keep the latest versions of the images, as they contain the latest features and bug fixes. If you want to remove all unused Docker images, you can use the following command:
Copy code
bash docker image prune -a
This will remove all images without at least one container associated with them. Be careful with this command, as it can remove more than you might expect if you have images that are not being used by any container.
h
@kapa.ai where can I see the used vs unused images?
k
A message from kapa.ai
@Hubert Delandre That's a great question, but unfortunately I am only capable of responding to a maximum of five questions per thread. Don't hesitate to create a new thread for any further questions you have though. I'm here to help!
2 Views
Open in Slack
PreviousNext
Powered by