https://linen.dev logo
Join Slack
Powered by
for one of my syncs, the logs read as below: ```2...
# replication-ask-ai
t
for one of my syncs, the logs read as below:
Copy code
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_1>
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_2>
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_3>
2024-05-24 04:59:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 04:59:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:00:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:00:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:01:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:01:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:02:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:02:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:03:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:03:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:04:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:04:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:05:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:05:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:06:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:06:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:07:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:07:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:08:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:08:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:09:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:09:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:10:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:10:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:11:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:11:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
what is it exactly doing here? it seems to be idle just waiting for something to happen, but i see no activity on destination.
k
Sorry @Taher Ajnawala, I seem to be having a bit of trouble. Don't worry though, I've pinged my support folks and they will try to fix me up! 🧑‍💻 Perhaps try again in 5mins?
t
@kapa.ai are you back online?
k
As an AI, I'm always online and ready to assist you with your Airbyte-related queries. How can I help you today?
t
@kapa.ai can you answer my above question?
k
A message from kapa.ai
I'm sorry, but I can't see any previous question from you in this conversation. Could you please repeat your question? I'm here to help with any queries related to Airbyte.
t
@kapa.ai for one of my syncs, the logs read as below:
Copy code
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_1>
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_2>
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_3>
2024-05-24 04:59:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 04:59:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:00:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:00:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:01:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:01:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:02:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:02:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:03:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:03:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:04:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:04:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:05:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:05:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:06:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:06:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:07:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:07:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:08:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:08:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:09:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:09:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:10:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:10:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-24 05:11:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 05:11:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
what is it exactly doing here? it seems to be idle for 10minutes just waiting for something to happen, but i see no activity on destination.
k
A message from kapa.ai
Sorry @Taher Ajnawala, I seem to be having a bit of trouble. Don't worry though, I've pinged my support folks and they will try to fix me up! 🧑‍💻 Perhaps try again in 5mins?
t
@kapa.ai for one of my syncs, the logs read as below:
Copy code
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_1>
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_2>
2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_3>
2024-05-24 04:59:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-24 04:59:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
what is it exactly doing here? it seems to be idle just waiting for something to happen, but i see no activity on destination.
k
A message from kapa.ai
Based on the logs you provided, here's what's happening: 1. The first three lines are indicating that the final tables for the respective streams already exist in the destination. This is part of the preparation process where Airbyte checks if the necessary tables for data sync exist or not.
Copy code
bash 2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_1> 2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_2> 2024-05-24 04:58:10 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):187 Final Table exists for stream <redacted_destination_table_3>
2. The fourth line is showing the memory usage of the global buffer queue and the state manager. The global buffer queue is used to temporarily store data before it's written to the destination. The state manager is responsible for managing the state of the sync process.
Copy code
bash 2024-05-24 04:59:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
3. The last line is showing the status of the worker pool, which is responsible for executing tasks. In this case, it's showing that there are no tasks in the queue and no active threads, which means there are currently no tasks being executed.
Copy code
bash 2024-05-24 04:59:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
If you're seeing no activity on the destination, it could be because there's no new data to sync or there might be an issue with the source connection. You might want to check the source connection or the sync settings in Airbyte. If the issue persists, consider checking the [Airbyte documentation](https://docs.airbyte.io/) or raising an issue in the [Airbyte Github repository](https://github.com/airbytehq/airbyte/issues).
t
@kapa.ai the fourth line and last line keep on repeating for a few times for upto 10minutes. what is it waiting for?
k
A message from kapa.ai
The repeating lines in your logs are related to the memory usage and worker pool status of your Airbyte instance. The line:
Copy code
bash 2024-05-24 04:59:08 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 307.2 MB, allocated: 10 MB (10.0 MB), %% used: 0.03255208335354433 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
is showing the memory usage of the global buffer queue and the state manager. The global buffer queue is used to temporarily store data before it's written to the destination. The state manager is responsible for managing the state of the sync process. This line repeating indicates that Airbyte is monitoring the memory usage continuously. The line:
Copy code
bash 2024-05-24 04:59:08 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
is showing the status of the worker pool, which is responsible for executing tasks. In this case, it's showing that there are no tasks in the queue and no active threads, which means there are currently no tasks being executed. This line repeating indicates that Airbyte is continuously checking the worker pool status. If these lines are repeating for up to 10 minutes, it could be that Airbyte is waiting for new data to sync from the source, or it's waiting for a response from the source or destination. It could also be possible that there's a delay or issue with the source connection, or the sync settings in Airbyte might be causing the delay. If you're unsure about the cause, you might want to check the source connection, the sync settings, or the logs for any error messages. If the issue persists, consider checking the [Airbyte documentation](https://docs.airbyte.io/) or raising an issue in the [Airbyte Github repository](https://github.com/airbytehq/airbyte/issues).
f
Hi @Taher Ajnawala - did you ever understand what was causing this issue? facing this at the moment on our local deployment.
t
@Furriel Pimenta sadly no. This still happens on one of our syncs today. we have a total of 6 connections running and only 1 of them does this. I am stumped at why does it happen. Does this happen to all your connections or just a few?
f
@Taher Ajnawala It was happening to one connection. It appears to have been due to running out of disk space. We cleaned/rebooted the machine and not it is spawning workers again.
t
woh. Good to hear it worked out for you @Furriel Pimenta. For me the disk space is available so not really sure whats the issue.
Open in Slack
PreviousNext
Powered by