The

replication-orchestrator

pod in my incremental stream is logging many lines "Source state message checksum is valid for stream

stream_name

. It seems like it is slowing down my incremental sync. What is the cause of this?

It sounds like the issue you're experiencing with the

replication-orchestrator

pod logging many lines about "Source state message checksum is valid for stream `stream_name`" could be related to state message handling and validation during the sync process. From the provided knowledge sources, there are a couple of relevant points that might help explain the cause: 1. State Checksum Validation: - In the issue [Disable In app counts on State Collisions](https://github.com/airbytehq/airbyte/issues/34359), it was mentioned that state stats checksum validation should be disabled when a state message collision is detected. This suggests that excessive state validation could be a performance bottleneck, especially if there are many state messages being processed. 2. High CPU Usage During Sync: - The issue [High orchestrator CPU during slow sync](https://github.com/airbytehq/airbyte/issues/37837) describes a scenario where the orchestrator pod consumes a lot of CPU during slow syncs, particularly when waiting for messages from a slow source. This high CPU usage could be exacerbated by frequent state message validations. Given these points, it seems that the frequent logging and validation of state message checksums could be contributing to the slowdown of your incremental sync. Disabling or optimizing the state checksum validation process might help alleviate this issue. ### Actionable Steps: 1. Review and Optimize State Validation: - Consider reviewing the state validation logic to see if it can be optimized or if unnecessary validations can be skipped, especially in the case of state collisions. 2. Monitor CPU Usage: - Monitor the CPU usage of the orchestrator pod to identify if the high CPU usage correlates with the state message validations. This can help confirm if the state validation is indeed the bottleneck. 3. Adjust Logging Levels: - If the logging itself is causing performance issues, consider adjusting the logging levels to reduce the verbosity of the logs during the sync process. For more detailed guidance, you might want to refer to the specific issues mentioned above or consult the [Airbyte documentation](https://docs.airbyte.com/) for further troubleshooting steps.