@kapa.ai I execute it manually because copy command seems to not end...will it impacting in next sync?

BEGIN
BEGIN TRANSACTION;
MERGE INTO "IMPORT"."LRCS_PHASE_INFO" as TARGET_TABLE
USING (
  WITH intermediate_data AS (

UPDATE "airbyte_internal"."IMPORT_raw__stream_LRCS_PHASE_INFO" 
SET "_airbyte_loaded_at" = CURRENT_TIMESTAMP() 
WHERE "_airbyte_loaded_at" IS NULL;
COMMIT;
EXCEPTION
WHEN OTHER THEN
    ROLLBACK;
    BEGIN
        RAISE;
    END;
END;

Based on the knowledge sources available, I can provide some context, though the sources don't directly address your specific scenario of manually executing the typing and deduplication SQL. What the SQL is doing The SQL you're running is part of Airbyte's typing and deduplication (T&D) process. Specifically: 1. The

MERGE

statement moves data from the raw table into the final destination table. 2. The

UPDATE

on

_airbyte_loaded_at

marks records in the raw table as processed (setting

_airbyte_loaded_at = CURRENT_TIMESTAMP()

where it was previously

NULL

). The

_airbyte_loaded_at

field indicates when Airbyte loaded the record into the destination, and it's used by the T&D process to know which raw records still need to be processed. [Airbyte metadata fields] Potential Impact on Next Sync The knowledge sources do not explicitly describe what happens when you manually execute these T&D SQL statements. However, based on what is documented: • If you successfully set

_airbyte_loaded_at

on records that were previously

NULL

, Airbyte's next sync may skip re-processing those raw records during the T&D phase, since it uses

_airbyte_loaded_at IS NULL

to identify unprocessed records. • If the

MERGE

into the final table was not completed successfully (e.g., rolled back), but

_airbyte_loaded_at

was set, those records could be lost — present in the raw table but never merged into the final table. Recommendation I'd strongly caution against manually running these internal SQL statements, as it can put the raw and final tables in an inconsistent state. The knowledge sources do not provide a supported method for manually intervening in the T&D process. If your sync is hanging, it may be worth investigating the root cause (e.g., network issues, destination timeouts) rather than manually completing the SQL. If you need to reprocess data, consider using Airbyte's built-in refresh or reset functionality instead. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai did that on snowflake..loading data from mysql

Based on the knowledge sources available, I can confirm that the context you've described — manually running the T&D SQL on Snowflake as a destination (loading from MySQL) — is consistent with Airbyte's typing and deduplication process for Snowflake. The key concern remains the same as mentioned previously: the

_airbyte_loaded_at

column in the raw table (

airbyte_internal.IMPORT_raw__stream_LRCS_PHASE_INFO

) is what Airbyte uses to track which records have been processed. [Airbyte metadata fields] If you need to recover from an inconsistent state on Snowflake, the documented approach is: 1. Drop the final table:

DROP TABLE <your_final_table>

2. Reset the raw table's

_airbyte_loaded_at

column:UPDATE airbyte_internal.your_raw_table SET _airbyte_loaded_at = NULL 3. Run a new sync. This will allow Airbyte to rebuild the final table correctly from the raw data. [Typing and deduping] The knowledge sources do not provide specific details about how Airbyte handles a partially completed manual execution of the T&D SQL beyond this recovery approach. I'd recommend using the steps above to ensure a clean and consistent state before your next sync. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I've start a new sync...seems airbyte is trying to do a re-sync again....want to avoid that.

Based on the conversation history, it sounds like Airbyte may be re-processing data because of the manual changes made to the

_airbyte_loaded_at

column or the state of the raw/final tables. The knowledge sources describe the following about how Airbyte handles full refresh and table re-creation: • For certified database and data warehouse destinations (like Snowflake), during a full refresh, Airbyte writes to a temporary table (

final-table-tmp

) while the sync runs, then drops the old final table and renames the new one into place. [Full Refresh - Overwrite] • Airbyte uses

_airbyte_loaded_at IS NULL

in the raw table to identify records that still need to be processed. If those values were reset to

NULL

(as suggested in the previous recovery steps), Airbyte will re-process all raw records. [Typing and deduping] To avoid a full re-sync going forward, the key is to ensure the raw table's

_airbyte_loaded_at

values are not NULL for records that have already been processed. If you manually set them back to

NULL

as part of the recovery steps, that would explain why Airbyte is re-processing everything. Unfortunately, the knowledge sources do not provide a way to stop an already-running sync mid-process or to selectively prevent re-processing of specific records. The documented options are limited to the reset/recovery flow described previously. I'd recommend letting the current sync complete, as interrupting it again could leave the tables in an inconsistent state once more. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

it is not null, but airbyte is taking a lower pk value that should pick

Based on the knowledge sources available, I don't have specific documentation about MySQL's

MySqlInitialSyncStateIterator

or the

PrimaryKeyLoadStatus

internal state mechanism that your log is referencing. However, what the log shows is that Airbyte is performing an initial sync using primary key (PK) chunking — this is a MySQL-specific behavior where Airbyte reads data in chunks ordered by primary key, tracking progress via a

pkVal

state. The log line:

stateType=primary_key, pkName=PHASE_INFO_ID, pkVal=158905182, incrementalState={}

The

incrementalState={}

being empty is notable — it suggests Airbyte is treating this as an initial load rather than a continuation of an incremental sync. This could explain why it appears to be re-loading data from a lower PK value. This behavior is consistent with what was discussed in the MySQL to Snowflake incremental loading issues thread, where a sporadic failure could cause the sync state to be lost or reset, leading Airbyte to restart from the beginning. [MySQL to Snowflake] The knowledge sources do not provide enough detail about the internal PK chunking state mechanism to give you a definitive answer on why it picked a lower PK value. What I can suggest based on the documented guidance is: • If the sync state was corrupted or partially applied due to the earlier manual SQL execution, Airbyte may have fallen back to an initial sync state. • The recommended fix in such cases is to reset the connection and start fresh, accepting that a full re-sync will occur. [MySQL to Snowflake] I'd recommend checking the Airbyte GitHub issues or community forum for more specific guidance on the

MySqlInitialSyncStateIterator

behavior. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.