UPDATE: I found the reason... Connection has two failed attempts and each attempt pulled some rows. Now I have very important question: Data will be duplicated or not? ORIGINAL MESSAGE: Guys I need help, Airbyte shows incorrect number of emitted and committed records. On screens you can see 515 on Airbyte side, which in fact gives 1034 rows on bigquery. Table was empty. anyway I checked my and Airbyte processing dates:

-- 2023-01-02 19:10:20.191 <- service call/processing start datetime
-- 2023-01-02 19:45:03.546000 UTC <- airbyte min emitted datetime
-- 2023-01-02 21:00:10.844000 UTC <- airbyte max emitted datetime
-- 2023-01-02 21:05:01.717 <- service call/processing end datetime

Hi Zaza! Thanks for your patience - I was out last week for the holidays and am catching up with messages now. Usually when records are emitted but the table is empty there is a normalization problem. Are you using normalization?

@Nataly Merezhuk (Airbyte) Nope, I am just pulling raw data. I am not using normalization. Down you can find my last message:

I found the reason...
Connection has two failed attempts and each attempt pulled some rows.
Now I have very important question:
Data will be duplicated or not?

I still have no answer about duplicates.

I see. Please make sure to review our code of conduct - posting follow up comments like this goes against it: Rule 8: Use threads for discussion. The simplest way to keep conversations on track in Slack is to use threads. The Airbyte Slack community relies heavily on threads, and if you break from this convention, rest assured one of our community members will respectfully inform you quickly! The reason for the disparity in the records could be that there is something wrong with the initial data. Something is wrong with part of that data so Airbyte is not able to process it. I would advise you to look through some of the rows that have not been copied and look for type errors. This could also be in the logs.

@Nataly Merezhuk (Airbyte) Sorry again about that! I removed that message and update the current one. Rules are fine, but what about answer?

From my message above: The reason for the disparity in the records could be that there is something wrong with the initial data. Something is wrong with part of that data so Airbyte is not able to process it. I would advise you to look through some of the rows that have not been copied and look for type errors. This could also be in the logs. Were you able to look at the data that has not been copied? Are there any type errors or other discrepancies in it? Could you provide the logs?

@Nataly Merezhuk (Airbyte) Sorry, looks like we have some misunderstanding. My question is more general. I am not looking for an exact error reason. Actually I know the reason for the error:

You exceeded your quota for the requested resource.

This error is from ASP service. I just care about different stuff and my question sounds like: If I have a connection, which before successfully complete made 3 attempts. The First 2 failed, but each pulled X amount of rows and the third attempt completed successfully, if I take all those records, data will be duplicated there or not. I am asking this because I need to understand, do I need to ignore/remove records pulled on time failed attempts or not? Please help me clearly understand this case, because this is a very important question in order to understand the logic of Airbyre, at the same time I am contributing to it 🙂

I see! Sorry for misunderstanding your question 🙂 Currently if a sync fails, the records/data that's been duplicated doesn't get cleaned up. We are working on it, but that hasn't been implemented yet. You can delete these records. This will not cause duplicate records when a sync completes successfully! Does that answer your question?

@Nataly Merezhuk (Airbyte) Thank you!

No problem!