Hello! I'm migrating from Fivetran to Airbyte to ingest data from Shopify to BigQuery. After implemented, I noticed that Airbyte is not bringing all data that Fivetran brings. The inventory_items, product_variant and order_line tables seems to come incomplete. For example, the Fivetran's inventory_items table has 183 lines, while Airbyte's has 11. May somebody help me out with it please? Thank you very much.

Hi! First of all, welcome 🙂 Second - I wonder what may be different between the two connections, could you help me brainstorm? Perhaps the timeframe window is different or something like that? Also - could you please attach the logs you're getting from the sync? Sometimes it helps to look through them in cases like this.

Hello! Thanks for answering. So, yes, the timeframe window is equal. Here goes the server logs. I looked them closely, but haven't found anything that would explain it.

Hey João, you sent the logs from Airbyte in general, but that doesn't includes the sync logs. Could you share those? Just go to your Shopify connection and expand it, on any attempt the should be a button to download the logs

Here's the logs from the very first sync.

Can you try recreating the connection or resetting the data? Something strange is going on with the state. State is where Airbyte stores the cursor, marking the latest record it read for each stream. In your log it is spamming a lot of state messages, and also there is some hash conflicts. I just want to be sure it is starting with no state at all.

That's a great point about state, Marcelo! I see this exception:

State hash 765392289 was already committed, likely indicating a state hash collision

Are you using the

updated_at

field as a cursor? Would you be able to do a full refresh sync as a baseline test? I think the append dedupe sync mode is doing something weird to the state

Hey! I've recreated the connection and it still has the same problem. Here are its logs:

And what about the cursor field? Also - I see the sync seems to be cancelled

"status" : "cancelled"

, do you see any raw tables? If so, do they contain all the data? So sorry this is causing you trouble!

I agree with Nataly and I think you should run a full refresh just to eliminate all these cursor troubles.

So, I've already refreshed it all but the error still appears.

I think I was not clear, sorry. On the connection, instead of selecting an incremental sync, you select a Full refresh one, this way Airbyte won't use any state and will try to load all data always, but that way we can debug this issue

Oh, ok. I'll do it and send you the logs

What is your configured start_date on the source?

2000-01-01

Yeah, don't know what is happening. I looked at the Shopify connector code and found nothing strange. @Nataly Merezhuk (Airbyte), do you have any more ideas?

@João Larrosa there should be a table with raw data in BigQuery called something like

_airbyte_raw_raw_shopify_inventory_items

could you take a look and see if all the data is in there? I'm also seeing this: It's not the 183 lines, but closer...

{
    "streamName" : "raw_shopify_inventory_items",
    "streamNamespace" : "raw_shopify_projetoraiz_shopify",
    "stats" : {
      "recordsEmitted" : 176,
      "bytesEmitted" : 70789,
      "recordsCommitted" : 176
    }
  },

Yes, there is

Could you update both Airbyte and the Shopify connector? I see it got an update recently and you're a version behind: https://docs.airbyte.com/integrations/sources/shopify#changelog If this doesn't help, I've found a GitHub issue that's already open: https://github.com/airbytehq/airbyte/issues/19782 So then I think the best course of action would be to include your logs there