Hi Team I am trying to load the data from source as postgres to destination as snowflake, I am loading one table whose number of records is 185940496, After running the pipeline in destination I am getting only 5734317 there. Help me with this how can I load the data into chunks successfully. Right now I am getting some error and it is terminating

Hi Abdul, to help you debug this I would need to see the full logs with the exact error in them. Please provide those if you can!

sure

My first thought is that you've potentially got a type error in your data:

ERROR[m c.n.s.DateTimeValidator(tryParse):82 - Invalid date-time: No timezone information: 2022-12-13T11:10:05.692195

I also see this:

PgConnection@7004e3d marked as broken because of SQLSTATE(08006), ErrorCode(0)

Other users have run into it, but using different conenctors: https://discuss.airbyte.io/t/hikaripool-2-connection-org-postgresql-jdbc-pgconnection-[…]marked-as-broken-because-of-sqlstate-08006-errorcode-0/3366 Could you try running a smaller sync as a test and see if it errors out or is successful?

Yeah smaller sync was successful

Then I think it must be a type error. I'm not sure what the best way to do this is, but you need to identify where it is. Maybe a series of smaller syncs would do it!

Got it

Please take a look at the scaling document: https://docs.airbyte.com/operator-guides/scaling-airbyte

And I am using open source airbyte on my machine

Take a look at that doc, it outlines how resources are used and their limitations!