Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hi!
We are trying to use the <https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py|Slack source> to fetch specific slack channels to big query but we are having trouble it.
It seems like some of the messages that exist in threads are not being synced.
We looked at the airbyte code and the Slack's API and it seems correct. You indeed call `conversations.replies` with the timestamp of the slack thread. It seems to be done <https://github.com/airbytehq/airbyte/blob/d05755523007499c84028f6c37c80feefcf81bdf/airbyte-integrations/connectors/source-slack/source_slack/source.py#L260|here>.
The thing is, that when we call the API directly, it seems to give the correct results, so we are not sure what's wrong with Airbyte's code exactly..
Did anyone else see this weird behavior of missing messages while trying to integrate to Slack?

what kind of sync mode are you using for that specific stream?

Moreover, are you "normalizing" data or using "raw data"?

We are normalizing the data using BigQuery (denormalized typed struct)

We are using Incremental sync for that specific stream (threads)

I would suggest you to try "raw data", and then explore the JSON by your own. If the data you look for are there, it means that the normalisation process is not optimal. For this reason,  I always use "raw" data and I parse data with by my own  using DBT.

Another potential source of issue is the incremental process not setting properly the state. It is happening to me with the mailchimp connector, so instead of the incremental sync I use the full refresh overwrite one.

Thanks :slightly_smiling_face:
Will update here about the results.

yes please, it would be great to know your discoveries

<@U03QTTH55HQ> thanks, indeed the issue was that the incremental process didn't work well.

Everything was synced correctly with full refresh