Is this your first time deploying Airbyte: no OS Version / Instance: Linux EC2 m5.4xlarge Deployment: Docker Airbyte Version: 0.35.50-alpha Source: Survey Monkey 0.1.7 Destination: Snowflake 0.4.17 Description: Incremental + Dedupe results in duplicates rows for the responses stream

loading...

Hey what are the primary keys you are using?

It's preconfigured in the survey monkey connector

This is the only record duplicated?

No. There were a handful. And this was the first subsequent sync. I imagine if I do it again. It will happen again

are there related tests on this connector for the incremental responses stream?

should have a test validating that a incremental sync or full refresh produce the correct output, maybe not spcific to duplicate records

okay. so next steps?

Yeah as I said I could see that the primary key is

id

you can find this here https://github.com/airbytehq/airbyte/blob/eeb35872c2d30e348709589fd56480c3f9b513f0[…]s/connectors/source-surveymonkey/source_surveymonkey/streams.py

cheers, i was looking for a link to where this is defined. confirming the

id

is the same in the duplicated rows. however i do not see

start_modified_at

in the full response record

yeah it doesn't. Basic normalisation takes care of the normalisation

okay so that’s the issue here? i have no normalisation process (only

_AIRBYTE_RAW

tables) and thus the deduplication process (in incremental + dedupe) will not work. however, as for incremental. this duplicate record should still not appear as it is the same data i.e. only records after

date_modified

should be inserted. correct?

Yeah what you said is what ideally should happen. I will take a look at it today

Sure will do it and ping back here

any luck?

Hey I went through the code

params = super().request_params(stream_state=stream_state, **kwargs)
        params["sort_order"] = "ASC"
        params["sort_by"] = "date_modified"
        params["per_page"] = 1000  # maybe as user input or bigger value
        since_value = pendulum.parse(stream_state.get(self.cursor_field)) if stream_state.get(self.cursor_field) else self._start_date

        since_value = max(since_value, self._start_date)
        params["start_modified_at"] = since_value.strftime("%Y-%m-%dT%H:%M:%S")
        return params

It doesn't look off for me. Can you help in understanding if there is pattern for duplicate records like it's just happening in the borders?

i just created a seperate ticket as there appears to be a broader issue occurring, possibly to do with the state of the job in the DB see new issue here

Hey as of now the credentials we have don't have access to responses thus it would be great if you can help me with info on the duplicates like Is it just happening on the start or end dates or any pattern in them ?

Aren't responses the main piece of data for the survey monkey connector? that's like having a GitHub connector with no commits or PRs, Google Ads with no actual visits, databases with empty tables. I would think having responses is fundamental to the airbyte test connector. I'll try investigate on my end however this is tricky as I'm hitting our daily API limits without being able to properly test.

Checking it

@Harshith (Airbyte) @Harshith (Airbyte) i think i may have found the issue https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-surveymonkey/source_surveymonkey/streams.py#L263-L275 can you confirm this API pulls responses after this timestamp. and not on or after this timestamp? i.e. are responses being pulled

WHERE timestamp > cursor_field

or is it being pulled as

WHERE timestamp >= cursor_field

??