Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

I am experiencing very slow performance on the freshdesk connector for airbyte while importing tickets data. Has anyone else faced the same? Is there a way to mitigate it? As an example, it took me over 5 hours to import 20000 tickets

image.png

Hi.

Based on the api docs - it`s probably an api limit.

<https://developers.freshdesk.com/api/#ratelimit>

<@U01MMSDJGC9> turned this thread into Zendesk ticket <https://airbyte7538.zendesk.com/agent/tickets/2606 | 2606> to ensure timely resolution!

Thanks for the question <@U03NKEF2LBW>. I think Eli has highlighted the issue, please create an issue on Github as it shouldn’t normally be hitting the API limits. (Perhaps they modified these limits recently?)

That's what my initial thought was, but I have the limit set as 500, but as you can see in the above screenshot it is taking progressively longer to import the next 1000 set of batches (taking upto 40 minutes) for a batch even. This doesn't look normal to me. I'll raise an issue for it if required

Is there an issue in GH that I can subscribe :bell: to?

<@U03NKEF2LBW>, did you ever raise an issue for this. If so, can you share the link here? If not, I can go ahead and create the issue if this has not been resolved yet.

I'm experiencing the same issue of incremental times for Airbyte 0.40.1 with latest Freshdesk connector (0.3.6).

Update: I just forked the repo and ran the connector. I'm getting a 429 with the following message: `"You have exceeded the limit of requests per hour"` . When looking at the URL I noticed that there was no `/v2/` in it. ~I'm guessing that's the issue but I'm still trying to catch where's happening~. I just found where to fix it, will upload a PR now.

<https://github.com/airbytehq/airbyte/pull/18397>

Thanks <@U02UCEVDDR9>, I’ll assign someone from our maintainer program to review the PR soon.

Thanks <@U02UCEVDDR9> for finding the issue :raised_hands: . <@U03J7N2M6BD> please let me know if I should still raise an issue

freshdesk_tickets_airbyte_logs.log

Seems like there is an issue where the data from the source isn’t stopping for ‘tickets’ entity. The integration shows completed but the sync fails.
PF the logs

<@U03J7N2M6BD> Please do look into <https://airbytehq.slack.com/archives/C021JANJ6TY/p1667989807127719?thread_ts=1665022330.815169&amp;cid=C021JANJ6TY|this> issue.

<@U0409B7B1C3> can you share the complete file log?

logs-784 (1).txt

From your logs:
```Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 137. This warning is normal if the job was cancelled.```
Exit code 137 means *a container or pod is trying to use more memory than it’s allowed*. The process gets terminated to prevent memory usage ballooning indefinitely, which could cause your host system to become unstable.
You must increase the memory of the connector container/pod