https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Does airbyte have a gorgias connect...
# replication-ask-ai
k
@kapa.ai Does airbyte have a gorgias connector, cause the one in the marketplace does not seem to sync incrementally, so can we create one using airbyte builder as a custom connector that works incrementally cause it is time consuming to pull millions of records everyday
k
Good news on both fronts! Airbyte Does Have a Gorgias Connector Airbyte already has a native Gorgias connector in the marketplace. According to the documentation, it does support incremental sync for most of its streams, including tickets, messages, customers, events, and more. [Gorgias connector] Here's a summary of incremental support per stream: There was also a changelog entry from 2024-11-06 that specifically mentions a fix for incremental sync format, so if you're on an older version, that could explain the issue you're experiencing. [Gorgias changelog] If You Still Want to Build a Custom One Yes, you can absolutely build a custom connector using the Connector Builder that supports incremental sync. The Connector Builder supports datetime-based incremental syncs by: ā€¢ Setting a cursor field (e.g., 
updated_at
) ā€¢ Configuring start/end datetime injection into the API request ā€¢ Automatically saving state so the next sync picks up from where it left off For APIs that accept date/time filters in the request body (which Gorgias's API may use), you can use 
stream_interval
variables like this:
Copy code
{
  "filter": {
    "fromDateTime": "{{ stream_interval.start_time }}",
    "toDateTime": "{{ stream_interval.end_time }}"
  }
}
[incremental sync docs] My recommendation: First, try updating the existing Gorgias connector to the latest version (0.0.4 as of 2024-12-11), as the incremental sync fix was applied in version 0.0.3. If it still doesn't work as expected, you can open the existing connector in the Connector Builder (since it's a manifest-only connector), edit it, and republish it to your workspace. [Gorgias README] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
k
The marketplace connector does support incremental, but it does not work. Can you check why...? Whenever it sync, the connector pulls in whole of the data from start, and not only the new data... @kapa.ai
k
Based on the knowledge sources available to me, I can identify some common reasons why an incremental sync might pull all data on every sync instead of only new records. While I don't have specific debugging information about the Gorgias connector's current behavior, here are the known root causes that apply generally: Common Reasons Incremental Sync Fetches All Records 1. Connection State is Empty If the connection state is not being saved or read correctly, the connector has no cursor value to reference and defaults to the configured 
start_date
, pulling everything from scratch. This is a known issue reported by users. [incremental sync issue] 2. Cursor Value Not Being Injected Correctly A very common bug is when the 
start_time_option
is configured to inject into 
request_parameter
(query params), but the API endpoint uses a POST request ā€” in which case the API silently ignores query parameters. This exact bug was confirmed in the 
source-gong
connector: > 
start_time_option
injects into 
request_parameter
ā€” but the endpoint is a POST request. The API ignores query parameters on POST endpoints, so the cursor value is silently discarded. [Gong bug report] If Gorgias's API also uses POST endpoints, this could be the same class of bug. 3. Hardcoded start_date in Request Body Another common bug is when 
request_body_json
has a filter like 
fromDateTime
hardcoded to 
config['start_date']
 instead of using the cursor state. This means every sync always starts from the user-configured start date, regardless of what was previously synced. [Gong bug report] What You Can Do 1. Check the connector version ā€” make sure you're on the latest version of the Gorgias connector (0.0.4 as of 2024-12-11), since incremental sync fixes were applied in 0.0.3. 2. Inspect the connection state ā€” after a sync, check if the connection state is populated with a cursor value. If it's empty, the state is not being saved. 3. Open a GitHub issue ā€” if the connector is still broken on the latest version, I'd recommend opening a bug report on the Airbyte GitHub repository with your connector version and observed behavior. 4. Build a custom connector ā€” as a workaround, you can fork the Gorgias connector in the Connector Builder, fix the 
start_time_option
injection (ensuring it targets 
body_json
if the API uses POST), and publish it as a custom connector to your workspace. [incremental sync docs] The knowledge sources don't contain the specific Gorgias connector YAML manifest, so I can't pinpoint the exact line causing the issue, but the patterns above are the most likely culprits based on similar reported bugs. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
k
@kapa.ai How to check point number 2..
k
The user is asking how to check whether the connection state is empty (point #2 from the previous answer ā€” checking if the cursor value is being saved after a sync). Here's how to do it: How to Check the Connection State in Airbyte 1. In the Airbyte UI, click Connections and select the Gorgias connection. 2. Click the Settings tab on the Connection page. 3. Click the Advanced dropdown arrow. The Connection State will be displayed there. [manage connection state] What to Look For ā€¢ If the state is empty 
{}
 ā€” the connector is not saving the cursor after each sync, which means every sync starts from the configured 
start_date
and pulls all data. ā€¢ If the state has a cursor value, for example:{ "tickets": { "updated_at": "2024-06-01T000000Z" } }Then the state is being saved correctly, and the issue lies elsewhere (e.g., the cursor value is not being injected into the API request properly). This check will tell you whether the problem is with state not being saved or with state not being used when making API requests. [manage connection state] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
k
{ "streamDescriptor": { "name": "tickets" }, "streamState": { "updated_datetime": "2026-04-19T000447.393761+0000" } It is like this... Still whenever the sync starts, there are 300000 records loaded, whereas there might be only 1 to 2k tickets daily @kapa.ai
k
The connection state is being saved correctly ā€” the cursor value 
updated_datetime: 2026-04-19T00:04:47.393761+0000
is there. This means the problem is almost certainly point #2 from the earlier analysis: the cursor value is saved but not being injected correctly into the API request. Based on the patterns documented in the knowledge sources, the most likely cause is one of these two bugs: Most Likely Culprits 1. 
start_time_option
injecting into query params on a POST endpoint If the Gorgias API endpoint for tickets uses a POST request, and the connector is configured to inject the cursor as a 
request_parameter
(query string), the API will silently ignore it and return all records. This exact bug was confirmed in the Gong connector: > "`start_time_option` injects into 
request_parameter
ā€” but the endpoint is a POST request. The API ignores query parameters on POST endpoints, so the cursor value is silently discarded." [Gong bug report] 2. 
fromDateTime
hardcoded to 
config['start_date']
in the request body If the request body has the filter date hardcoded to the user-configured 
start_date
instead of using the cursor state, every sync will always start from the beginning. [Gong bug report] What to Do Next Since the state is confirmed to be populated, I'd recommend: 1. Open a GitHub issue on the Airbyte repository specifically for the Gorgias connector, referencing the state value and the behavior (300k records pulled despite a recent cursor). This is the most direct path to a fix. 2. As a workaround, open the Gorgias connector in the Connector Builder, inspect the 
incremental_sync
block and how 
start_time_option
is configured (specifically whether 
inject_into
is set to 
request_parameter
vs 
body_json
), fix it, and publish it as a custom connector to your workspace. [incremental sync docs] The knowledge sources don't contain the Gorgias connector's YAML manifest directly, so I can't confirm the exact misconfiguration, but the pattern strongly matches the documented bugs above. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
k
If I have to builld a custom gorgias connector for incremental sync of tickets, can you get me the yaml that would do that perfectly...? @kapa.ai
k
I appreciate the request, but I have to be transparent: the knowledge sources I have access to do not contain the Gorgias connector's YAML manifest, nor do they contain the specific Gorgias Tickets API endpoint details (URL, request body structure, filter parameters, etc.) needed to write a fully accurate and working YAML. Writing a YAML without that information would require speculation, which could result in a broken connector. What I Can Offer Based on the documented 
DatetimeBasedCursor
pattern from the low-code CDK docs, here is the structural skeleton for an incremental tickets stream. You would need to fill in the Gorgias-specific API details:
Copy code
version: "0.1.0"

definitions:
  base_requester:
    type: HttpRequester
    url_base: "https://{{ config['domain_name'] }}.gorgias.com/api"
    authenticator:
      type: BasicHttpAuthenticator
      username: "{{ config['username'] }}"
      password: "{{ config['password'] }}"

  tickets_stream:
    type: DeclarativeStream
    name: tickets
    primary_key: id
    retriever:
      type: SimpleRetriever
      requester:
        $ref: "#/definitions/base_requester"
        path: /tickets  # verify exact path from Gorgias API docs
        http_method: GET
        request_parameters:
          # Verify the exact filter param names from Gorgias API docs
          updated_datetime_gte: "{{ stream_interval['start_time'] }}"
      paginator:
        type: DefaultPaginator
        # configure based on Gorgias pagination style
      record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path: ["data"]  # verify the response wrapper key
    incremental_sync:
      type: DatetimeBasedCursor
      cursor_field: updated_datetime
      datetime_format: "%Y-%m-%dT%H:%M:%S.%f%z"
      start_datetime:
        type: MinMaxDatetime
        datetime: "{{ config['start_date'] }}"
        datetime_format: "%Y-%m-%dT%H:%M:%S.%f%z"
      start_time_option:
        type: RequestOption
        inject_into: request_parameter  # change to body_json if POST endpoint
        field_name: updated_datetime_gte  # verify exact param name
      step: "P1D"
      cursor_granularity: "PT1S"

streams:
  - $ref: "#/definitions/tickets_stream"

check:
  type: CheckStream
  stream_names:
    - tickets

spec:
  type: Spec
  connection_specification:
    type: object
    required:
      - domain_name
      - username
      - password
      - start_date
    properties:
      domain_name:
        type: string
      username:
        type: string
      password:
        type: string
        airbyte_secret: true
      start_date:
        type: string
Critical Things to Verify from Gorgias API Docs 1. Exact API path for tickets 2. Filter parameter name for filtering by 
updated_datetime
(e.g., 
updated_datetime_gte
) 3. GET vs POST ā€” if POST, change 
inject_into
from 
request_parameter
to 
body_json
4. Response wrapper key (e.g., 
data
, 
results
, etc.) 5. Pagination style (cursor-based, offset, etc.) The safest approach is to open the existing Gorgias connector in Connector Builder, inspect its current configuration, and fix only the 
start_time_option
injection issue identified earlier. [Gorgias README] [incremental syncs] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
4 Views
Open in Slack
PreviousNext
Powered by