https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I have a mixpanel to bigquery conne...
# replication-ask-ai
s
@kapa.ai I have a mixpanel to bigquery connection and it is SO slow when it syncs. it often takes around 2 and a half hours to load about 200 records. what is going on and how can i improve it? attached is an example log file from the currently running sync job
k
Sorry @Stuart Horgan, I encountered an error processing the file(s) you uploaded: Text file is too large (max 60K characters)
s
@kapa.ai ok i wll paste the start of it
k
Sorry @Stuart Horgan, I encountered an error processing the file(s) you uploaded: File type is not supported
s
@kapa.ai ok i wll paste the start of it below:
Copy code
2026-10-02 09:21:04 platform INFO APPLY Stage: BUILD — (workloadId=fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync)
2026-10-02 09:21:04 platform INFO APPLY Stage: CLAIM — (workloadId=fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync)
2026-10-02 09:21:04 platform INFO Claimed: true for workload fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync via API in dataplane AUTO (be7b0151-d7b3-4855-8687-db7578ab8a5d)
2026-10-02 09:21:04 platform INFO APPLY Stage: LOAD_SHED — (workloadId=fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync)
2026-10-02 09:21:04 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync)
2026-10-02 09:21:04 platform INFO No pod found running for workload fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync
2026-10-02 09:21:04 platform INFO APPLY Stage: MUTEX — (workloadId=fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync)
2026-10-02 09:21:04 platform INFO Mutex key: fcc210e4-e35a-430e-b0af-5bfb151c4f00 specified for workload: fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync. Attempting to delete existing pods...
2026-10-02 09:21:04 platform INFO Mutex key: fcc210e4-e35a-430e-b0af-5bfb151c4f00 specified for workload: fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync found no existing pods. Continuing...
2026-10-02 09:21:04 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync)
2026-10-02 09:21:04 platform INFO APPLY Stage: LAUNCH — (workloadId=fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync)
2026-10-02 09:21:04 platform INFO [initContainer] image: airbyte/workload-init-container:2.0.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=512Mi, cpu=250m}, additionalProperties={})
2026-10-02 09:21:04 platform INFO Launching replication pod: replication-job-77559-attempt-0 (selectors = {}) with containers:
2026-10-02 09:21:04 platform INFO [source] image: airbyte/source-mixpanel:4.0.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=512Mi, cpu=250m}, additionalProperties={})
2026-10-02 09:21:04 platform INFO [destination] image: airbyte/destination-bigquery:2.0.25 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=250m}, additionalProperties={})
2026-10-02 09:21:04 platform INFO [orchestrator] image: airbyte/container-orchestrator:2.0.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=512Mi, cpu=250m}, additionalProperties={})
2026-10-02 09:21:30 platform INFO Attempting to update workload: fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync to LAUNCHED.
2026-10-02 09:21:30 platform INFO Pipeline completed for workload: fcc210e4-e35a-430e-b0af-5bfb151c4f00_77559_0_sync.
2026-10-02 09:21:56 replication-orchestrator INFO Running replication worker...
2026-10-02 09:21:56 replication-orchestrator INFO Starting replication worker. job id: 77559 attempt: 0
2026-10-02 09:21:56 replication-orchestrator INFO
2026-10-02 09:21:56 replication-orchestrator INFO ----- START REPLICATION -----
2026-10-02 09:21:56 replication-orchestrator INFO
2026-10-02 09:21:56 replication-orchestrator INFO Running destination...
2026-10-02 09:21:56 replication-orchestrator INFO Writing messages to protocol version 0.2.0
2026-10-02 09:21:56 replication-orchestrator INFO Reading messages from protocol version 0.2.0
2026-10-02 09:21:56 replication-orchestrator INFO Reading messages from protocol version 0.2.0
2026-10-02 09:21:56 replication-orchestrator INFO SourceReader started.
2026-10-02 09:21:56 replication-orchestrator INFO MessageProcessor started.
2026-10-02 09:21:56 replication-orchestrator INFO DestinationReader started.
2026-10-02 09:21:56 replication-orchestrator INFO Starting workload heartbeat (interval=10s; timeout=600s)
2026-10-02 09:21:56 replication-orchestrator INFO DestinationWriter started.
2026-10-02 09:21:56 replication-orchestrator INFO Transitioning workload to running state
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:54 [32mINFO[m i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {catalog=/dest/catalog.json, write=null, config=/dest/connectorConfig.json}
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:54 [32mINFO[m i.a.c.i.b.IntegrationRunner(runInternal):132 - Running integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:54 [32mINFO[m i.a.c.i.b.IntegrationRunner(runInternal):133 - Command: WRITE
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:54 [32mINFO[m i.a.c.i.b.IntegrationRunner(runInternal):134 - Integration config: IntegrationConfig{command=WRITE, configPath='/dest/connectorConfig.json', catalogPath='/dest/catalog.json', statePath='null'}
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword always_show - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.d.b.BigQueryUtils(getLoadingMethod):415 - Selected loading method is set to: GCS
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.b.d.t.CatalogParser(toStreamConfig):110 - Detected column name collision for source_mixpanel_web.export.url
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.b.d.t.CatalogParser(toStreamConfig):110 - Detected column name collision for source_mixpanel_web.export.utm_term
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.b.d.t.CatalogParser(toStreamConfig):110 - Detected column name collision for source_mixpanel_web.export.loan_term
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.b.d.t.CatalogParser(toStreamConfig):110 - Detected column name collision for source_mixpanel_web.export.utm_medium
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.b.d.t.CatalogParser(toStreamConfig):110 - Detected column name collision for source_mixpanel_web.export.utm_source
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.b.d.t.CatalogParser(toStreamConfig):110 - Detected column name collision for source_mixpanel_web.export.loan_amount
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.b.d.t.CatalogParser(toStreamConfig):110 - Detected column name collision for source_mixpanel_web.export.experiment_name
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.c.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"format_type":"CSV","flattening":"No flattening"}
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:55 [32mINFO[m i.a.i.d.b.BigQueryUtils(isKeepFilesInGcs):431 - All tmp files will be removed from GCS when replication is finished
2026-10-02 09:21:56 destination INFO Malformed non-Airbyte record (connectionId = fcc210e4-e35a-430e-b0af-5bfb151c4f00): 2026-10-02 08:21:56 [32mINFO[m i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$createWriteConfigs$1):108 - BigQuery write config: BigQueryWriteConfig[streamName=cohorts, namespace=source_mixpanel_web, datasetId=airbyte_internal, datasetLocation=europe-west2, tmpTableId=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=_airbyte_tmp_stq_cohorts}}, targetTableId=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=source_mixpanel_web_raw__stream_cohorts}}, tableSchema=Schema{fields=[Field{name=_airbyte_raw_id, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null}, Field{name=_airbyte_extracted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null}, Field{name=_airbyte_loaded_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null},
k
Thinking completed Based on the logs and the Airbyte documentation for Mixpanel, the long sync times are almost certainly due to Mixpanel API rate limits and how the source streams—particularly `Export`—are configured [Performance considerations]. Why is it so slow? 1. Mixpanel API Rate Limits: Mixpanel imposes strict rate limits depending on the endpoint [Performance considerations]:Because Airbyte breaks date ranges into slices, querying even a modest date range can quickly exhaust the 60-queries-per-hour limit, forcing the connector to wait/back off for extended periods [Performance considerations]. ◦ Raw Data Export API (
Export
stream): 100 concurrent queries, but limited to 60 queries per hour and 3 queries per second [Performance considerations]. ◦ Query API (
Cohorts
, 
Engage
, 
Funnels
, etc.): Maximum of 5 concurrent queries and 60 queries per hour [Performance considerations]. 1. Large Date Windows or Small Slices: If your source configuration has a wide date range (e.g., from a long time ago up to the present) and small date slices, it makes dozens of API calls to check for data day-by-day—even if there are only ~200 records in total [Performance considerations]. 2. Schema Generation Overheads: The 
Export
stream uses dynamic schemas fetched via API calls to determine custom event properties, which adds extra calls against your rate limits before/during data extraction [streams.py; Forum Post: Source MixPanel - Syncs hanging / stuck for hours without any error messages]. How to improve performance 1. Tune Date Slicing & Threads in Source Settings: In your Mixpanel Source settings: ◦ Increase the Date slicing window: Setting a larger window reduces the total number of API requests required to cover the date range [Performance considerations]. ◦ Adjust the Number of concurrent threads: Balance this with your Mixpanel plan’s concurrency limits to avoid throttling [Performance considerations]. 1. Switch to Incremental Syncs: Ensure you are using Incremental sync modes rather than Full Refresh where possible [Supported sync modes]. Once the initial historical sync completes, subsequent syncs will only query the newest date range, greatly reducing the number of requests against the hourly rate limit. 2. Adjust the Start Date: If you do not need all historical data, advance the Start Date in your Mixpanel source settings to only sync the timeframe you actually care about. 3. Deselect Unused Streams: Disable any streams you do not need (such as 
Engage
, 
Cohorts
, or 
Annotations
), allowing the connector to spend its hourly query quota exclusively on the data you require [Supported sync modes; Performance considerations]. (Note: The log lines mentioning 
Malformed non-Airbyte record
are standard stdout/INFO logs emitted by the connector runner that are not JSON-formatted messages, and do not indicate a replication failure.) 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai we are using incremental mode already. This is what the state looks like:
Copy code
[
  {
    "streamDescriptor": {
      "name": "engage"
    },
    "streamState": {
      "last_seen": "2026-10-02T09:25:52"
    }
  },
  {
    "streamDescriptor": {
      "name": "cohorts"
    },
    "streamState": {
      "created": "2026-09-22 13:25:40"
    }
  },
  {
    "streamDescriptor": {
      "name": "cohort_members"
    },
    "streamState": {
      "state": {
        "last_seen": "2026-10-02T07:09:59"
      },
      "states": [
        {
          "cursor": {
            "last_seen": "2026-09-30T16:54:57"
          },
          "partition": {
            "id": 5600829,
            "parent_slice": {}
          }
        },
        {
          "cursor": {
            "last_seen": "2026-10-02T06:28:26"
          },
          "partition": {
            "id": 5600823,
            "parent_slice": {}
          }
        },
        {
          "cursor": {
            "last_seen": "2026-10-01T19:27:13"
          },
          "partition": {
            "id": 5785825,
            "parent_slice": {}
          }
        },
        {
          "cursor": {
            "last_seen": "2026-09-30T09:17:38"
          },
          "partition": {
            "id": 5785831,
            "parent_slice": {}
          }
        },
        {
          "cursor": {
            "last_seen": "2026-10-01T19:27:13"
          },
          "partition": {
            "id": 5784652,
            "parent_slice": {}
          }
        },
        {
          "cursor": {
            "last_seen": "2026-10-02T06:15:26"
          },
          "partition": {
            "id": 5881521,
            "parent_slice": {}
          }
        },
        {
          "cursor": {
            "last_seen": "2026-10-02T09:23:59"
          },
          "partition": {
            "id": 6078635,
            "parent_slice": {}
          }
        },
        {
          "cursor": {
            "last_seen": "2026-10-02T06:14:13"
          },
          "partition": {
            "id": 6180401,
            "parent_slice": {}
          }
        },
        {
          "cursor": {
            "last_seen": "2026-10-02T07:09:59"
          },
          "partition": {
            "id": 6180400,
            "parent_slice": {}
          }
        }
      ],
      "parent_state": {},
      "lookback_window": 3180,
      "use_global_cursor": false
    }
  },
  {
    "streamDescriptor": {
      "name": "funnels"
    },
    "streamState": {
      "states": [
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "76992311"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "77417047"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "77417091"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "77418907"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "77419169"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "78627568"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "78698540"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "78785353"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "78791875"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "79425545"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "79851197"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "79851369"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "79851439"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "79851603"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "79851687"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "79852093"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "79984271"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "80740580"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "81538516"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "81538791"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "81788793"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "83926727"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "84502660"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "84506434"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "84969797"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85101987"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85115706"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85116626"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85358008"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85389561"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85510852"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85510853"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85510854"
          }
        },
        {
          "cursor": {
            "date": "2025-10-27"
          },
          "partition": {
            "funnel_id": "85510855"
          }
        }
      ]
    }
  },
  {
    "streamDescriptor": {
      "name": "export"
    },
    "streamState": {
      "time": "2026-10-01T23:56:53Z"
    }
  }
]
The date slicing window is set to 30, that would cover those saved states in a single slice wouldn't it? We have it set to 3 concurrent threads at the moment. would it be more helpful to reduce or increase this?
k
Thinking completed Looking closely at your state, the main culprit for the 2.5-hour sync time is the 
funnels
stream, not the 
export
stream. 1. Does a 30-day window cover the states in a single slice? • For 
export
and `engage`: Yes. Their state timestamps are from late yesterday and today (e.g., 
2026-10-01
), so a 30-day window covers the remaining range up to today in just 1 slice. • For `funnels`: No, absolutely not. ◦ Every single one of your 34 funnel partitions has a saved cursor date of 
2025-10-27
, which is nearly a full year behind today (
2026-10-02
). ◦ A 30-day window divided over ~340 days means each funnel requires roughly 11 to 12 date slices. ◦ Multiplied across 34 funnels, that is roughly 370–400 API requests just for the 
funnels
stream. ◦ The Query API (which powers 
funnels
) is strictly capped by Mixpanel at 60 queries per hour [Performance considerations]. In fact, the connector enforces a mandatory sleep after requests to respect this: 
time.sleep(3600 / 60)
= 60 seconds per request [streams.py]. ◦ Sleeping ~60 seconds for ~150–200 requests easily totals 2.5+ hours, which explains your sync duration. 2. Should you reduce or increase concurrent threads? Do not increase threads. • The Query API (used by 
funnels
, 
cohorts
, 
engage
, and 
cohort_members
) has a hard Mixpanel limit of 5 concurrent queries and 60 queries per hour [Performance considerations]. • With 3 concurrent threads, you are already close to the 5 concurrent query maximum. Increasing threads will not bypass the global 60-requests-per-hour limit and will likely trigger 
429 Too Many Requests (Query rate limit exceeded)
errors [Forum Post: Mixpanel source connector error with cohorts endpoint limit exceeded]. • Keeping it at 2–3 threads is safe. How to fix it 1. Disable 
funnels
if not needed: If you only care about events and user data (
export
/ 
engage
), disable the 
funnels
stream in your connection schema. Your sync will drop from 2.5 hours to just a few minutes. 2. Increase the Date Slicing Window: If you do need 
funnels
, you can increase Date slicing window significantly (e.g., from 30 days to 90 or 180 days) [Performance considerations]. This cuts the required number of slices and requests proportionally. 3. Bring the 
funnels
State Forward: If you do not need historical funnels data from 2025, you can reset or update your stream state / start date so it does not backfill 11 months of funnels data across 34 partitions. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai funnels is already disabled so this is not the culprit
k
Thinking completed If 
funnels
is disabled, the primary reason for the extended sync time is the 
cohort_members
 stream (and how the Mixpanel Query API throttles it) combined with the lookback window logic. 1. cohort_members queries every single cohort and does not skip records Even though 
cohort_members
has a saved state cursor in incremental syncs, members in cohorts can change at any time. As noted in the connector codebase, the stream always checks every parent cohort partition to fetch updated cohort members [test_streams.py]: 1. The connector first queries the 
cohorts
endpoint (
/api/2.0/cohorts/list
) to retrieve all available cohort IDs [manifest.yaml; test_streams.py]. 2. For every individual cohort discovered, it routes through the 
CohortMembersSubstreamPartitionRouter
and executes a separate 
POST
request to 
query/engage
with 
filter_by_cohort
[manifest.yaml]. 3. If any cohort contains paginated user records, each page requires an additional API call [manifest.yaml]. Because 
cohort_members
uses Mixpanel's Query API (
query/engage
), it falls under the strict rate limit policy: • The declarative connector budget enforces 
limit: 1
request per 60 seconds (
PT60S
), mirroring Mixpanel's Query API limit of 60 queries per hour [manifest.yaml; CONTRIBUTING.md]. • Even if your Mixpanel project only has 50–100 cohorts or pages, querying them through the 1-request-per-minute throttle automatically takes 1 to 2+ hours—even if almost no new records are returned [manifest.yaml; CONTRIBUTING.md]. 2. The export Stream's Built-in Sleep and Lookback Window Even if 
funnels
and 
cohort_members
are excluded: • 60-Second Sleep Per Request: In the Python implementation for the 
export
stream, after every HTTP response, the stream sleeps for 
3600 / reqs_per_hour_limit
(60 seconds) to honor the 60 queries/hour limit [CONTRIBUTING.md]. • Lookback Windows: In incremental mode, the export stream automatically moves the query starting date back by 
export_lookback_window
(default is often 60 seconds or up to several days depending on 
attribution_window
) [test_streams.py; streams.py]. • Dynamic Schema Discovery: Before exporting data, 
ExportSchema
queries 
engage/properties
and 
events/properties/top
to determine the dynamic schema [CONTRIBUTING.md; test_streams.py]. How to test and resolve this 1. Temporarily Disable 
cohort_members
and `cohorts`: If you only need raw event/action data (
export
) or user profiles (
engage
), disable 
cohort_members
in your connection settings [Supported sync modes]. Check if the sync time immediately drops. 2. Leave Concurrency as-is (or keep at 2–3): Increasing concurrency does not bypass the 60-requests-per-hour limit on the Query API and risks triggering 429 errors [Performance considerations; CONTRIBUTING.md]. 3. Inspect the Running Log: During the sync, search the connector logs for which requests are actively executing. If you see repeated calls to 
query/engage
with 
filter_by_cohort
or sleeps between 
export
calls, that confirms which stream is waiting on the 60-second-per-request rate limit [manifest.yaml; CONTRIBUTING.md]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Open in Slack
PreviousNext
Powered by