https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I'm having some issues with multipl...
# replication-ask-ai
f
@kapa.ai I'm having some issues with multiple connectors that have incremental replication, where the connection state gets stuck in the start_date and it replicates data for multiple sources on every hour. Help me fix it. What information do you need from the connectors and source yaml files to diagnose? I'm using airbyte cloud
k
Diagnosing Stuck Incremental Sync State on Airbyte Cloud This is a known class of issues with incremental connectors. To help diagnose, here's what's needed and what to check: Information Needed for Diagnosis 1. Connector details (for each affected connector): • Connector name and version • Which streams are affected • Your sync schedule (you mentioned hourly) 2. Connection state: • The current connection state JSON for each affected connection (visible in Connections → Settings → Advanced → Connection State) • This will show whether the cursor is truly stuck at 
start_date
or at some other timestamp 3. Source YAML (if using Connector Builder): • The full YAML manifest, especially: ◦ 
incremental_sync
block (
cursor_field
, 
datetime_format
, 
cursor_granularity
, 
step
) ◦ 
cursor_datetime_formats
◦ How the cursor field is populated (e.g., via 
AddFields
transformation?) ◦ Paginator type (
OffsetIncrement
, etc.) 4. Relevant log output from a recent failed/stuck sync Common Root Causes (from known issues) Immediate Steps You Can Take on Airbyte Cloud 1. Review the connection state for each affected connection: ◦ Go to Connections → [your connection] → Settings → Advanced → Connection State ◦ Check if the cursor value is literally your 
start_date
or a recent timestamp 1. Manually edit the connection state to a more recent date if it's stuck: ◦ Click anywhere in the Connection State field to edit it ◦ Update the cursor value to a recent timestamp ◦ Click "Update state" ◦ ⚠️ Do this with care — incorrect edits can break syncs [Managing state] 1. Check connector versions — if you're using a managed connector (Intercom, Datadog, etc.), verify you're on the latest version, as cursor bugs are often fixed in patches. What to Share for Further Help Please share: • The connector names and versions • The connection state JSON for 1–2 affected connectors • The source YAML if using Connector Builder • Any relevant log snippets showing the date range being requested This will allow pinpointing whether it's a 
cursor_granularity
mismatch, a hardcoded 
start_date
in the request, a transformation issue, or a connector-specific bug. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
f
@kapa.ai let's do it for the first connector: # Connector 1 Connection name: Respira - AirGradient → respira-do-postgres-dev Connection state JSON: [ { "streamDescriptor": { "name": "measures_past" }, "streamState": { "states": [ { "cursor": { "timestamp": "2026-05-01T000000Z" }, "partition": { "locationId": "192586" } }, { "cursor": { "timestamp": "2026-05-01T000000Z" }, "partition": { "locationId": "194860" } }, { "cursor": { "timestamp": "2026-05-01T000000Z" }, "partition": { "locationId": "191357" } }, { "cursor": { "timestamp": "2026-05-01T000000Z" }, "partition": { "locationId": "192812" } }, { "cursor": { "timestamp": "2026-05-01T000000Z" }, "partition": { "locationId": "196465" } }, { "cursor": { "timestamp": "2026-05-01T000000Z" }, "partition": { "locationId": "191359" } } ], "lookback_window": 0, "use_global_cursor": false } }, { "streamDescriptor": { "name": "measures_current_list" }, "streamState": { "__ab_no_cursor_state_message": true } } ] Source YAML: attached An interesting log warning: 2026-07-10 122650 source INFO Syncing stream: measures_current_list 2026-07-10 122650 source WARN Skipping cursor update for stream 'measures_past': failed to parse cursor field 'timestamp' value '2026-05-08T090500.000Z': No format in ['%Y-%m-%dT%H:%M:%SZ'] matching 2026-05-08T090500.000Z 2026-07-10 122650 source INFO Marking stream measures_past as RUNNING 2026-07-10 122650 replication-orchestrator INFO Stream status TRACE received of status: STARTED for stream measures_past Note: Active streams are only measures_past and measures_current_list
respira_air_gradient.yaml
k
Root Cause Identified The log warning tells us exactly what's wrong:
Copy code
Skipping cursor update for stream 'measures_past': failed to parse cursor field  'timestamp' value '2026-05-08T09:05:00.000Z': No format in ['%Y-%m-%dT%H:%M:%SZ']  matching 2026-05-08T09:05:00.000Z
The API is returning timestamps with milliseconds (e.g., 
2026-05-08T09:05:00.000Z
), but your 
cursor_datetime_formats
only includes the second-precision format 
%Y-%m-%dT%H:%M:%SZ
. Because the cursor value can't be parsed, the cursor is never updated — so every sync restarts from 
2026-05-01T00:00:00Z
. [Cursor datetime formats] The Fix You need to add 
%Y-%m-%dT%H:%M:%S.%fZ
(milliseconds format) to 
cursor_datetime_formats
in the 
measures_past
stream's 
incremental_sync
block. [DatetimeBasedCursor reference] Change this in your YAML:
Copy code
# BEFORE (measures_past incremental_sync) incremental_sync:   type: DatetimeBasedCursor   step: PT6H   cursor_field: timestamp   datetime_format: "%Y-%m-%dT%H:%M:%SZ"   cursor_granularity: PT1S   # ... (no cursor_datetime_formats defined)
To this:
Copy code
# AFTER (measures_past incremental_sync) incremental_sync:   type: DatetimeBasedCursor   step: PT6H   cursor_field: timestamp   datetime_format: "%Y-%m-%dT%H:%M:%SZ"   cursor_granularity: PT1S   cursor_datetime_formats:     - "%Y-%m-%dT%H:%M:%S.%fZ"   # milliseconds (what the API actually returns)     - "%Y-%m-%dT%H:%M:%SZ"      # seconds (fallback)   # ... rest of config unchanged
The 
cursor_datetime_formats
list is tried in order — the first matching format wins. Since the API returns millisecond-precision values, put that format first. [DatetimeBasedCursor docs] Also Check: measures_raw Stream Your 
measures_raw
stream has the same configuration pattern (same 
datetime_format
, no 
cursor_datetime_formats
). If that API endpoint also returns millisecond timestamps, it will have the same bug. Apply the same fix there proactively. After Fixing the YAML 1. Publish a new version of the connector in the Connector Builder 2. Update your connection to use the new version 3. Optionally, manually update the connection state to a more recent date (e.g., today minus a few hours) so you don't re-sync all data from 
2026-05-01
— the cursor is now 2+ months behind 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
f
@kapa.ai this is an example response from measures_past: are you sure the fallabck cursor dates are ok?
Copy code
{
    "locationId": 192586,
    "locationName": "Lambaré",
    "pm01": 4.6,
    "pm02": 9.1,
    "pm10": 9.2,
    "pm01_corrected": 4.6,
    "pm02_corrected": 5.5,
    "pm10_corrected": 9.2,
    "pm003Count": 513,
    "atmp": 22.3,
    "rhum": 58,
    "rco2": 430,
    "atmp_corrected": 21.2,
    "rhum_corrected": 81,
    "rco2_corrected": 430,
    "tvoc": 93.3,
    "wifi": -65,
    "timestamp": "2026-06-01T00:00:00.000Z",
    "serialno": "d83bda1c15c8",
    "model": "O-1PST",
    "firmwareVersion": "3.6.2",
    "tvocIndex": 99,
    "noxIndex": 2,
    "datapoints": "5"
k
Confirming the Cursor Format Yes, looking at the actual API response, the 
timestamp
field is:
Copy code
"timestamp": "2026-06-01T00:00:00.000Z"
This is always millisecond precision (
.000Z
), so the format 
%Y-%m-%dT%H:%M:%S.%fZ
will always match. The fallback 
%Y-%m-%dT%H:%M:%SZ
(no milliseconds) would never match this API's responses. The fix remains the same, but the fallback is technically unnecessary — though harmless to keep for safety:
Copy code
cursor_datetime_formats:   - "%Y-%m-%dT%H:%M:%S.%fZ"   # matches API response: 2026-06-01T00:00:00.000Z   - "%Y-%m-%dT%H:%M:%SZ"      # fallback, won't match this API but safe to keep
The 
datetime_format
(used for outgoing requests to the API) should stay as 
%Y-%m-%dT%H:%M:%SZ
— that controls how the `from`/`to` query parameters are sent, not how the cursor is parsed from responses. [Incremental sync docs] So the two formats serve different purposes: 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
f
@kapa.ai now for the next connector: # Connector 2 Connector name: MADES - Open Data -> respira-do-postgres-dev Connection state: [ { "streamDescriptor": { "name": "stations" }, "streamState": { "__ab_no_cursor_state_message": true } }, { "streamDescriptor": { "name": "pm10_data" }, "streamState": { "state": { "date_time": "2026-07-10" }, "states": [ { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 8, "parent_slice": {} } }, { "cursor": { "date_time": "2026-06-17" }, "partition": { "station_id": 9, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 10, "parent_slice": {} } }, { "cursor": { "date_time": "2026-03-10" }, "partition": { "station_id": 2, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 1, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 3, "parent_slice": {} } } ], "parent_state": {}, "lookback_window": 6, "use_global_cursor": false } }, { "streamDescriptor": { "name": "pm25_data" }, "streamState": { "state": { "date_time": "2026-07-10" }, "states": [ { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 8, "parent_slice": {} } }, { "cursor": { "date_time": "2026-06-17" }, "partition": { "station_id": 9, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 10, "parent_slice": {} } }, { "cursor": { "date_time": "2026-03-10" }, "partition": { "station_id": 2, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 1, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 3, "parent_slice": {} } } ], "parent_state": {}, "lookback_window": 5, "use_global_cursor": false } } ] Source YAML: attached Logs: attached, no errors or warnings I can see that relate to this issue
mades_open_data.yaml
k
Sorry @Fernanda Carles, I encountered an error processing the file(s) you uploaded: File type is not supported
f
@kapa.ai try again
k
Could you clarify what you'd like me to try again? For example: • Do you want me to re-examine the cursor format for the 
measures_raw
stream as well? • Do you want the full corrected YAML for 
measures_past
? • Is there something specific in my previous answer that seemed incorrect? Please let me know so I can give you the most accurate and helpful response! 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
f
@kapa.ai review this connector that has the same issue: # Connector 2 Connector name: MADES - Open Data -> respira-do-postgres-dev Connection state: [ { "streamDescriptor": { "name": "stations" }, "streamState": { "__ab_no_cursor_state_message": true } }, { "streamDescriptor": { "name": "pm10_data" }, "streamState": { "state": { "date_time": "2026-07-10" }, "states": [ { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 8, "parent_slice": {} } }, { "cursor": { "date_time": "2026-06-17" }, "partition": { "station_id": 9, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 10, "parent_slice": {} } }, { "cursor": { "date_time": "2026-03-10" }, "partition": { "station_id": 2, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 1, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 3, "parent_slice": {} } } ], "parent_state": {}, "lookback_window": 6, "use_global_cursor": false } }, { "streamDescriptor": { "name": "pm25_data" }, "streamState": { "state": { "date_time": "2026-07-10" }, "states": [ { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 8, "parent_slice": {} } }, { "cursor": { "date_time": "2026-06-17" }, "partition": { "station_id": 9, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 10, "parent_slice": {} } }, { "cursor": { "date_time": "2026-03-10" }, "partition": { "station_id": 2, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 1, "parent_slice": {} } }, { "cursor": { "date_time": "2026-07-10" }, "partition": { "station_id": 3, "parent_slice": {} } } ], "parent_state": {}, "lookback_window": 5, "use_global_cursor": false } } ] Source YAML: attached Logs: no errors or warnings I can see that relate to this issue
mades_open_data.yaml
k
Connector 2 (MADES Open Data) Analysis This connector is in a different situation from Connector 1. The state looks mostly healthy — cursors are advancing to recent dates like 
2026-07-10
. However, there are two things worth flagging. Issue 1: Stale Partitions (station_id 2 and 9) Looking at the state for 
pm10_data
and `pm25_data`: Stations 2 and 9 are significantly behind. This is likely not a connector bug — it probably means those stations have no new data since those dates (the API returns 404/400 for them, which your error handler correctly ignores). The cursor only advances when records are actually returned and parsed. Issue 2: Potential cursor_datetime_formats Mismatch This is the same class of problem as Connector 1. Your YAML has:
Copy code
cursor_datetime_formats:   - "%Y-%m-%dT%H:%M:%S%z" datetime_format: "%Y-%m-%d"
The 
cursor_datetime_formats
expects a datetime with timezone like 
2026-07-10T14:30:00+00:00
, but you haven't shared an actual API response sample. The state cursors are currently saving as date-only values (
2026-07-10
), which means either: 1. The API is returning date-only 
date_time
values (e.g., 
"2026-07-10"
), and the cursor is being saved in 
datetime_format
(
%Y-%m-%d
) rather than parsed from 
cursor_datetime_formats
— which would work but is inconsistent, or 2. The cursor is being parsed correctly from a datetime value and then stored as a date To confirm: please share a sample API response record showing the exact 
date_time
field value. This is critical to verify whether 
%Y-%m-%dT%H:%M:%S%z
actually matches what the API returns, or if you need a different format (e.g., 
%Y-%m-%dT%H:%M:%S.%f%z
with milliseconds, as was the case in Connector 1). Issue 3: No start_time_option / end_time_option — Hardcoded URL Unlike Connector 1, the date range is injected directly in the URL template:
Copy code
url: >-   <https://calidadaire.mades.gov.py/api/v1/stations/{{>   stream_slice.station_id }}/data?parameter_name=pm10&date_from={{   stream_slice.start_time }}&date_to={{ stream_slice.end_time   }}&per_page=50
This relies on 
stream_slice.start_time
and 
stream_slice.end_time
being populated by the 
DatetimeBasedCursor
. This is a valid pattern, but confirm that the 
datetime_format: "%Y-%m-%d"
matches what the API expects for `date_from`/`date_to` parameters — if the API requires datetime precision, the requests may be missing time components. Summary The most important next step is sharing a sample 
pm10_data
API response so we can confirm whether 
%Y-%m-%dT%H:%M:%S%z
correctly matches the 
date_time
field format. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
f
@kapa.ai example response for pm10: station_id: 10 parent_slice: {} start_time: 2026-05-27 end_time: 2026-06-02
Copy code
[
  {
    "date_time": "2026-06-02T23:54:55-03:00",
    "parameter_name": "pm10",
    "parameter_value": 11.04,
    "station_id": 10
  },
  {
    "date_time": "2026-06-02T23:44:55-03:00",
    "parameter_name": "pm10",
    "parameter_value": 14.39,
    "station_id": 10
  },
  {
    "date_time": "2026-06-02T23:34:56-03:00",
    "parameter_name": "pm10",
    "parameter_value": 13.7,
    "station_id": 10
  },
  {
k
Connector 2: Cursor Format Analysis Good news — the 
cursor_datetime_formats
is correctly configured for this connector. The API returns:
Copy code
"date_time": "2026-06-02T23:54:55-03:00"
And your YAML has:
Copy code
cursor_datetime_formats:   - "%Y-%m-%dT%H:%M:%S%z"
The 
%z
placeholder matches timezone offsets like 
-03:00
, so this format will correctly parse the API response. [DatetimeBasedCursor reference] However: cursor_granularity Mismatch ⚠️ This is the real issue. Your configuration has:
Copy code
datetime_format: "%Y-%m-%d"        # day precision cursor_granularity: PT1S            # 1 second
This is the same mismatch pattern documented in multiple connector fixes: the 
cursor_granularity
must match the precision of 
datetime_format
. With 
PT1S
granularity and a day-precision format, the CDK computes slice boundaries that cannot be merged, potentially causing the cursor to not advance correctly. [Datadog fix] [Productboard fix] The fix is to align 
cursor_granularity
with `datetime_format`:
Copy code
# BEFORE datetime_format: "%Y-%m-%d" cursor_granularity: PT1S  # AFTER datetime_format: "%Y-%m-%d" cursor_granularity: P1D
This applies to all 10 incremental streams in this connector (
pm10_data
, 
pm25_data
, 
temperature_data
, 
co_data
, 
no2_data
, 
o3_data
, 
so2_data
, 
relative_humidity_data
, 
wind_speed_data
, 
wind_direction_data
). Summary of Both Connectors 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
f
@kapa.ai maybe the granularity should be less than 1 day, the data comes in in irregular intervals between a few minutes and 1 hour
k
Clarifying cursor_granularity Purpose The 
cursor_granularity
is not about how frequently data arrives from the API. It serves a specific technical purpose: it represents the smallest time unit of the 
datetime_format
 used for outgoing requests, ensuring consecutive slice boundaries don't overlap. [DatetimeBasedCursor docs] In your case:
Copy code
datetime_format: "%Y-%m-%d"   # day precision — this is what's sent to the API
cursor_granularity: P1D        # must match datetime_format precision
The 
datetime_format
controls how `date_from`/`date_to` are sent in requests (e.g., 
2026-05-27
). Since that format has day precision, the granularity must be 
P1D
— regardless of how frequently the API produces records within that day. The irregular data intervals (minutes to hours) are handled correctly by the `cursor_datetime_formats`:
Copy code
cursor_datetime_formats:
  - "%Y-%m-%dT%H:%M:%S%z"   # parses the full datetime from records (e.g. 2026-06-02T23:54:55-03:00)
This is what tracks the exact last record timestamp internally. The 
cursor_granularity
only affects slice boundary computation, not record-level precision. If You Want Sub-Day Request Precision If you wanted to send time-precise `date_from`/`date_to` to the API, you would need to change both together:
Copy code
datetime_format: "%Y-%m-%dT%H:%M:%S%z"   # send full datetime to API
cursor_granularity: PT1S                   # matches second precision
But this would only make sense if the MADES API actually accepts and filters by time-of-day in those parameters — which based on your current setup using date-only values, it may not. The 
P1D
fix is correct as-is for your current 
datetime_format: "%Y-%m-%d"
. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
f
@kapa.ai i think i can make it work with more granularity, this is an example request:
Copy code
fer-dev@fer-yoga:~$ curl -H "X-Api-Key: <my-api-key> "   "<https://calidadaire.mades.gov.py/api/v1/stations/1/data?parameter_name=pm10&date_from=2026-07-01T23:00:00&date_to=2026-07-02T10:00:00&per_page=50>"
{"data":[{"date_time":"2026-07-02T10:00:00-03:00","parameter_name":"pm10","parameter_value":10.532764766216},{"date_time":"2026-07-02T09:00:00-03:00","parameter_name":"pm10","parameter_value":9.4810050443808},{"date_time":"2026-07-02T08:00:00-03:00","parameter_name":"pm10","parameter_value":7.4015013668935},{"date_time":"2026-07-02T07:00:00-03:00","parameter_name":"pm10","parameter_value":6.2413980998596},{"date_time":"2026-07-02T06:00:00-03:00","parameter_name":"pm10","parameter_value":6.0862020568053},{"date_time":"2026-07-02T05:00:00-03:00","parameter_name":"pm10","parameter_value":8.9730847521623},{"date_time":"2026-07-02T04:00:00-03:00","parameter_name":"pm10","parameter_value":9.3849821937084},{"date_time":"2026-07-02T03:00:00-03:00","parameter_name":"pm10","parameter_value":8.1943784634272},{"date_time":"2026-07-02T02:00:00-03:00","parameter_name":"pm10","parameter_value":5.9506860693296},{"date_time":"2026-07-02T01:00:00-03:00","parameter_name":"pm10","parameter_value":9.560748590827},{"date_time":"2026-07-02T00:00:00-03:00","parameter_name":"pm10","parameter_value":11.735060909986},{"date_time":"2026-07-01T23:00:00-03:00","parameter_name":"pm10","parameter_value":11.640759622256}],"links":{"first":null,"last":null,"prev":null,"next":null},"meta":{"path":"https:\/\/calidadaire.mades.gov.py\/api\/v1\/stations\/1\/data","per_page":50,"next_cursor":null,"prev_cursor":null}}
k
Good News: The API Supports Datetime Precision The API does accept full datetime values in `date_from`/`date_to` (e.g., 
2026-07-01T23:00:00
), so you can switch to second-precision throughout. Here's the updated configuration for all incremental streams:
Copy code
datetime_format: "%Y-%m-%dT%H:%M:%S"      # send full datetime to API (no timezone needed in request) cursor_granularity: PT1S                    # matches second precision ✅ cursor_datetime_formats:   - "%Y-%m-%dT%H:%M:%S%z"                  # parses API response: 2026-07-02T10:00:00-03:00 ✅ step: P7D                                   # keep as-is
This is now fully consistent: One Uncertainty to Flag The knowledge sources do not explicitly document whether 
datetime_format
without a timezone suffix (e.g., 
%Y-%m-%dT%H:%M:%S
vs 
%Y-%m-%dT%H:%M:%S%z
) is safe for outgoing requests in all CDK versions. Based on your working curl example the API accepts it without timezone, so it should be fine — but if you see issues, try 
%Y-%m-%dT%H:%M:%SZ
(with literal 
Z
) as a fallback. Also Update start_datetime and end_datetime Since you're changing the 
datetime_format
, the 
start_datetime
and 
end_datetime
blocks must also match:
Copy code
end_datetime:   type: MinMaxDatetime   datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%S') }}"   datetime_format: "%Y-%m-%dT%H:%M:%S" start_datetime:   type: MinMaxDatetime   datetime: "{{ config['start_date'] }}"   datetime_format: "%Y-%m-%d"              # config input stays date-only, that's fine
This change applies to all 10 incremental streams in the MADES connector. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
f
@kapa.ai give me exactly what I should change, a recap
k
Recap: Exact Changes Needed Connector 1 — AirGradient (respira_air_gradient.yaml) In the 
measures_past
stream's 
incremental_sync
block, add `cursor_datetime_formats`:
Copy code
# BEFORE
incremental_sync:
  type: DatetimeBasedCursor
  step: PT6H
  cursor_field: timestamp
  datetime_format: "%Y-%m-%dT%H:%M:%SZ"
  cursor_granularity: PT1S
  # no cursor_datetime_formats defined
Copy code
# AFTER
incremental_sync:
  type: DatetimeBasedCursor
  step: PT6H
  cursor_field: timestamp
  datetime_format: "%Y-%m-%dT%H:%M:%SZ"
  cursor_granularity: PT1S
  cursor_datetime_formats:
    - "%Y-%m-%dT%H:%M:%S.%fZ"   # matches API response e.g. 2026-06-01T00:00:00.000Z
    - "%Y-%m-%dT%H:%M:%SZ"      # fallback
> ⚠️ Also apply the same 
cursor_datetime_formats
fix to 
measures_raw
— it has the same configuration and likely the same API response format. Connector 2 — MADES (mades_open_data.yaml) In all 10 incremental streams (
pm10_data
, 
pm25_data
, 
temperature_data
, 
co_data
, 
no2_data
, 
o3_data
, 
so2_data
, 
relative_humidity_data
, 
wind_speed_data
, 
wind_direction_data
), change the 
incremental_sync
block:
Copy code
# BEFORE
datetime_format: "%Y-%m-%d"
cursor_granularity: PT1S
cursor_datetime_formats:
  - "%Y-%m-%dT%H:%M:%S%z"
end_datetime:
  type: MinMaxDatetime
  datetime: "{{ now_utc().strftime('%Y-%m-%d') }}"
  datetime_format: "%Y-%m-%d"
start_datetime:
  type: MinMaxDatetime
  datetime: "{{ config['start_date'] }}"
  datetime_format: "%Y-%m-%d"
Copy code
# AFTER
datetime_format: "%Y-%m-%dT%H:%M:%S"
cursor_granularity: PT1S
cursor_datetime_formats:
  - "%Y-%m-%dT%H:%M:%S%z"      # matches API response e.g. 2026-06-02T23:54:55-03:00
end_datetime:
  type: MinMaxDatetime
  datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%S') }}"
  datetime_format: "%Y-%m-%dT%H:%M:%S"
start_datetime:
  type: MinMaxDatetime
  datetime: "{{ config['start_date'] }}"
  datetime_format: "%Y-%m-%d"   # config input stays date-only, fine as-is
After Making Changes (Both Connectors) 1. Publish a new version of each connector in the Connector Builder 2. Update each connection to use the new connector version 3. For Connector 1, consider manually updating the connection state cursor from 
2026-05-01T00:00:00Z
to a more recent date to avoid re-syncing 2+ months of data 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
f
for connector 2, should I update the connection state or leave it as is? @kapa.ai
k
Connector 2: Connection State Decision Looking at the current state, most partitions are already at recent dates (
2026-07-10
), so you can largely leave the state as-is. However, there are two stale partitions worth addressing: For stations 2 and 9, the stale cursors are most likely because those stations returned no data (404/400, which your error handler ignores) — not a connector bug. After deploying the fix, the next sync will naturally attempt to catch up from those dates forward, which is the correct behavior. > ⚠️ The one caveat is that after changing 
datetime_format
from 
%Y-%m-%d
to 
%Y-%m-%dT%H:%M:%S
, the existing date-only cursor values like 
"2026-07-10"
will need to be parsed on the next sync. The knowledge sources do not explicitly confirm whether the CDK handles this date-only → datetime transition gracefully via 
cursor_datetime_formats
. To be safe, you could add 
"%Y-%m-%d"
as a fallback in `cursor_datetime_formats`:
Copy code
cursor_datetime_formats:
  - "%Y-%m-%dT%H:%M:%S%z"   # new records from API
  - "%Y-%m-%d"               # existing state values (transition safety)
This way the existing state is still parseable after the format change, and no manual state update is needed. [Cursor datetime formats] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
f
won't the change in format for the cursor affect the streams? @kapa.ai
k
This is a great question, and I want to be transparent about the limits of what the knowledge sources confirm here. Potential Impact of datetime_format Change on Existing State The concern is valid: the existing state cursors are stored as date-only values (e.g., 
"2026-07-10"
), but after the change, the connector will use 
datetime_format: "%Y-%m-%dT%H:%M:%S"
for outgoing requests. The 
cursor_datetime_formats
list is what the CDK uses to parse the saved cursor value back into a datetime on the next sync. [DatetimeBasedCursor reference] So with the fallback I suggested:
Copy code
cursor_datetime_formats:
  - "%Y-%m-%dT%H:%M:%S%z"   # parses new records from API
  - "%Y-%m-%d"               # parses existing date-only state values
The existing 
"2026-07-10"
state values should be readable via the 
%Y-%m-%d
fallback, and then formatted as 
2026-07-10T00:00:00
when injected into the next request. However, I cannot confirm this with certainty from the available knowledge sources — they don't explicitly document the state-read → format → inject pipeline behavior when transitioning between format precisions. The safest approach would be to manually update the connection state to full datetime values before deploying the new connector version, e.g.:
Copy code
{
  "date_time": "2026-07-10T00:00:00"
}
instead of:
Copy code
{
  "date_time": "2026-07-10"
}
This removes any ambiguity about how the CDK handles the transition. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Open in Slack
PreviousNext
Powered by