https://linen.dev logo
Join Slack
Powered by
2026-05-07 12:42:27 source INFO Start date over 90...
# airbyte-agents-ask-ai
n
2026-05-07 124227 source INFO Start date over 90 days, using start_date: 2026-02-05T230000-08:00 2026-05-07 124227 source INFO Using start_date: 2026-02-05T230000-08:00, end_date: 2026-05-07T001227.208148-07:00 2026-05-07 124227 source ERROR In app events reports: reports are limited to 31 days per query. Select a narrower date range and submit your query again. 2026-05-07 124227 source ERROR Encountered an exception while reading stream organic_in_app_events @kapa.ai yesterday job stared and in logs I see logs like Using start_date: 2026-02-04T230000-08:00, end_date: 2026-05-06T001227.208148-07:00 and it worked today its gailing why?
k
Why Your AppsFlyer Sync Is Failing Today Looking at your logs and the source code, here's what's happening: The 90-Day Cap Issue The 
source-appsflyer
connector enforces a 90-day maximum start date for all streams, regardless of what you configure. In 
SourceAppsflyer.streams()
, this line caps your start date: [source code]
Copy code
earliest_date = pendulum.today(timezone) - timedelta(days=90)
config["start_date"] = self.is_start_date_before_earliest_date(start_date, earliest_date)
So your effective start date is always 
today - 90 days
. Why It Worked Yesterday But Fails Today The 
organic_in_app_events
and 
in_app_events
streams use 
RawDataMixin
with 
intervals = 31
, meaning they chunk requests into 31-day windows. [source code] • Yesterday: 
start_date
was 
2026-02-04
, 
end_date
was 
2026-05-06
→ the 90-day window happened to fit within 31-day chunks cleanly enough to work. • Today: 
start_date
is 
2026-02-05
, 
end_date
is 
2026-05-07
→ the span is now slightly over a boundary that causes a chunk to exceed 31 days, triggering the AppsFlyer API error: "reports are limited to 31 days per query". This is a known bug — the connector caps all streams at 90 days even for endpoints that don't need it, and the chunking logic can produce slices that exceed the 31-day API limit for raw data streams. [GitHub issue] What You Can Do 1. Set 
start_date
to within the last 90 days in your connector config — the connector will ignore anything older anyway. 2. Set 
start_date
to within the last 31 days to ensure raw data stream chunks stay within the API's 31-day limit for 
in_app_events
/ 
organic_in_app_events
. 3. Monitor the open issue/PR for a fix: a community PR (#61520) was proposed to address the 90-day cap problem, though it was marked stale. [GitHub issue] The safest workaround right now is to keep your configured 
start_date
recent enough that the 31-day chunking never produces a slice wider than 31 days. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
@kapa.ai is there any fix in any new version of airbyte and also what if I set start_date like 2026-05-01 in source's appyflyer start_date field?
k
Is There a Fix in a Newer Version? Based on the changelog, the versions released after the issue was filed (0.2.39) only contain dependency updates — no fix for the 90-day cap or the 31-day chunking problem has been merged. The community PR (#61520) that proposed a fix was marked stale and closed without being merged. [changelog] [GitHub issue] The latest version is 0.3.0 (released 2026-04-29), which migrated to airbyte-cdk 7.x but did not address this bug. [changelog] What If You Set start_date: 2026-05-01? This would likely fix your immediate problem. Here's why: • The connector caps your start date at 
today - 90 days
(≈ 
2026-02-05
), but only if your configured date is older than 90 days. Since 
2026-05-01
is within the last 90 days, it will be used as-is. [source code] • The window from 
2026-05-01
to 
2026-05-07
is only 6 days, well within the 31-day chunk limit for 
in_app_events
/ 
organic_in_app_events
. • You would avoid the API error: "reports are limited to 31 days per query". Trade-off: You will only sync data from May 1st onward — any historical data before that date won't be pulled. Recommendation Set 
start_date: 2026-05-01
(or any date within the last 31 days) to keep syncs stable. For ongoing incremental syncs, once the cursor advances, the sync windows will naturally stay small and within limits. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
@kapa.ai but I am using inncremental_append sync mode then how come start_date is being used?
k
Why start_date Is Still Used in Incremental Append Mode This is a great question. Here's what's happening: First Sync (No State Yet) On the very first sync, there is no saved cursor state. The connector has nothing to resume from, so it falls back to 
start_date
as the initial lower bound for fetching records. This is standard Airbyte behavior — 
start_date
seeds the initial cursor. [incremental sync docs] Subsequent Syncs Once the first sync completes, the cursor state is saved (e.g., the last synced date). On the next sync, the connector should use that saved cursor value instead of 
start_date
. The Problem With source-appsflyer Looking at the source code, the 90-day cap is applied before the stream even checks the cursor state — it overrides 
config["start_date"]
globally in the 
streams()
method: [source code]
Copy code
earliest_date = pendulum.today(timezone) - timedelta(days=90)
config["start_date"] = self.is_start_date_before_earliest_date(start_date, earliest_date)
Then in 
IncrementalAppsflyerStream.stream_slices()
, the cursor state is compared against this already-capped `start_date`: [source code]
Copy code
start_date = self.get_date(parse_date(cursor_value, self.timezone), self.start_date, max)
So it takes the maximum of the cursor value and 
start_date
. This means: • If your cursor state is more recent than 
start_date
→ cursor state is used ✅ (normal incremental behavior) • If your cursor state is older than the capped 
start_date
, or doesn't exist yet → 
start_date
is used ⚠️ What This Means For You If your sync has been running successfully before and has a valid recent cursor state, subsequent syncs should use the cursor and not be affected by the 
start_date
issue. The error you're seeing today is most likely because: 1. It's your first sync (no cursor state yet), or 2. The cursor state was reset at some point Setting 
start_date: 2026-05-01
is still the right fix for the initial sync. Once the cursor advances past that date, incremental syncs will use the cursor state and the 
start_date
becomes irrelevant. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
Can I reset cursor column values for failed streams and make it work? @kapa.ai
k
Resetting Cursor State for Failed Streams Yes, you can manually edit the connection state to reset the cursor for failed streams. Airbyte provides a built-in way to do this through the UI. Via the Airbyte UI (Recommended) 1. Go to Connections and click your AppsFlyer connection 2. Click the Settings tab 3. Click the Advanced dropdown 4. Click anywhere in the Connection State to edit it 5. Modify the cursor value for the failed streams (e.g., set it to 
2026-05-01
) 6. Click "Update state" to confirm [manage connection state] > ⚠️ Warning: Updates to connection state should be handled with extreme care. Updates may break your syncs, requiring a full historical sync to fix. Make changes only as directed by the Airbyte team. [manage connection state] What to Set the Cursor To For your failed streams (
in_app_events
, 
organic_in_app_events
), you'd want to set the cursor to a recent date within the last 31 days, for example:
Copy code
[
  {
    "streamDescriptor": {
      "name": "organic_in_app_events"
    },
    "streamState": {
      "date": "2026-05-01T00:00:00Z"
    }
  }
]
This way: • The cursor starts from 
2026-05-01
• The sync window to today is only ~6 days • It stays well within the 31-day API limit Important Consideration The knowledge sources do not explicitly confirm the exact state schema for AppsFlyer streams, so check your current connection state first before editing — the field names in the state JSON must match exactly what the connector expects, otherwise the sync may break further. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
can I check in airbyte DB and reset it from there? or how to check its current value in DB ? @kapa.ai
k
Checking and Resetting Cursor State in Airbyte DB Yes, you can directly access the Airbyte internal Postgres database to check and update the cursor state. Step 1: Connect to the Airbyte DB
Copy code
docker exec -ti airbyte-db psql -U docker -d airbyte
[incremental sync tutorial] Step 2: Check Current Cursor State
Copy code
SELECT * FROM state WHERE connection_id = '<your-connection-id>';
This will return something like:
Copy code
connection_id | state | stream_name | type
--------------+-------+-------------+------
<uuid> | {"state": {"streams": [{"cursor": "2026-02-05T...", "stream_name": "organic_in_app_events", "cursor_field": ["date"]}]}} | | LEGACY
[state table docs] You can find your 
connection_id
from the Airbyte UI URL when you open your connection. Step 3: Reset the Cursor for Failed Streams Once you've confirmed the current cursor value, update it to a recent date:
Copy code
UPDATE state 
SET state = jsonb_set(state, '{state, streams, 0, cursor}', '"2026-05-01T00:00:00.000000Z"', false) 
WHERE connection_id = '<your-connection-id>';
[forum post] > If you have multiple streams in the state, 
streams, 0
refers to the first stream. Change the index (
0
, 
1
, 
2
...) to target the correct failed stream. If Running on Kubernetes You can port-forward the 
airbyte-db
service and then connect using a UI client like DBeaver to make it easier to inspect and edit the state table visually. [forum post] ⚠️ Important Notes • Always run the 
SELECT
query before the 
UPDATE
to confirm the exact JSON structure of your state — the path (
{state, streams, 0, cursor}
) must match exactly. • Make a note of the original value before changing it, in case you need to revert. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
Copy code
{
  "event_time": "2026-05-06 00:12:46"
}
this is the value for in_app_events in airbyte and I am using inncremental_append then why 90 days issues is comng? @kapa.ai
k
Why the 90-Day Issue Occurs Despite a Recent Cursor State Your cursor state 
{"event_time": "2026-05-06 00:12:46"}
is very recent (yesterday), so the cursor itself is not the problem. The issue is in how the connector code works. Looking at the 
stream_slices()
method: [source code]
Copy code
start_date = self.get_date(parse_date(cursor_value, self.timezone), self.start_date, max)
This takes the maximum of: • Your cursor value: 
2026-05-06
• `self.start_date`: which is the already 90-day-capped config start date (
2026-02-05
) So 
2026-05-06
wins here ✅ — the cursor is used correctly. Then Why Is It Still Failing? The error you're seeing:
Copy code
In app events reports: reports are limited to 31 days per query
is not coming from the cursor state. It's most likely coming from a different stream in the same sync job. Looking at your original log:
Copy code
source ERROR Encountered an exception while reading stream organic_in_app_events
The 
organic_in_app_events
stream may have a different or missing cursor state, causing it to fall back to the 90-day-capped 
start_date
(
2026-02-05
), which creates a window larger than 31 days. [source code] What to Check Run this query to see the cursor state for all streams in your connection:
Copy code
SELECT state FROM state WHERE connection_id = '<your-connection-id>';
You likely have a recent cursor for 
in_app_events
but a missing or old cursor for 
organic_in_app_events
. Setting the cursor for 
organic_in_app_events
to a recent date (e.g., 
2026-05-01
) via the UI or DB should fix the issue. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
see these logs 2026-05-07 124553 replication-orchestrator INFO Stream status TRACE received of status: RUNNING for stream daily_report 2026-05-07 124553 source INFO Read 58 records from daily_report stream 2026-05-07 124553 source INFO Marking stream daily_report as STOPPED 2026-05-07 124553 replication-orchestrator INFO Stream status TRACE received of status: COMPLETE for stream daily_report 2026-05-07 124553 source INFO Finished syncing daily_report 2026-05-07 124553 source INFO SourceAppsflyer runtimes: Syncing stream daily_report 00104.311769 Syncing stream installs 00044.597094 Syncing stream organic_in_app_events 00000.468925 Syncing stream organic_installs 00022.539645 Syncing stream retargeting_in_app_events 00000.358616 Syncing stream retargeting_installs 00011.447788 Syncing stream retargeting_partners_report 00102.144685 2026-05-07 124553 source ERROR In app events reports: reports are limited to 31 days per query. Select a narrower date range and submit your query again. 2026-05-07 124553 source ERROR Encountered an exception while reading stream in_app_events Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/abstract_source.py", line 130, in read stream_is_available, reason = stream_instance.check_availability(logger, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/core.py", line 269, in check_availability return self.availability_strategy.check_availability(self, logger, source) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 56, in check_availability is_available, reason = self.handle_http_error(stream, logger, source, error) @kapa.ai
k
Analysis of Your Logs Looking at your logs carefully, the issue is specifically with 
in_app_events
 stream, not 
organic_in_app_events
as previously suspected. What the Logs Tell Us 1. These streams completed successfully: ◦ 
daily_report
✅ ◦ 
installs
✅ ◦ 
organic_in_app_events
✅ ◦ 
organic_installs
✅ ◦ 
retargeting_in_app_events
✅ ◦ 
retargeting_installs
✅ ◦ 
retargeting_partners_report
✅ 1. 
in_app_events
failed with the 31-day limit error. Root Cause Since 
organic_in_app_events
succeeded but 
in_app_events
failed, it strongly suggests that 
in_app_events
has a missing or outdated cursor state, causing it to fall back to the 90-day-capped 
start_date
(
2026-02-05
), creating a window larger than 31 days. Fix: Check and Update the Cursor in DB First, check the current state:
Copy code
SELECT state FROM state WHERE connection_id = '<your-connection-id>';
Look for 
in_app_events
in the JSON output. You likely see something like:
Copy code
{
  "event_time": "2026-05-06 00:12:46"
}
for 
organic_in_app_events
but nothing or an old date for 
in_app_events
. Then update it:
Copy code
UPDATE state  SET state = jsonb_set(   state,    '{state, streams, 0, cursor}',    '"2026-05-06T00:00:00.000000Z"',    false )  WHERE connection_id = '<your-connection-id>';
> ⚠️ Make sure to check the exact JSON path for 
in_app_events
in your state before updating — the index (
0
, 
1
, 
2
...) must point to the correct stream entry. [forum post] Alternatively, use the Airbyte UI under Connection → Settings → Advanced → Connection State to edit it visually, which is safer as you can see the full JSON structure before editing. [manage connection state] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
I have checked in DB cursor value for in_pp_event is {"event_time": "2026-05-06 001246"} then why to reset @kapa.ai
k
Good Point — Cursor Is Recent, So Reset Is Not Needed You're absolutely right. If 
in_app_events
cursor is already 
2026-05-06 00:12:46
, then: • The connector should use 
2026-05-06
as start date (max of cursor vs capped start_date) • The window from 
2026-05-06
to 
2026-05-07
is only 1 day • This is well within the 31-day limit So resetting the cursor is not the solution here. What Could Actually Be Causing the Failure The error occurs during 
check_availability
, which happens before the cursor state is even applied:
Copy code
File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 56, in check_availability
    is_available, reason = self.handle_http_error(stream, logger, source, error)
The 
check_availability
method makes a test request to validate the stream is accessible, and this test request may be using the 90-day-capped 
start_date
directly from config, completely ignoring the cursor state. What I'd Recommend The knowledge sources do not contain enough detail about the exact 
check_availability
implementation in 
source-appsflyer
to confirm this with certainty. However, I'd suggest: 1. Check your configured 
start_date
 in the AppsFlyer source settings — if it's older than 90 days (e.g., 
2020-01-01
), the availability check likely uses the capped date (
2026-02-05
) and creates a >31 day window 2. Change the configured 
start_date
to 
2026-05-01
 in the source settings — this would fix the availability check without needing to touch the cursor state 3. The cursor state will still take over for actual data syncing since 
2026-05-06 > 2026-05-01
👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
why 
start_date
is coming into picture
@kapa.ai in this case
k
Why start_date Is Coming Into Picture Looking at the actual source code, here is exactly why 
start_date
is involved even when you have a valid cursor state. The check_availability Call From your traceback:
Copy code
File "airbyte_cdk/sources/abstract_source.py", line 130, in read
    stream_is_available, reason = stream_instance.check_availability(logger, self)
This 
check_availability
is called before any cursor state is applied. It makes a test API request to verify the stream is accessible. How start_date Gets Into the Test Request Looking at the 
SourceAppsflyer.streams()
method, the 
start_date
is set globally at connector startup: [source code]
Copy code
earliest_date = pendulum.today(timezone) - timedelta(days=90)
start_date = parse_date(config.get("start_date") or pendulum.today(timezone), timezone)
config["start_date"] = self.is_start_date_before_earliest_date(start_date, earliest_date)
And then 
request_params
uses it directly: [source code]
Copy code
def request_params(self, stream_state, stream_slice=None, next_page_token=None):
    params = {
        "from": pendulum.yesterday(self.timezone).to_date_string(),
        "to": pendulum.today(self.timezone).to_date_string(),
        ...
    }
The Key Sequence
Copy code
1. Connector starts
2. config["start_date"] gets capped to 90 days (2026-02-05)
3. check_availability() fires → makes test API request using start_date (2026-02-05)
4. Window = 2026-02-05 to 2026-05-07 = ~90 days → EXCEEDS 31-day limit
5. API returns error → stream marked as unavailable
6. Cursor state is NEVER even checked
So the cursor state 
{"event_time": "2026-05-06 00:12:46"}
is completely bypassed because the failure happens at the availability check stage, before incremental logic kicks in. The Real Fix The only reliable fix given the current connector code is to set 
start_date
to a recent date (e.g., 
2026-05-01
) in your AppsFlyer source configuration. This ensures: • The availability check uses a recent date ✅ • The window stays within 31 days ✅ • Your cursor state (
2026-05-06
) still takes over for actual data syncing since it's more recent ✅ 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
so where start_date is getting saved is it like per stream getting saved? tell me table name if I can make chagne value for that stream ? @kapa.ai
k
Where start_date Is Saved The 
start_date
is not saved per stream in the database. It is part of the source connector configuration, not the state table. It is stored in the 
actor
table in the Airbyte database, which holds connector configurations: [data catalog docs]
Copy code
SELECT id, name, configuration 
FROM actor 
WHERE actor_type = 'source' 
AND name LIKE '%appsflyer%';
The 
configuration
column is a JSONB field that contains your source settings including `start_date`:
Copy code
{
  "app_id": "...",
  "api_token": "...",
  "start_date": "2020-01-01",
  "timezone": "UTC"
}
How to Update It
Copy code
UPDATE actor
SET configuration = jsonb_set(configuration, '{start_date}', '"2026-05-01"', false)
WHERE id = '<your-source-id>';
Important Distinction So 
start_date
applies globally to all streams — you cannot set it differently per stream. That's why changing it to 
2026-05-01
in the source config is the cleanest fix for your issue. > ⚠️ I must note that the knowledge sources do not explicitly confirm the 
actor
table name and structure — please verify this before making any changes to avoid breaking your connector configuration. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
n
Copy code
SELECT id, name, configuration
FROM actor
WHERE actor_type = 'source'
AND name LIKE '%onevoneme_appsflyer%';
this giving start _data like { "app_id": "id1522405338", "timezone": "US/Pacific", "api_token": { "_secret": "airbyte_workspace_047d1871-35a1-4c41-ae8e-dd66bae22c5d_secret_7a014453-4be3-40e4-80a1-bf2a5c1d0c42_v3" }, "start_date": "2024-01-01" } but where this start_date is getting saved? where its picking 2026-02-04 then? @kapa.ai
k
Why 2026-02-04 Instead of 2024-01-01 Your 
start_date
in the config is 
2024-01-01
, which is correctly saved in the 
actor
table. However, the connector overrides it at runtime — it never actually uses 
2024-01-01
. Here is exactly where it happens in the connector code: [source code]
Copy code
earliest_date = pendulum.today(timezone) - timedelta(days=90) start_date = parse_date(config.get("start_date") or pendulum.today(timezone), timezone) config["start_date"] = self.is_start_date_before_earliest_date(start_date, earliest_date)
The Runtime Override Logic
Copy code
1. Your config start_date = "2024-01-01"  ← stored in actor table 2. earliest_date = today - 90 days = 2026-02-05 3. is_start_date_before_earliest_date("2024-01-01", "2026-02-05")    → "2024-01-01" is older than 90 days → returns "2026-02-05" 4. config["start_date"] is overridden to "2026-02-05" IN MEMORY 5. Connector uses "2026-02-05" for all requests
So: • 
actor
table stores 
2024-01-01
← what you configured • Runtime memory overrides it to 
2026-02-05
← what connector actually uses • Nothing is saved back to DB — this override happens purely in memory every time the sync runs The Fix Since the override happens in code at runtime, changing the DB value alone won't help. The only effective fix is to set 
start_date
to a value within the last 90 days so the override doesn't trigger:
Copy code
UPDATE actor SET configuration = jsonb_set(configuration, '{start_date}', '"2026-05-01"', false) WHERE id = '<your-source-id>';
Or simply update it via the Airbyte UI → Sources → AppsFlyer → Edit → start_date → 
2026-05-01
. This way: • 
2026-05-01
is within 90 days → no override happens ✅ • Availability check uses 
2026-05-01
→ window is ~6 days → within 31-day limit ✅ • Your cursor state 
2026-05-06
is more recent → cursor takes over for actual sync ✅ 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
when it runtime overrides it then where exaclty its getting saved? @kapa.ai
k
It Is NOT Saved Anywhere — It's Pure In-Memory Override The runtime override is never persisted to any database table. Looking at the code: [source code]
Copy code
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
    config["timezone"] = config.get("timezone", "UTC")
    timezone = pendulum.timezone(config.get("timezone", "UTC"))
    earliest_date = pendulum.today(timezone) - timedelta(days=90)
    start_date = parse_date(config.get("start_date") or pendulum.today(timezone), timezone)
    config["start_date"] = self.is_start_date_before_earliest_date(start_date, earliest_date)
The 
config
object here is just a Python dictionary passed in memory during the sync job. When it does 
config["start_date"] = ...
, it is: • ✅ Overriding the value in the in-memory Python dict • ❌ NOT writing back to the 
actor
table • ❌ NOT saving to the 
state
table • ❌ NOT persisted anywhere Lifecycle of the Override
Copy code
Sync starts
    → actor table config loaded into memory (start_date = "2024-01-01")
    → streams() method overrides in-memory config (start_date = "2026-02-05")
    → Sync runs using "2026-02-05"
Sync ends
    → Memory cleared
    → actor table still has "2024-01-01" unchanged

Next sync starts
    → Same override happens again from scratch
So every single sync recalculates and re-applies the 90-day cap fresh from 
today - 90 days
. That's why you'll always see a different capped date each day in your logs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
I understand that in memory overriding it but how it gets to know 2026-02-04? it could have been like 2026-01-10 so how airbyte is deciding in memory date ?@kapa.ai
k
How Airbyte Decides the In-Memory Date 2026-02-05 It is a simple calculation done at the exact moment the sync runs. Looking at the code: [source code]
Copy code
earliest_date = pendulum.today(timezone) - timedelta(days=90)
That's it — it is literally today's date minus 90 days, calculated fresh every time a sync runs. Your Specific Case
Copy code
Today = 2026-05-07
2026-05-07 - 90 days = 2026-02-05
So 
2026-02-05
is the in-memory override value. Why It Was 2026-02-04 Yesterday
Copy code
Yesterday = 2026-05-06
2026-05-06 - 90 days = 2026-02-04
That's exactly why your logs showed:
Copy code
# Yesterday's log
Using start_date: 2026-02-04T23:00:00-08:00

# Today's log  
Using start_date: 2026-02-05T23:00:00-08:00
The date shifts by 1 day every day because it is always recalculated as 
today - 90 days
at sync runtime. There is no stored value anywhere — it is purely a mathematical calculation based on the current date when the sync executes. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
Copy code
1. Your config start_date = "2024-01-01"  ← stored in actor table
2. earliest_date = today - 90 days = 2026-02-05
3. is_start_date_before_earliest_date("2024-01-01", "2026-02-05")
   → "2024-01-01" is older than 90 days → returns "2026-02-05"
4. config["start_date"] is overridden to "2026-02-05" IN MEMORY
5. Connector uses "2026-02-05" for all requests
this is all fine when Airbyte itself calculating this 90 days back date then how come getting --- 2026-05-07 124553 source ERROR In app events reports: reports are limited to 31 days per query. Select a narrower date range and submit your query again. 2026-05-07 124553 source ERROR Encountered an exception while reading stream in_app_events @kapa.ai is it because of day light savier? has anyone else reported this issue @kapa.ai
k
Why the 31-Day Error Occurs Even With 90-Day Cap I need to be honest here — the knowledge sources do not contain enough detail to give a fully confident answer on this specific interaction. But here is the most logical explanation: The 90-Day Cap vs 31-Day API Limit Are Two Different Things These are two separate restrictions: The connector caps 
start_date
to 90 days, but the 
in_app_events
stream (a raw data stream) chunks requests using `intervals = 60`: [source code]
Copy code
class IncrementalAppsflyerStream(AppsflyerStream, ABC):     intervals = 60
So it creates 60-day chunks from the 90-day window, and the AppsFlyer API rejects any chunk larger than 31 days for raw data streams. Regarding Daylight Saving Time The knowledge sources do not explicitly mention DST as a contributing factor. However, looking at your logs:
Copy code
# Yesterday worked: start_date: 2026-02-04T23:00:00-08:00  # Today failed: start_date: 2026-02-05T23:00:00-08:00
The 
-08:00
offset suggests your timezone is 
US/Pacific
. The time component 
23:00:00
instead of 
00:00:00
is indeed a timezone/DST conversion artifact — but the knowledge sources do not contain enough information to confirm whether DST specifically caused today's failure vs yesterday's success. Has Anyone Reported This? Yes — this is a known bug reported in GitHub issue #60994, where all streams are capped at 90 days even for endpoints that don't need it, and the fix was never merged. [GitHub issue] Bottom Line The safest fix remains setting 
start_date
to 
2026-05-01
 in your source config so: • No 90-day override triggers • No 60-day chunk exceeds the 31-day API limit • Your cursor state 
2026-05-06
takes over for actual syncing 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
n
do you think tomorrow if I run ppeline I'll not get this issue @kapa.ai
k
Will the Issue Occur Tomorrow? Yes, it will very likely happen again tomorrow if you don't change the 
start_date
config. Here's why — every day the calculation shifts:
Copy code
Tomorrow = 2026-05-08
earliest_date = 2026-05-08 - 90 days = 2026-02-06
And the connector will still use 
intervals = 60
to chunk requests: [source code]
Copy code
class IncrementalAppsflyerStream(AppsflyerStream, ABC):
    intervals = 60
So the chunk size (60 days) will always exceed the AppsFlyer API's 31-day limit for raw data streams, regardless of what day you run it. The Pattern Will Repeat Daily
Copy code
Today:     start_date = 2026-02-05, chunk = 60 days → FAILS ❌
Tomorrow:  start_date = 2026-02-06, chunk = 60 days → FAILS ❌
Day after: start_date = 2026-02-07, chunk = 60 days → FAILS ❌
Permanent Fix Simply update 
start_date
to 
2026-05-01
in your source configuration via the Airbyte UI. This is a one-time change that will: • Keep the sync window small (~6 days) ✅ • Stay within the 31-day API limit ✅ • Let your cursor state take over for subsequent syncs ✅ Without this change, the pipeline will fail every single day. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
if I reset 
start_date
at connection level it means Airbyte might create duplicate data and I need to dedup it in downstreams @kapa.ai
k
Will Resetting start_date Cause Duplicate Data? The knowledge sources do not contain enough information to give a fully confident answer specific to AppsFlyer, but based on general Airbyte incremental sync behavior: Good News — Your Cursor Protects You Since you are using Incremental Append and your cursor state for 
in_app_events
is 
2026-05-06 00:12:46
, the connector will use the maximum of cursor vs start_date: [source code]
Copy code
start_date = self.get_date(parse_date(cursor_value, self.timezone), self.start_date, max)
So if you set `start_date = 2026-05-01`:
Copy code
cursor_value = 2026-05-06
start_date   = 2026-05-01
max(2026-05-06, 2026-05-01) = 2026-05-06 ← cursor wins
No historical re-sync, no duplicates ✅ When Duplicates WOULD Occur Duplicates would only happen if you: [incremental append docs] • Reset the connection state entirely → cursor cleared → full re-sync from 
start_date
• Manually set cursor to an older date → re-syncs already synced data Summary So simply changing 
start_date
to 
2026-05-01
without touching the cursor state is safe and should not cause any duplicate data. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
n
I am using inncremental_append sync mode so you means to say if I set start_date = 2026-05-01 and cursor_value = 2026-05-06 then max(2026-05-06, 2026-05-01) = 2026-05-06 ← cursor wins will use this @kapa.ai
k
Yes, That Is Correct Exactly right. Looking at the connector code: [source code]
Copy code
def stream_slices(self, *, sync_mode, cursor_field, stream_state):
    stream_state = stream_state or {}
    cursor_value = stream_state.get(self.cursor_field)
    start_date = self.get_date(parse_date(cursor_value, self.timezone), self.start_date, max)
The 
max
comparator ensures:
Copy code
cursor_value = 2026-05-06  ← from saved state
start_date   = 2026-05-01  ← your new config value
max(2026-05-06, 2026-05-01) = 2026-05-06 ← cursor wins ✅
What This Means Practically • Sync will start from 
2026-05-06
✅ • No data before 
2026-05-06
will be re-synced ✅ • No duplicates ✅ • Window from 
2026-05-06
to today is only 1 day ✅ • Well within the 31-day API limit ✅ One Thing to Note However, I must be transparent — the knowledge sources confirm this logic for the actual data sync part. But as we discussed earlier, the 
check_availability
call happens before this cursor logic kicks in. Setting 
start_date = 2026-05-01
fixes that too since it keeps the availability check window small and within the 31-day limit. So changing 
start_date = 2026-05-01
in your source config is the safe and correct fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
n
what if I create another pipeline which dumps data i same current warehouse only and I set start_date as 2026-05-01 it means for next 6 days data i'll have duplicates so can I use _datagol_extracted_at and use dedup logic? @kapa.ai
k
Creating a New Pipeline With Duplicate Handling The knowledge sources do not contain enough specific information about 
_datadog_extracted_at
or the exact metadata columns AppsFlyer connector writes. However, based on general Airbyte knowledge: I Would NOT Recommend Creating a New Pipeline Here's why: • You already have a working pipeline with cursor state 
2026-05-06
• Simply changing 
start_date = 2026-05-01
on the existing source config fixes the issue • The cursor state 
2026-05-06
will still win → no duplicates, no new pipeline needed If You Still Want a New Pipeline Yes, you would get duplicates for the overlapping period (
2026-05-01
to 
2026-05-06
). You could use Airbyte's metadata column 
_airbyte_extracted_at
(not 
_datadog_extracted_at
) for dedup: [incremental append docs]
Copy code
SELECT * FROM in_app_events
QUALIFY ROW_NUMBER() OVER (
    PARTITION BY <primary_key>
    ORDER BY _airbyte_extracted_at DESC
) = 1
But this adds unnecessary complexity when the simpler fix is: Recommended Approach
Copy code
1. Keep existing pipeline
2. Just update start_date = "2026-05-01" on existing source config
3. Cursor state 2026-05-06 wins → no duplicates
4. No new pipeline needed
> ⚠️ I must note the knowledge sources do not explicitly confirm the exact metadata column name for AppsFlyer — please verify whether it is 
_airbyte_extracted_at
or 
_airbyte_emitted_at
in your destination table before using it in dedup logic. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Open in Slack
PreviousNext
Powered by