<@U04S8PLJWMT> does airbyte not give full listed m...
# replication-ask-ai
p
@kapa.ai does airbyte not give full listed metrics from stream in incremental_append method
k
Airbyte’s Incremental | Append mode is designed to read only new or modified records since the last sync, not the full history on every run. The first time you run an incremental sync, it is effectively a full refresh of the stream; subsequent runs only fetch the delta based on the cursor. [[Sync modes](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes); [Incremental append](https://docs.airbyte.com/platform/1.7/using-airbyte/core-concepts/sync-modes/incremental-append)] Key points: - First incremental sync = full table (all available records at that time). [[Sync modes](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes)] - Later incremental syncs = only new/updated records based on the configured cursor field. [[Configured stream](https://docs.airbyte.com/platform/understanding-airbyte/airbyte-protocol#configuredairbytestream); [Incremental append](https://docs.airbyte.com/platform/1.7/using-airbyte/core-concepts/sync-modes/incremental-append)] - Incremental append never deletes or mutates existing rows in the destination; it just appends new ones, so you may see multiple versions of the same logical record. [[Incremental append](https://docs.airbyte.com/platform/1.7/using-airbyte/core-concepts/sync-modes/incremental-append)] If you are missing historical data after the first incremental run, that is not expected behavior according to the docs, but the provided knowledge sources do not explain specific root causes or bugs for that situation.
p
@kapa.ai Currently, we are storing the data in JSONL format. However, we’ve noticed that a few expected metrics are missing from the JSONL files returned by the source. This appears to be an issue with the data being provided, rather than with our storage or processing logic.
or does airbyte gives us only the updated or metric with values
k
A message from kapa.ai
From the knowledge we have, there are several known situations where metrics or fields are missing from the data coming from the source connector itself, not from the destination or storage format: - Some connectors simply don’t request or expose all available fields from the upstream API. Example: the Bing Ads connector only selects a limited set of columns from the CampaignPerformanceReport, even though the API supports many more. [[Bing Ads fields](https://discuss.airbyte.io/t/1824)] - Some connectors have known bugs or gaps where certain fields/metrics are not included in the stream schema or output, even though the API returns them: - HubSpot Marketing Emails: some fields (e.g.
allEmailCampaignIds
,
primaryEmailCampaignId
,
subscriptionName
) are missing from the declared schema; for JSON destinations they may end up only in
_airbyte_additional_properties
, and for Avro/Parquet they are silently dropped. [[HubSpot missing fields](https://github.com/airbytehq/airbyte/issues/66479)] - Klaviyo: attribution-related fields were missing or not properly mapped after connector updates. [[Klaviyo missing fields](https://github.com/airbytehq/airbyte/issues/54174); [Klaviyo attributions](https://github.com/airbytehq/airbyte/issues/55208)] - Pardot: large portions of data missing for several streams (visits, visitors, etc.), even though new data appears. [[Pardot missing data](https://github.com/airbytehq/airbyte/issues/58666)] - Facebook Marketing: under‑reported purchases due to using a less suitable API method. [[Facebook purchases](https://github.com/airbytehq/airbyte/issues/64554)] In all these cases, the problem is upstream of storage: the connector either doesn’t request the fields, doesn’t include them in the schema, or the incremental logic/API choice causes data to be missed. Your observation that the JSONL files themselves are missing metrics is consistent with these patterns. However, the knowledge sources you provided do not mention your specific source connector or stream, so I can’t say exactly why your metrics are missing. Given that: 1. Compare the JSONL content to the source API’s raw response (or its documentation) to confirm the fields really aren’t being requested/returned. 2. Check whether those metrics might be present only in an “additional properties” field (for JSON destinations) and thus not visible as top‑level columns. [[HubSpot missing fields](https://github.com/airbytehq/airbyte/issues/66479)] 3. If the API clearly returns those metrics but they’re absent from the Airbyte stream, this likely warrants a GitHub issue against the specific source connector, similar to the Klaviyo/HubSpot/Pardot reports above. If you share which connector and stream you’re using, plus an example of the API response vs. the JSONL record, that would be the next step to align with the known patterns in these issues.
p
@kapa.ai my source is tiktok-marketing
k
A message from kapa.ai
For the TikTok Marketing source, there are a few relevant points in the docs and issues that match what you’re seeing (missing metrics in the raw records, independent of storage): 1. Not all possible metrics are exposed by the connector. The connector defines a fixed schema per stream (e.g.
ads_audience_reports_daily
,
ads_reports_daily
, etc.), and only those fields are emitted. The TikTok API may support more metrics than the connector currently maps. The changelog shows multiple releases whose only purpose was to “add fields/metrics” to report streams, which implies earlier versions did not expose everything the API can return. [[TikTok changelog](https://docs.airbyte.com/integrations/sources/tiktok-marketing#changelog)] 2. Some metrics/dimensions were added later via PRs. For example,
country_code
and
platform
audience reports were added in PR #22134 and only became available from version 2.0.5 onward. [[TikTok changelog](https://docs.airbyte.com/integrations/sources/tiktok-marketing#changelog)] If you expect metrics/dimensions that were introduced in later versions but are running an older connector, they will simply not appear in your JSONL. 3. Schema mismatches / evolution are known for TikTok reports. There are forum posts showing detailed JSON schemas for
ads_audience_reports_daily
and related streams, and the changelog mentions several schema fixes and expansions (e.g. “Update schema types for AudienceReports and BasicReports streams”, “Add fields to ad reports streams”, “Add new metrics to ads_reports_daily stream”). [[TikTok audience schema](https://discuss.airbyte.io/t/3939); [TikTok changelog](https://docs.airbyte.com/integrations/sources/tiktok-marketing#changelog)] This supports the idea that some metrics may simply not be part of the current manifest/schema you’re running. 4. Incremental vs. full data does not change which metrics are present. Incremental sync affects which records are fetched (by time window / cursor), but the set of fields per record is determined by the connector’s schema and request options, not by the sync mode. The docs and issues do not indicate any behavior where incremental sync would drop specific metrics while full refresh would include them. Given the knowledge sources, the most likely explanations are: - The metrics you expect are not included in the current TikTok Marketing connector schema/version you’re using, even though the TikTok API supports them. - Or they were added in a newer version (per the changelog) and you’re on an older one. What you can do next, based on the docs: 1. Check your connector version against the TikTok Marketing changelog to see if the metrics you expect were added in a later release. [[TikTok changelog](https://docs.airbyte.com/integrations/sources/tiktok-marketing#changelog)] 2. Inspect the stream schema in the Airbyte UI (Schema tab) for the specific stream (e.g.
ads_reports_daily
,
ads_audience_reports_daily
) and confirm whether the missing metrics are defined there. [[TikTok supported streams](https://docs.airbyte.com/integrations/sources/tiktok-marketing#supported-streams)] 3. If the TikTok API returns those metrics but they are not in the Airbyte schema or JSONL, that’s consistent with prior “missing fields” issues in other connectors and would warrant opening a GitHub issue for
source-tiktok-marketing
, referencing the exact stream, connector version, and example API vs. Airbyte record. The knowledge sources do not list a specific open bug for “TikTok metrics missing in incremental JSONL,” so I can’t point to a confirmed defect beyond the general schema/field‑coverage limitations described above.