I am using tiktok_marketing connector and pulling data of ad_groups_reports_by_country_daily stream so far the data type of stat_time_day column was coming like timestamp but now suddenly started coming like struct why @kapa.ai

The knowledge sources do not contain enough information to directly explain why the

stat_time_day

field in the

ad_groups_reports_by_country_daily

stream changed from a timestamp type to a struct type. What I can share from the available documentation is that for related streams like Ad Groups Reports Daily, the

stat_time_day

field is documented as type

string

(in

YYYY-MM-DD HH:MM:SS

format) [Ad Groups Reports Daily]. There is no mention of

ad_groups_reports_by_country_daily

specifically in the knowledge sources, nor any documented change in the data type behavior for

stat_time_day

. A few things worth noting: • The stream you're referencing (

ad_groups_reports_by_country_daily

) is not covered in the available documentation, so I cannot confirm its expected schema. • A similar type-related issue was previously reported for the TikTok Marketing connector (wrong type for

create_time

), but it was closed as resolved in connector version

5.0.8

[Github Issue]. Recommended next steps: 1. Check which version of the TikTok Marketing connector you are running — a recent update may have changed the schema. 2. File a bug report or open a discussion on the Airbyte GitHub repository with details about the connector version and the observed schema change, so the maintainers can investigate. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

same problem with "Facebook Marketing" with ad_sets, ads_insights streams and created_time, updated_time columns earlier they were coming like datetime and now coming as structs @kapa.ai

Based on the knowledge sources, I can share some relevant context, though the exact cause of your current issue isn't directly documented. What the connector does with datetime fields: The Facebook Marketing connector has a

fix_date_time

method that normalizes specific time fields — including

created_time

,

updated_time

,

event_time

,

first_fired_time

, and

last_fired_time

— into a standardized ISO 8601 format (e.g.,

2023-01-19T20:38:59+0000

). [test_base_streams.py] Known related issue (S3/Parquet destinations): If you are syncing to an S3 destination in Parquet or Avro format, there is a known historical issue where datetime/timestamp fields were stored as dictionary/struct types (e.g.,

{"member0": "2020-02-04T17:01:27.000+0000", "member1": null}

) instead of proper datetime types. This was a known bug related to how Avro logical type unions were handled. [S3 Migration Guide] A more recent S3 destination update simplified this: all time types are now converted to integral logical types, and the old union-based struct representation has been removed. If you recently upgraded your S3 destination connector, this schema change could explain why the column type appears different now. [S3 Migration Guide] Recommended steps: 1. Check your destination connector version — if you're on S3/Parquet, a recent upgrade may have changed how datetime fields are represented. 2. Refresh your schema in the affected connections (Connections → Schema tab → Refresh source schema) to ensure Airbyte picks up the latest schema. [Facebook Marketing Migration Guide] 3. If the issue persists, consider filing a bug report on the Airbyte GitHub repository with your connector versions and destination type. The knowledge sources do not explicitly document a recent breaking change to `created_time`/`updated_time` in the

ad_sets

or

ads_insights

streams specifically, so I'd recommend the above investigation steps to pinpoint the root cause. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

on S3 it was timestamp and now its struct on GCS @kapa.ai so is it because of AWS vs GCS thing @kapa.ai

Based on the knowledge sources, I cannot confirm that the difference is due to AWS S3 vs GCS specifically. The knowledge sources do not document a behavioral difference between the S3 and GCS destinations regarding how datetime fields are represented. What the knowledge sources do explain is that the struct/dictionary representation of datetime fields (e.g.,

{"member0": "2020-02-04T17:01:27.000+0000", "member1": null}

) was a known bug in both S3 and GCS Parquet/Avro destinations. It was caused by how Avro logical type unions were handled — datetime types were stored as a union of an integral logical type and a string, which in Parquet appeared as a disjoint record (struct). [S3 Migration Guide; Forum Post] The S3 destination version 1.0.0 introduced a fix that simplified all time types to integral logical types, eliminating the struct representation. [S3 Migration Guide] However, the GCS destination changelog in the knowledge sources does not show an equivalent fix being applied. The GCS connector appears to still be on an older version (latest listed is

0.4.9

), and no similar datetime simplification is documented for it. [GCS Changelog] So the most likely explanation is: • Your S3 destination was upgraded to v1.0.0+, which fixed the struct issue and now correctly outputs timestamps. • Your GCS destination has not received the same fix, so it still outputs the old struct/dictionary format. This would be a connector version difference, not an AWS vs GCS platform difference per se. I'd recommend checking the GCS connector version you're running and monitoring the GCS destination changelog for a similar fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

tell me GCS define d schema for postgres @kapa.ai

Based on the knowledge sources, I can share the defined schema/type mappings for the relevant connectors, but I want to clarify what's available: GCS Destination Output Schema The GCS destination does not write to Postgres — it writes files (CSV, JSONL, Avro, Parquet) to Google Cloud Storage buckets. Each stream gets its own directory. [GCS Output Schema] Postgres Destination Schema For the Postgres destination, the Airbyte type → Postgres type mapping is: [Postgres Schema Map] Metadata columns added by Airbyte to every Postgres table: •

_airbyte_raw_id

→

VARCHAR

•

_airbyte_extracted_at

→

TIMESTAMP WITH TIME ZONE

•

_airbyte_meta

→

JSONB

•

_airbyte_generation_id

→

BIGINT

Could you clarify what you mean by "GCS defined schema for Postgres"? Are you asking about syncing from GCS to Postgres, or something else? That would help me give a more targeted answer. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

assume that postgres is source and GCS is destination, now tell me datatype mapping for GCS @kapa.ai

When syncing from Postgres (source) → GCS (destination), the data type mapping depends on the output format you choose in GCS (Avro, Parquet, CSV, JSONL). For Avro/Parquet formats (the most structured), Airbyte first converts source data to its internal type system, then maps to Avro types as follows: [JSON to Avro Conversion] All fields are nullable in Avro (typed as

["null", <type>]

). [JSON to Avro Conversion] For CSV/JSONL formats, the data is written as flat text/JSON without strict type enforcement. Root-level fields can be flattened into columns, but nested objects/arrays remain as JSON blobs. [GCS Output Schema] Important note on datetime fields: As discussed earlier, older versions of the GCS connector stored datetime types as a union of an integral logical type and a string, which appeared as a struct in Parquet. The current documented behavior converts them to integral logical types directly. [S3 Migration Guide] The GCS destination does not have a native SQL type mapping like Postgres does — it maps to file-format types (Avro/Parquet logical types), not database column types. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

does it mean that datatype timestamp will be written as struct in parquet in GCS @kapa.ai

Based on the knowledge sources, yes — in older versions of the GCS destination connector, timestamp/datetime fields written in Parquet format were stored as a struct (disjoint record) rather than a native timestamp type. This was because datetime types were represented as a union of an integral logical type and a string, which in Parquet appeared as a record like:

Optional Record { member0: Optional Int64(logical_type=Timestamp(Microseconds)), member1: Optional String }

This is the same bug that was reported in forum discussions where users saw datetime fields as dictionaries/structs. [Forum Post; S3 Migration Guide] The S3 destination received a fix (in its migration to v1.0.0) that simplified this to a proper:

Optional Int64(logical_type=Timestamp(Microseconds))

[S3 Migration Guide] However, the knowledge sources do not document an equivalent fix being applied to the GCS destination. The GCS changelog available does not show a similar datetime simplification update. [GCS Changelog] So to directly answer your question: if you are using an older version of the GCS destination with Parquet output, timestamps are likely still written as structs. I'd recommend checking the current GCS connector version and its release notes for any recent fixes to this behavior. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

whats latest version of GCS which has this fix, I am using 0.4.9 version @kapa.ai

Based on the knowledge sources available, 0.4.9 is the latest version of the GCS destination connector listed in the changelog. [GCS Changelog] The knowledge sources do not document any fix for the datetime/timestamp struct issue in the GCS destination connector at any version, including 0.4.9. The fix for this behavior was documented only for the S3 destination (as part of its v1.0.0 migration). There is no equivalent fix mentioned in the GCS destination changelog. So unfortunately, based on the available information, there is no GCS destination version that has this fix documented. I'd recommend: 1. Opening a GitHub issue on the Airbyte repository specifically requesting this fix be applied to the GCS destination, referencing the S3 fix as a precedent. 2. As a workaround, consider using JSONL or CSV format in GCS if you need datetime fields as plain strings rather than structs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.