Hey everybody,
I have a question regarding the da...
# ask-community-for-troubleshootingp
Pierre Kerschgens
11/03/2022, 1:40 PMHey everybody,
I have a question regarding the data types of the parquet format in the S3 destination.
As source I want to sync data from a PostgreSQL database.
The PostgreSQL DDL looks like this:
CREATE*TABLE* public.task_events (serial4NOT*NULL*,int4*NULL*,*varchar*(20) *NULL*,timestampNOT*NULL*,*CONSTRAINT* task_events_pkey *PRIMARY**KEY* (id)CREATE*INDEX* ix_task_events_task_id *ON* public.task_events *USING* btree (task_id);CREATE*INDEX* ix_task_events_ts *ON* public.task_events *USING* btree (ts);print(json_schema['properties']['id']){'type': 'number', 'airbyte_type': 'integer'}print(json_schema['properties']['ts']){'type': 'string', 'format': 'date-time', 'airbyte_type': 'timestamp_without_timezone'}print(json_schema['properties']['task_id']){'type': 'number', 'airbyte_type': 'integer'}print(json_schema['properties']['event_name']){'type': 'string'}id doublets structtask_id doubleevent_name string💯 1
👍 2
f
Floyd Berndsen
11/03/2022, 2:55 PMI’d love to see this issue resolved, currently we have to build our own JSON to Parquet conversion for Airbyte-based data sources, which works, but I’d prefer to rely on your solution!
u
user
11/03/2022, 3:21 PMHi, Floyd! Sorry to hear you're running into this issue. It looks like the types are not being converted correctly in Parquet for other users as well, I've found these discussions/issues:
https://github.com/airbytehq/airbyte/issues/17564
https://github.com/airbytehq/airbyte/pull/14362#discussion_r928033781
In your case it looks like int4 should be converting correctly to int32. I'm looking into why that's happening, but while I do that I'd say keep using the converter you've built!
🙏 1
f
Floyd Berndsen
11/03/2022, 3:23 PMHey Nataly, thank you! I’m glad to hear that it’s being discussed, I’ll watch the issues to see if we could migrate our Parquet conversion to Airbyte in the future 🙂
p
Pierre Kerschgens
11/03/2022, 3:54 PMThank you for answering Nataly! Do you think this issue is covered by the Github links you provided or should I create a separate issue?
u
user
11/03/2022, 4:44 PMCreate a separate one and tag me in it as @natalyjazzviolin ! I'll triage and bring it to the attention of the team!
🙏 1
p
Pierre Kerschgens
11/03/2022, 5:08 PMThank you again Nataly, here is the issue https://github.com/airbytehq/airbyte/issues/18922
u
user
11/04/2022, 2:17 PMGot it, thanks!
j
Julien Ruey
10/03/2023, 3:04 PMHi, I'm having the same issue for timezoned postgresql datetimes : 1. they are converted to struct field {member0: datetime, member1: string} in parquet 2. There's a loss of informations about the timezone, that I'm expecting to find at 'member1' but all values are null.
4 Views