Hi,
I've setup Airbyte and I'm liking it so far. O...
# replication-troubleshootings
Steven Herweijer
10/11/2022, 6:09 PMHi,
I've setup Airbyte and I'm liking it so far. One thing bothers me though, when moving data from postgres to another postgres, table names on the destination are getting shortened well below the maximum table name length of postgres. Why is this? Can I turn this off and make it use the same table name as from source?
✍️ 1
u
user
10/11/2022, 6:10 PM@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2738 to ensure timely resolution!
m
Marcos Marx (Airbyte)
10/11/2022, 7:26 PMWhat version of postgres are you using? Normalization reduces name of column based on limitation of the destination.
s
Steven Herweijer
10/12/2022, 8:08 AMHi, I’m on 0.40.14
Steven Herweijer
10/12/2022, 8:08 AMThe postgres destination connector is on 0.3.26
Steven Herweijer
10/12/2022, 8:09 AMThe funny thing is that it also creates these “temporary” tables (the ones which names start with _airbyte), which do have the full table name
Steven Herweijer
10/12/2022, 10:37 AM@Marcos Marx (Airbyte)
m
Marcos Marx (Airbyte)
10/12/2022, 3:22 PMI’m asking about the Postgres database not connectors, sorry if it was not clear
s
Steven Herweijer
10/12/2022, 4:00 PMHi @Marcos Marx (Airbyte), sorry didn’t read well. Using Amazon RDS for Postgres 13.7
m
Marcos Marx (Airbyte)
10/14/2022, 5:49 PMNormalization applies the default of maximum chars allowed by Pg documentation:
https://github.com/airbytehq/airbyte/blob/e5fccaaf035c01f79c41391902e403c622b817c5[…]normalization/transform_catalog/destination_name_transformer.py
Marcos Marx (Airbyte)
10/14/2022, 5:49 PMYou can raise an github issue to make this configurable
u
user
10/14/2022, 5:50 PMThanks
s
Steven Herweijer
10/14/2022, 6:06 PMHi
Steven Herweijer
10/14/2022, 6:06 PMIt is not using the 63
Steven Herweijer
10/14/2022, 6:07 PMIt is shortening to 43
Steven Herweijer
10/14/2022, 6:08 PMe.g.:
user_organizationplatformprofile_proposition_typesuser_organizationpla__ile_proposition_typesSteven Herweijer
10/14/2022, 6:09 PMSince both source and target are postgres anyway there isn't a reason to change any table names anyway @Marcos Marx (Airbyte)
Steven Herweijer
10/14/2022, 6:15 PMThis seems to have to do with these:
Copy code
# DBT also needs to generate suffix to table names, so we need to make sure it has enough characters to do so...
TRUNCATE_DBT_RESERVED_SIZE = 12
# we keep 4 characters for 1 underscore and 3 characters for suffix (_ab1, _ab2, etc)
# we keep 4 characters for 1 underscore and 3 characters hash (of the schema)
TRUNCATE_RESERVED_SIZE = 8Steven Herweijer
10/14/2022, 6:17 PMAlthough I can't work out how it exactly comes to 43. But it seems unneeded to reserve these characters for table names that don't need a suffix. After all, airbyte also creates the table
_airbyte_raw_user_organizationplatformprofile_proposition_typesm
Marcos Marx (Airbyte)
10/14/2022, 6:29 PMyep, because dbt uses some character for internal purpose
Marcos Marx (Airbyte)
10/14/2022, 6:30 PMprobably there are some improvements to be made in normalization @Steven Herweijer do you mind creating a Github issue to request this?
s
Steven Herweijer
10/14/2022, 6:52 PMI can do this on Monday @Marcos Marx (Airbyte)
Steven Herweijer
10/14/2022, 6:52 PMIn the main project?
Steven Herweijer
10/14/2022, 6:52 PMor any specific repo
m
Marcos Marx (Airbyte)
10/14/2022, 7:05 PMmain repo