Hi, When using delta-lake source to ingest s3 parq...
# ingestionl
lemon-terabyte-66903
03/15/2022, 7:28 PMHi, When using delta-lake source to ingest s3 parquet files, each part file is separately shown on UI. How to avoid that and display only main file name instead of individual chunks?
h
helpful-optician-78938
03/15/2022, 9:55 PM@chilly-holiday-80781, could you answer this?
c
chilly-holiday-80781
03/15/2022, 9:56 PMSure-we’re working on partitioning at the moment but don't have support for it just yet. If you have a specific partition format please let us know!
l
lemon-terabyte-66903
03/15/2022, 10:40 PMwhat do you mean by “specific partition format”?
c
chilly-holiday-80781
03/16/2022, 12:45 AMHow are your partitions named?
l
lemon-terabyte-66903
03/16/2022, 1:12 AMwe have multiple partitions for each of our datasets. one of them is daily date.
c
chilly-holiday-80781
03/16/2022, 3:05 AMIs the date specified explicitly like
/folder/date=<daily_date>/file.xyzl
lemon-terabyte-66903
03/16/2022, 8:36 PMYes, the structure is something like
/folder/date=2022-01-01/id=ABC/name=DEF/partfile.parquetlemon-terabyte-66903
03/16/2022, 8:36 PM@chilly-holiday-80781 ^
c
chilly-holiday-80781
03/16/2022, 8:58 PMGot it
l
lemon-terabyte-66903
03/16/2022, 9:08 PMJust checking about the implementation timeline.
c
chilly-holiday-80781
03/17/2022, 2:19 AMMost likely a week or two
thankyou 1
l
lemon-terabyte-66903
04/11/2022, 6:23 PMHi @chilly-holiday-80781 Is this included in the current release?
c
chilly-holiday-80781
04/11/2022, 6:35 PMOh not yet—I’m still working on it!
l
lemon-terabyte-66903
04/14/2022, 6:21 PMThanks @chilly-holiday-80781. Do you think it will be included in the next release?
c
chilly-holiday-80781
04/14/2022, 10:39 PMNot sure yet