Hi, all! I have a docker compose instance of pinot...
# troubleshootingа
Андрей Морозов
10/08/2025, 10:59 AMHi, all! I have a docker compose instance of pinot:latest and try to batch ingestion from parquet file. I have a problem with deleting old segments , if I try to reload data with deleting segments via API , then from disk, deleting a table and creating this table again.
After ingestion job I see an old and new segments and rowcount with count of segments is incrementally increases.
What I'm doing wrong ?
m
Mayank
10/08/2025, 1:17 PM
What version of Pinot are you using? When you delete the table, can you check the api succeeded and all metadata from ZK is cleaned up? Another possibility is that you have old segments lying around in the output dir of the ingestion job and they end up getting pushed again.
cc @Shounak Kulkarni
а
Андрей Морозов
10/08/2025, 1:45 PMLatest version (apachepinot/pinot:latest ) , installed on monday 06/10/2025
Yes. I checked ZK API after deleting segments - there was {}
But when I reload table, It found old .parquet files in deleted subdirs and added it to segments again. I checked mounted dir from docker contaner and don't see this files / dirs, but Pinot registered them. Why ?
m
Mayank
10/08/2025, 2:20 PM
Can you list the api's are you calling to:
• Deleting segments
• Deleting table
• Checking table and all its segments are deleted
• Reload (I think you mean re-create + re-ingest?)
а
Андрей Морозов
10/08/2025, 2:22 PMDeleting segs
Copy code
curl -X 'DELETE' \
'<http://localhost:9000/segments/tb_name?type=OFFLINE>' \
-H 'accept: application/json'Андрей Морозов
10/08/2025, 2:23 PMDeleting tab
Copy code
curl -X DELETE "<http://localhost:9000/tables/tab?type=OFFLINE>"Андрей Морозов
10/08/2025, 2:23 PMChecking segs
Copy code
curl -X 'GET' \
'<http://localhost:9000/segments/tab_name?type=OFFLINE&excludeOverlapping=false>' \
-H 'accept: application/json'Андрей Морозов
10/08/2025, 2:24 PMChecking ZK meta - in API Swager
Андрей Морозов
10/08/2025, 2:27 PMTable checking in UI Pinot before and after creating.
Reload steps:
• delete segs, check
• delete seg files from dir / segments
• delete table
• delete schema
• create schema
• create table
• start Ingestion Job
Андрей Морозов
10/08/2025, 2:28 PMI create another table with new name and rename subdir with parquet file - and new table didn't see/create old segments/files
m
Mayank
10/08/2025, 2:37 PM
Hmm if everything is deleted, then this cannot happen. So either everything is not deleted (check External View), or wait for a good amount of time. You should also look at the ingestion job in what it is pushing, may be it is including previous data because it was not cleaned up
а
Андрей Морозов
10/08/2025, 2:38 PMI have another question about ingestion data from parquet. I exporting parquet from Clickhouse with parameter ..string_as_string and cheked data in the file by parquet-tools. It contains readable string data in string attributes, but Ingestion load it in hex format. Ho to fix it ?
m
Mayank
10/08/2025, 2:48 PM
What type did you define in Pinot schema? Probably bytes?
а
Андрей Морозов
10/08/2025, 2:49 PMAs string, example from schema (it loads as hex)
Copy code
{"name": "ad_href", "dataType": "STRING"},m
Mayank
10/08/2025, 2:55 PM
Then it is probably stored as hex in parquet
Mayank
10/08/2025, 2:55 PM
The only way you see hex is when Pinot stores something as bytes, or the original data was such
s
Shounak Kulkarni
10/08/2025, 4:10 PMHey @Андрей Морозов can you share the ingestion job spec?
а
Андрей Морозов
10/09/2025, 7:26 AMHi ! I found the problem of hex type of string data - the string data, which I decided as plain text was in parquet-tools in b' some text' and it is a binary format in parquet. When I re-made a parquet file with ..string_as_string (previuos data made without this flag, as I saw) - the string data was parsed correctly.
m
Mayank
10/09/2025, 12:56 PM
Thanks @Андрей Морозов for confirming.