https://pinot.apache.org/ logo
Join Slack
Powered by
we have pinot realtime data backup on deepstore (g...
# troubleshooting
s
we have pinot realtime data backup on deepstore (gcs )as tar .. data was published on kafka as json but since its pushed on gcs as tar .. how can we restore data back into realtime / offline table .. q1)can we restore the tar segments present on gcs directly into realtime table q2)If not possible to restore into realtime table as of now (since zookeeper metadata also needed and feature may not be ready).. can we restore data stored in deep store as tar into offline table .. and then create hybrid table in order to not lose old data when Disaster Recovery happens .. q3)how to restore into offline table .. using Job Segment Metadata Push or Segment URI Push where can I refer to get the steps to restore into offline table? (also should we use org.apache.pinot.plugin.inputformat.json.JSONRecordReader https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-input-format/pinot[…]/org/apache/pinot/plugin/inputformat/json/JSONRecordReader.java*) in config since data was published on kafka as json for .pinot/RecordReaderSpec.java at master · apache/pinot* some links already explored :- https://docs.pinot.apache.org/basics/getting-started/frequent-questions/operations-faq https://docs.pinot.apache.org/basics/data-import/pinot-file-system/import-from-gcp https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-file-system/pinot-[…]rc/main/java/org/apache/pinot/plugin/filesystem/GcsPinotFS.java https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-segment-uploader/p[…]apache/pinot/plugin/segmentuploader/SegmentUploaderDefault.java https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion#Bypassingdeeps[…]on-Configchange https://docs.pinot.apache.org/basics/components/table
@Mayank @Xiang Fu @Kishore G
g
I would create a little standalone application that is a Kafka producer that reads from GCS and writes your data back out to Kafka. If you are only updating part of your records, then this should help: https://docs.pinot.apache.org/basics/data-import/upsert
m
1) is not available. 2) you can simply push the segment tar in GCS to Pinot via a curl command. Or if the ingestion job has a skip build, then you can just do the push metadata + uri
You don’t need to use Kafka or redo segment generation for pushing to offline
✔️ 1
For RT, you need to have ZK state restored currently, and that would work
s
cc: @Mohamed Hussain
what is the curl command for pushing tar in GCS to pinot @Mayank..any link or reference..is it available on pinot controller UI in swagger API section? ie http://localhost:9001/help#/Table/ingestFromURI
For RT, you need to have ZK state restored currently, and that would work --> we dont need to restore into RT table.. we will restore realtime table generted tar into offline table and use hybrid table for compliance dashboards to have historical data as well when we restore
uri.JPG
m
Yeah, you can just push the tar.gz files to offline table
s
when I try to uplaod file from gcs into offline table from the above mentioned swagger rest api (http://localhost:9001/help#/Table/ingestFromURI).. then get the error:- { "_code": 500, "_error": "Caught exception when ingesting file into table: test2_OFFLINE. Illegal character ((CTRL-CHAR, code 31)): only regular white space (\\r, \\n, \\t) is allowed between tokens\n at [Source: (BufferedInputStream); line: 1, column: 2]" }
I have provided correct accesskey,secret key,offline table is available in sql query editor with schemaName given in table creation script
batch config given was
Copy code
{
  "inputFormat": "json",
  "input.fs.className": "org.apache.pinot.plugin.filesystem.GcsPinotFS",
  "input.fs.prop.region": "us-central1-c",
  "input.fs.prop.accessKey": "asd",
  "input.fs.prop.secretKey": "sd/asd/fd"
}
table name is test1_OFFLINE
offline table created using below script:- { "tableName": "test2", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "timeColumnName": "timestampInEpoch", "timeType": "MILLISECONDS", "schemaName": "audit_log_schema", "retentionTimeUnit": "DAYS", "retentionTimeValue": 365 }, "tenants": { "broker":"DefaultTenant", "server":"DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "metadata": {} }
response json had:- { "_code": 500, "_error": "Caught exception when ingesting file into table: test2_OFFLINE. Illegal character ((CTRL-CHAR, code 31)): only regular white space (\\r, \\n, \\t) is allowed between tokens\n at [Source: (BufferedInputStream); line: 1, column: 2]" }
@Xiang Fu @Mayank @Jackie
seems some bug while reading data from tar backup of realtime table stored on gcs as backup using the file ..the realtime table name was audit_log while offlien table name is test2_OFFLINEhttps://github.com/apache/pinot/blob/master/pinot-plugins/pinot-file-system/pinot-[…]rc/main/java/org/apache/pinot/plugin/filesystem/GcsPinotFS.java
cc: @Richard Startin @Daniel Lavoie @Seunghyun
I guess http://localhost:9001/help#/Table/ingestFromURI supports only flat normal files like csv,json.. not tar ..any way to upload tar on gcs into pinot table .. any tutorial link which might be helpful..
x
you need to untar the file. Also this API is not built for prod use cases, it’s just using your controller resources, for big data file, it may just fail because of out of memory
s
https://docs.pinot.apache.org/users/tutorials/batch-data-ingestion-in-practice is this the correct link for restore the tar generated from realtime table on gcs back into offline table
x
no, this is for generating segments from raw data
s
is there any link for my use case : restore the tar generated from realtime table on gcs back into offline table
what is the tar .. which format is data available on gcs .. is it parquet/avro/orc and what sort of compression is done viz. snappy/gzip etc
x
this is from raw data file, not segment tar file, you can use pinot ingestion job to push segments, change the job type from 
SegmentCreationAndTarPush
to 
SegmentTarPush
ideally you should back up your zookeeper then restore your cluster with it
s
any reference for the ingestion job with jobType SegmentTarPush..to restore the tar generated from realtime table on gcs back into offline table ..or it is not possible?
x
you can read pinot source code for the details
s
link of the segment push job type java class
its very difficult to understand from a beginner perspective since steps are not clear .. it will be very helpful if someone contributes in documentation of steps needed for performing this task from scratch cc: @Mayank
m
Thanks for the feedback @Sadim Nadeem, do you mind sharing which parts are lacking specifically, and what info would you like to see in that?
x
I think @Sadim Nadeem wants a table/segments restore story
s
yes .. steps needed for restoring the tar backup of realtime table from gcs into an offline table to get reporting using hybrid table after disaster recovery or migrating to another k8s namespace/cluster
Hi @Mayank is this documentation in pipeline / some way we can get it prioritized / do I need to add some issue for this on Apache Pinot Github
m
Thanks @Sadim Nadeem, the community can definitely use contributions, if you can document.
s
The steps are not clear to me for table/segments restore story for realtime segments into offline table from GCS and thus we were expecting some description on that .. If we were able to perform this task .. then will be more than happy to document it for others reference but facing blocker restoring and thus need some doc for help
Open in Slack
PreviousNext
Powered by