Hi Team i am trying to use ` segments tableName startReplace Apache Pinot #general

Hi Team, i am trying to use `/segments/{tableName}...

vishal

02/16/2023, 1:02 PM

Hi Team, i am trying to use

/segments/{tableName}/startReplaceSegments

API in swagger api and trying to replace segment here i am trying to replace one segment with other but i am not bale to do it. i am passing

segmentsFrom

those segment which we want to replace with new one.

segmentsTo

is the other segment but before that i have to create and upload that segment. how to create and upload it? i tried to use segment from same table but getting a error as below:

Copy code

Input:
{
    "segmentsTo":["tmptab__4__1__20230214T1305Z"],
    "segmentsFrom":["tmptab__0__0__20230214T0611Z"]
    }

Copy code

{
  "code": 500,
  "error": "Any segments from 'segmentsTo' should not be available in the table at this point. (tableName = 'tmptab_REALTIME', segmentsFrom = '[tmptab__0__0__20230214T0611Z]', segmentsTo = '[tmptab__4__1__20230214T1305Z]', segmentsFromTable = '[tmptab__0__1__20230214T1305Z, tmptab__2__1__20230214T1305Z, tmptab__3__1__20230214T1305Z, tmptab__0__0__20230214T0611Z, tmptab__1__1__20230214T1305Z, tmptab__4__1__20230214T1305Z, tmptab__5__1__20230214T1305Z, tmptab__2__0__20230214T0611Z, tmptab__1__0__20230214T0611Z, tmptab__4__0__20230214T0611Z, tmptab__3__0__20230214T0611Z, tmptab__5__0__20230214T0611Z]')"
}

@saurabh dubey @Xiang Fu @Tim Santos @Xiaobing @Jackie @Kartik Khare

saurabh dubey

02/16/2023, 1:15 PM

You should first hit

/segments/{tableName}/startReplaceSegments

API, then upload the new segments to the table, then hit the

/segments/{tableName}/endReplaceSegments

API

vishal

02/16/2023, 1:16 PM

but for startreplacesegment i have to define sefmentto and segmentfrom right?

saurabh dubey

02/16/2023, 1:17 PM

Yes. But that should be before you push to segment to the table. It needs just the segment names

vishal

02/16/2023, 1:18 PM

so i need to define segmentfrom only?

saurabh dubey

02/16/2023, 1:19 PM

you need to set both, just that you should startReplaceSegments -> upload the new segments (the ones in segmentsTo) -> endReplaceSegments This order

vishal

02/16/2023, 1:20 PM

can i upload through s3?

saurabh dubey

02/16/2023, 1:23 PM

Do you mean metadata push? Yes you can use that

vishal

02/16/2023, 1:25 PM

using SegmentGenerationAndPushTask task

saurabh dubey

02/16/2023, 1:38 PM

if you can determine the segments names beforehand (maybe by using fixed segment name generator type?), you can manually first hit the startReplace API with the name(s) in the from and to, save the id returned in the reponse somewhere, then run your SegmentGenerationAndPushTask task, and finally manually hit the endSegmentReplace API in the end with the id from first API, it could work. But generally speaking, no SegmentGenerationAndPushTask is not designed to replace segments like that

saurabh dubey

02/16/2023, 1:38 PM

@Haitao Zhang might have more context

vishal

02/16/2023, 1:39 PM

okay is there any easy way to upload segment?

Haitao Zhang

02/17/2023, 7:45 AM

do you mean replacing segments?

vishal

02/17/2023, 9:31 AM

yes correct @Haitao Zhang

vishal

02/17/2023, 10:10 AM

Copy code

curl -X POST -F segment=@<segment-tar-file-path> <http://controllerHost>:controllerPort/segments

i am trying to use above curl request to create a segment but getting a error that file format is not .gz

vishal

02/17/2023, 10:11 AM

i've created csv file and than converted to .tar.gz format but it is not working

saurabh dubey

02/17/2023, 10:27 AM

What is the API error message?

vishal

02/17/2023, 10:27 AM

Copy code

{
  "code": 500,
  "error": "Exception while uploading segment: Input is not in the .gz format"
}

saurabh dubey

02/17/2023, 10:34 AM

How was the segment created?

vishal

02/17/2023, 10:35 AM

created simple csv file

vishal

02/17/2023, 10:36 AM

than converted it to .tar.gz format

saurabh dubey

02/17/2023, 10:37 AM

No you need to create an actual pinot segment from the file first

vishal

02/17/2023, 10:37 AM

than how can i create it?

vishal

02/17/2023, 11:06 AM

https://docs.pinot.apache.org/users/tutorials/create-pinot-segments#pushing-offline-segments-to-pinot

vishal

02/17/2023, 11:06 AM

trying to follow this link

vishal

02/17/2023, 11:06 AM

but non of them working

saurabh dubey

02/17/2023, 11:09 AM

You should first run this

Copy code

bin/pinot-admin.sh CreateSegment -dataDir <input_data_dir> [-format [CSV/JSON/AVRO]] [-readerConfigFile <csv_config_file>] [-generatorConfigFile <generator_config_file>] -segmentName <segment_name> -schemaFile <input_schema_file> -tableName <table_name> -outDir <output_data_dir> [-overwrite]

Maybe you can run this inside some pinot container. This will generate your segment file. Then push that file using

Copy code

curl -X POST -F segment=@<segment-tar-file-path> <http://controllerHost>:controllerPort/segments

vishal

02/17/2023, 11:09 AM

bin/pinot-admin.sh CreateSegment -dataDir /home/vishalvanpariya/work/tmp/datadir/ -format CSV -schemaFile /home/vishalvanpariya/work/pinot-poc/s3/json-s3/schema.json -outDir output/ tried this command but returning error

vishal

02/17/2023, 11:10 AM

Copy code

java.lang.IllegalArgumentException: 'tableConfigFile' must be specified

saurabh dubey

02/17/2023, 11:10 AM

Add this? -tableName <table_name>

vishal

02/17/2023, 11:10 AM

bin/pinot-admin.sh CreateSegment -dataDir /home/vishalvanpariya/work/tmp/datadir/ -format CSV -segmentName json_test_OFFLINE_2 -schemaFile /home/vishalvanpariya/work/pinot-poc/s3/json-s3/schema.json -tableName json_test_OFFLINE -outDir output/

vishal

02/17/2023, 11:11 AM

its here

vishal

02/17/2023, 11:11 AM

Copy code

picocli.CommandLine$UnmatchedArgumentException: Unknown options: '-segmentName', 'json_test_OFFLINE_2', '-tableName', 'json_test_OFFLINE'

saurabh dubey

02/17/2023, 11:11 AM

What version are you on?

vishal

02/17/2023, 11:12 AM

0.13.0 code from github

saurabh dubey

02/17/2023, 11:15 AM

Try this

Copy code

bin/pinot-admin.sh CreateSegment -dataDir /home/vishalvanpariya/work/tmp/datadir/ -format CSV -schemaFile /home/vishalvanpariya/work/pinot-poc/s3/json-s3/schema.json -tableConfigFile <table config json as a file> -outDir output/

saurabh dubey

02/17/2023, 11:15 AM

table config you should be able to get using controller, copy it into a file

vishal

02/17/2023, 11:40 AM

its working

vishal

02/17/2023, 11:40 AM

but i am not able to set segment name

vishal

02/17/2023, 11:40 AM

-segmentName

vishal

02/17/2023, 11:40 AM

if i use this param it returning error

vishal

02/17/2023, 11:40 AM

Copy code

picocli.CommandLine$UnmatchedArgumentException: Unknown options: '-segmentName', 'json_test_OFFLINE_1'

saurabh dubey

02/17/2023, 11:41 AM

You can get the file name from the output/ dir?

vishal

02/17/2023, 11:43 AM

yes i can see the name

vishal

02/17/2023, 11:44 AM

but i want to set perticular name

vishal

02/17/2023, 11:44 AM

want to automate process

saurabh dubey

02/17/2023, 11:45 AM

typically, the segment replace protocol usage involved first building the segment, getting the generated name, calling the /startSegmentReplca with the from and the to list (to list will have the segment name that was just generated), then call /endSegmentReplace. Also, typically you'd want to do this via minion tasks

vishal

02/17/2023, 11:59 AM

yeah, i did same way only but as per that doc we can define segment name as well but it is not working

saurabh dubey

02/17/2023, 12:16 PM

Yes the documentations seems outdated. Will get that checked

vishal

02/21/2023, 6:42 AM

@saurabh dubey can we create segment through using docker command?

vishal

02/21/2023, 6:42 AM

Copy code

docker run --rm -ti \
    --network=pinot-demo \
    --name pinot-minion \
    -p 8097:8097 \
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log" \
    -d apachepinot/pinot:0.12.0 StartMinion \
    -zkAddress pinot-zookeeper:2181

lets say we are starting minion as above can we create segment like this?

3 Views

Open in Slack

Previous Next