Brian Olsen
04/10/2021, 2:00 AMMayank
Brian Olsen
04/10/2021, 3:24 AMDriver, record read time : 50771
Driver, stats collector time : 0
Driver, indexing time : 20623
Tarring segment from: /tmp/pinot-9a3c12a1-8374-45b0-9d12-7fbd8a2a2dee/output/covid_cases_OFFLINE_1971/01/01_2021/03/30_0 to: /tmp/pinot-9a3c12a1-8374-45b0-9d12-7fbd8a2a2dee/output/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0.tar.gz
Size for segment: covid_cases_OFFLINE_1971/01/01_2021/03/30_0, uncompressed: 135.04M, compressed: 52.17M
Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Start pushing segments: [/tmp/pinot-quick-start/segments/covid_cases_OFFLINE_1971/01/01_2021/03/30_0.tar.gz]... to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@2d778add] for table covid_cases
Pushing segment: 30_0 to location: <http://pinot-controller:9000> for table covid_cases
Sending request: <http://pinot-controller:9000/v2/segments?tableName=covid_cases> to controller: pinot-controller, version: Unknown
Response for pushing table covid_cases segment 30_0 to location <http://pinot-controller:9000> - 200: {"status":"Successfully uploaded segment: covid_cases_OFFLINE_1971/01/01_2021/03/30_0 of table: covid_cases"}
Got to this portion and have uploaded twice now and both times the segment says status bad. Any ideas?Mayank
Brian Olsen
04/10/2021, 3:55 AMorg.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Failed to get reason" while sending request: <http//<ip address>9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0>
Jackie
04/10/2021, 5:21 AM1970/01/01
is not in the valid time range. We allow time from 1971/01/01
to 2071/01/01
to prevent wrong format time being ingestedXiang Fu
Got temporary error status code: 500 while downloading segment from: <http://172.18.0.6:9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0> to: /tmp/pinotSegments/covid_cases_OFFLINE/tmp-covid_cases_OFFLINE_1971/01/01_2021/03/30_0-86673f59-1b0d-4a29-8c03-c03e772d182c/covid_cases_OFFLINE_1971/01/01_2021/03/30_0.enc
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Failed to get reason" while sending request: <http://172.18.0.6:9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0>
<http://172.18.0.6:9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0>
and see if you can download this segmentPushing segment: 30_0 to location: <http://pinot-controller:9000> for table covid_cases
Sending request: <http://pinot-controller:9000/v2/segments?tableName=covid_cases> to controller: pinot-controller, version: Unknown
Response for pushing table covid_cases segment 30_0 to location <http://pinot-controller:9000> - 200: {"status":"Successfully uploaded segment: covid_cases_OFFLINE_1971/01/01_2021/03/30_0 of table: covid_cases"}
so I think this means you are still on k8s.values.yaml
file:
pinot.set.instance.id.to.hostname=true
It should make pinot controller use hostname other than ip address to register itselfBrian Olsen
04/10/2021, 11:02 AMso I think this means you are still on k8s.I'm trying to set this up in docker compose right now...what is the default for this property if I haven't set it for other nodes?
controller.helix.cluster.name
pinot-controller:
image: apachepinot/pinot:latest
hostname: pinot-controller
volumes:
- ./pinot-docker-demo/pinot/controller:/tmp/data/controller
- ./conf:/opt/pinot/conf
ports:
- "9000:9000"
command: StartController -configFileName /opt/pinot/conf/pinot-controller.conf
depends_on:
- zookeeper
- kafka
networks:
- trino-network
pinot-broker:
image: apachepinot/pinot:latest
hostname: pinot-broker
ports:
- "8099:8099"
command: StartBroker -zkAddress zookeeper:2181
depends_on:
- zookeeper
- kafka
- pinot-controller
networks:
- trino-network
pinot-server:
image: apachepinot/pinot:latest
hostname: pinot-server
volumes:
- ./pinot-docker-demo/pinot/server:/tmp/data/server
ports:
- "8098:8098"
command: StartServer -zkAddress zookeeper:2181
depends_on:
- zookeeper
- kafka
- pinot-controller
networks:
- trino-network
Xiang Fu
./conf/pinot-controller.conf
, can you add that config : <http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
pinot-broker.conf
and pinot-server.conf
Brian Olsen
04/10/2021, 7:07 PMXiang Fu
Brian Olsen
04/10/2021, 7:23 PM{
"id": "Controller_pinot-controller_9000",
running the batch job again 🤞Xiang Fu
Brian Olsen
04/10/2021, 7:38 PMXiang Fu
Brian Olsen
04/10/2021, 7:53 PMCopying segment from /var/pinot/controller/data/pinot-controller_9000/fileUploadTemp/tmp-f4f3ebd3-0a20-4438-a303-940dd63e4aa5 to file:/var/pinot/controller/data/covid_cases/covid_cases_OFFLINE_2020%2F10%2F23_2021%2F01%2F28_0
Moved segment covid_cases_OFFLINE_2020/10/23_2021/01/28_0 from temp location /var/pinot/controller/data/pinot-controller_9000/fileUploadTemp/tmp-f4f3ebd3-0a20-4438-a303-940dd63e4aa5 to /var/pinot/controller/data/covid_cases/covid_cases_OFFLINE_2020/10/23_2021/01/28_0
Added segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 of table: covid_cases_OFFLINE to property store
Assigning segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 with instance partitions: {"instancePartitionsName":"covid_cases_OFFLINE","partitionToInstancesMap":{"0_0":["Server_pinot-server_8098"]}} for table: covid_cases_OFFLINE
Assigned segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to instances: [Server_pinot-server_8098] for table: covid_cases_OFFLINE
Assigning segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to instances: [Server_pinot-server_8098] for table: covid_cases_OFFLINE
Added segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to IdealState for table: covid_cases_OFFLINE
Handled request from 172.18.0.9 POST <http://pinot-controller:9000/v2/segments?tableName=covid_cases>, content-type multipart/form-data; boundary=XUAVzKwU1hHzxxiabTHvatwtQoWL9O1dwVIMCm status code 200 OK
Looks like that got fixed actually{
"OFFLINE": {
"tableName": "covid_cases_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"timeType": "MILLISECONDS",
"schemaName": "covid_cases",
"timeColumnName": "cdc_report_dt",
"replication": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": true,
"invertedIndexColumns": [],
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"loadMode": "MMAP"
},
"metadata": {},
"isDimTable": false
}
}
Xiang Fu
Brian Olsen
04/10/2021, 7:56 PMXiang Fu
"segment.offline.download.url": "<http://192.168.86.73:9000/segments/baseballStats/baseballStats_OFFLINE_0>"
timeType
to SIMPLE_DATE_FORMAT:yyyy/MM/dd
in your table configBrian Olsen
04/10/2021, 9:51 PM