https://pinot.apache.org/ logo
b

Brian Olsen

04/10/2021, 2:00 AM
same issue...i'm gonna update the default null values I think...it's only for a demo
m

Mayank

04/10/2021, 2:08 AM
If also try the default null value of 0 millisSinceEpoch.
b

Brian Olsen

04/10/2021, 3:24 AM
Copy code
Driver, record read time : 50771
Driver, stats collector time : 0
Driver, indexing time : 20623
Tarring segment from: /tmp/pinot-9a3c12a1-8374-45b0-9d12-7fbd8a2a2dee/output/covid_cases_OFFLINE_1971/01/01_2021/03/30_0 to: /tmp/pinot-9a3c12a1-8374-45b0-9d12-7fbd8a2a2dee/output/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0.tar.gz
Size for segment: covid_cases_OFFLINE_1971/01/01_2021/03/30_0, uncompressed: 135.04M, compressed: 52.17M
Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Start pushing segments: [/tmp/pinot-quick-start/segments/covid_cases_OFFLINE_1971/01/01_2021/03/30_0.tar.gz]... to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@2d778add] for table covid_cases
Pushing segment: 30_0 to location: <http://pinot-controller:9000> for table covid_cases
Sending request: <http://pinot-controller:9000/v2/segments?tableName=covid_cases> to controller: pinot-controller, version: Unknown
Response for pushing table covid_cases segment 30_0 to location <http://pinot-controller:9000> - 200: {"status":"Successfully uploaded segment: covid_cases_OFFLINE_1971/01/01_2021/03/30_0 of table: covid_cases"}
Got to this portion and have uploaded twice now and both times the segment says status bad. Any ideas?
m

Mayank

04/10/2021, 3:26 AM
The log above indicates that the push was successful. Do you have access to server logs to see what happeneed when the server tried to load it?
b

Brian Olsen

04/10/2021, 3:55 AM
Kind of a confusing message:
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Failed to get reason" while sending request: <http//&lt;ip address&gt;9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0>
j

Jackie

04/10/2021, 5:21 AM
The segment is rejected because
1970/01/01
is not in the valid time range. We allow time from
1971/01/01
to
2071/01/01
to prevent wrong format time being ingested
x

Xiang Fu

04/10/2021, 9:25 AM
I think this segment is successfully uploaded ,so you can find it through ui
server log shows that it’s failed to download the segment from the controller:
Copy code
Got temporary error status code: 500 while downloading segment from: <http://172.18.0.6:9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0> to: /tmp/pinotSegments/covid_cases_OFFLINE/tmp-covid_cases_OFFLINE_1971/01/01_2021/03/30_0-86673f59-1b0d-4a29-8c03-c03e772d182c/covid_cases_OFFLINE_1971/01/01_2021/03/30_0.enc
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Failed to get reason" while sending request: <http://172.18.0.6:9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0>
wondering where you run this setup? seems local deployment?
can you also try to make a call to
Copy code
<http://172.18.0.6:9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0>
and see if you can download this segment
Push job says
Copy code
Pushing segment: 30_0 to location: <http://pinot-controller:9000> for table covid_cases
Sending request: <http://pinot-controller:9000/v2/segments?tableName=covid_cases> to controller: pinot-controller, version: Unknown
Response for pushing table covid_cases segment 30_0 to location <http://pinot-controller:9000> - 200: {"status":"Successfully uploaded segment: covid_cases_OFFLINE_1971/01/01_2021/03/30_0 of table: covid_cases"}
so I think this means you are still on k8s.
then can you check segment metadata and see the downloadURL there
also for pinot controller ,can you check if u have this config in the
values.yaml
file:
Copy code
pinot.set.instance.id.to.hostname=true
It should make pinot controller use hostname other than ip address to register itself
b

Brian Olsen

04/10/2021, 11:02 AM
@Jackie yeah, I eventually fixed that to 1971 and then got this issue. As @Xiang Fu points out it is updated in the UI to see the segment but it never downloads.
This is running on an ec2 host running docker compose and sharing resources with mongo/Trino so might as well be a local setup 🙂
I’ll try what you said in a moment Xiang! Thanks 🙏
@Xiang Fu, I just noticed that you said this.
so I think this means you are still on k8s.
I'm trying to set this up in docker compose right now...what is the default for this property if I haven't set it for other nodes?
controller.helix.cluster.name
Here are the settings in docker compose
Copy code
pinot-controller:
    image: apachepinot/pinot:latest
    hostname: pinot-controller
    volumes:
      - ./pinot-docker-demo/pinot/controller:/tmp/data/controller
      - ./conf:/opt/pinot/conf
    ports:
      - "9000:9000"
    command: StartController -configFileName /opt/pinot/conf/pinot-controller.conf
    depends_on:
      - zookeeper
      - kafka
    networks:
      - trino-network

  pinot-broker:
    image: apachepinot/pinot:latest
    hostname: pinot-broker
    ports:
      - "8099:8099"
    command: StartBroker -zkAddress zookeeper:2181
    depends_on:
      - zookeeper
      - kafka
      - pinot-controller
    networks:
      - trino-network

  pinot-server:
    image: apachepinot/pinot:latest
    hostname: pinot-server
    volumes:
      - ./pinot-docker-demo/pinot/server:/tmp/data/server
    ports:
      - "8098:8098"
    command: StartServer -zkAddress zookeeper:2181
    depends_on:
      - zookeeper
      - kafka
      - pinot-controller
    networks:
      - trino-network
x

Xiang Fu

04/10/2021, 7:03 PM
oic, then in this
./conf/pinot-controller.conf
, can you add that config :
<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
also for
pinot-broker.conf
and
pinot-server.conf
b

Brian Olsen

04/10/2021, 7:07 PM
got it, will add for all 🙂
x

Xiang Fu

04/10/2021, 7:11 PM
Sure. You can verify by looking at the instance name. It should give Pinot-controller instead of IP address
b

Brian Olsen

04/10/2021, 7:23 PM
id is now
Copy code
{
  "id": "Controller_pinot-controller_9000",
running the batch job again 🤞
x

Xiang Fu

04/10/2021, 7:23 PM
🤞
b

Brian Olsen

04/10/2021, 7:38 PM
hmm, still no luck...
segment is still showing up as bad...I think I must be doing something off like one setting I missed or something
x

Xiang Fu

04/10/2021, 7:43 PM
hmm what’s the download uri this time?
b

Brian Olsen

04/10/2021, 7:53 PM
Copy code
Copying segment from /var/pinot/controller/data/pinot-controller_9000/fileUploadTemp/tmp-f4f3ebd3-0a20-4438-a303-940dd63e4aa5 to file:/var/pinot/controller/data/covid_cases/covid_cases_OFFLINE_2020%2F10%2F23_2021%2F01%2F28_0
Moved segment covid_cases_OFFLINE_2020/10/23_2021/01/28_0 from temp location /var/pinot/controller/data/pinot-controller_9000/fileUploadTemp/tmp-f4f3ebd3-0a20-4438-a303-940dd63e4aa5 to /var/pinot/controller/data/covid_cases/covid_cases_OFFLINE_2020/10/23_2021/01/28_0
Added segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 of table: covid_cases_OFFLINE to property store
Assigning segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 with instance partitions: {"instancePartitionsName":"covid_cases_OFFLINE","partitionToInstancesMap":{"0_0":["Server_pinot-server_8098"]}} for table: covid_cases_OFFLINE
Assigned segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to instances: [Server_pinot-server_8098] for table: covid_cases_OFFLINE
Assigning segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to instances: [Server_pinot-server_8098] for table: covid_cases_OFFLINE
Added segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to IdealState for table: covid_cases_OFFLINE
Handled request from 172.18.0.9 POST <http://pinot-controller:9000/v2/segments?tableName=covid_cases>, content-type multipart/form-data; boundary=XUAVzKwU1hHzxxiabTHvatwtQoWL9O1dwVIMCm status code 200 OK
Looks like that got fixed actually
hmmm
Anything off in my table config?
Copy code
{
  "OFFLINE": {
    "tableName": "covid_cases_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "timeType": "MILLISECONDS",
      "schemaName": "covid_cases",
      "timeColumnName": "cdc_report_dt",
      "replication": "1"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": true,
      "invertedIndexColumns": [],
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "loadMode": "MMAP"
    },
    "metadata": {},
    "isDimTable": false
  }
}
x

Xiang Fu

04/10/2021, 7:53 PM
ok, so it means it’s still server cannot download it ?
looks ok
can you click segment metadata and look at the downloadURI field
I think that one should be fixed as well
b

Brian Olsen

04/10/2021, 7:56 PM
I try clicking on segment and it fails
brings me to the homescreen
I have to run out for 15 min i'll be right back
x

Xiang Fu

04/10/2021, 7:58 PM
hmm it should send you to this:
and shows
Copy code
"segment.offline.download.url": "<http://192.168.86.73:9000/segments/baseballStats/baseballStats_OFFLINE_0>"
also can you try to set
timeType
to
SIMPLE_DATE_FORMAT:yyyy/MM/dd
in your table config
b

Brian Olsen

04/10/2021, 9:51 PM
had to take this offline with xiang..will update with answer