Mahesh babu
11/10/2022, 9:43 AMVaragini Karthik
11/10/2022, 2:14 PMjava.io.IOException: No space left on devicei
found the following exception
how to handle is ?Varagini Karthik
11/10/2022, 2:35 PMThomas Steinholz
11/10/2022, 2:55 PMxuyen
11/10/2022, 6:20 PMxuyen
11/10/2022, 6:21 PMPriyank Bagrecha
11/11/2022, 2:39 AMsuraj sheshadri
11/12/2022, 1:14 AMcheng
11/13/2022, 2:36 AM***************************************************
You can always go to <http://localhost:9000> to play around in the query console
Getting Helix leader: 192.168.0.165_9000, Helix version: 1.0.4, mtime: 1668291356337
Starting TaskMetricsEmitter with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: TaskMetricsEmitter
[TaskRequestId: auto] Finish running task: TaskMetricsEmitter in 3ms
Starting RetentionManager with running frequency of 21600 seconds.
[TaskRequestId: auto] Start running task: RetentionManager
Processing 6 tables in task: RetentionManager
Start managing retention for table: airlineStats_OFFLINE
Invalid retention time: null null for table: airlineStats_OFFLINE, skip
Segment lineage metadata clean-up is successfully processed for table: airlineStats_OFFLINE
Start managing retention for table: baseballStats_OFFLINE
Invalid retention time: null null for table: baseballStats_OFFLINE, skip
Segment lineage metadata clean-up is successfully processed for table: baseballStats_OFFLINE
Start managing retention for table: dimBaseballTeams_OFFLINE
Segment push type is not APPEND for table: dimBaseballTeams_OFFLINE, skip managing retention
Segment lineage metadata clean-up is successfully processed for table: dimBaseballTeams_OFFLINE
Start managing retention for table: starbucksStores_OFFLINE
Segment: starbucksStores_OFFLINE_0 of table: starbucksStores_OFFLINE has invalid end time in millis: -1
Segment lineage metadata clean-up is successfully processed for table: starbucksStores_OFFLINE
Start managing retention for table: githubEvents_OFFLINE
Segment push type is not APPEND for table: githubEvents_OFFLINE, skip managing retention
Segment lineage metadata clean-up is successfully processed for table: githubEvents_OFFLINE
Start managing retention for table: githubComplexTypeEvents_OFFLINE
Segment push type is not APPEND for table: githubComplexTypeEvents_OFFLINE, skip managing retention
Segment lineage metadata clean-up is successfully processed for table: githubComplexTypeEvents_OFFLINE
Removing aged deleted segments for all tables
Finish processing 6/6 tables in task: RetentionManager
[TaskRequestId: auto] Finish running task: RetentionManager in 11ms
Starting SegmentStatusChecker with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: SegmentStatusChecker
Processing 6 tables in task: SegmentStatusChecker
Reading segment sizes from 1 servers for table: airlineStats_OFFLINE with timeout: 30000ms
Finished reading information for table: airlineStats_OFFLINE
Reading segment sizes from 1 servers for table: baseballStats_OFFLINE with timeout: 30000ms
Finished reading information for table: baseballStats_OFFLINE
Reading segment sizes from 1 servers for table: dimBaseballTeams_OFFLINE with timeout: 30000ms
Finished reading information for table: dimBaseballTeams_OFFLINE
Reading segment sizes from 1 servers for table: starbucksStores_OFFLINE with timeout: 30000ms
Finished reading information for table: starbucksStores_OFFLINE
Reading segment sizes from 1 servers for table: githubEvents_OFFLINE with timeout: 30000ms
Finished reading information for table: githubEvents_OFFLINE
Reading segment sizes from 1 servers for table: githubComplexTypeEvents_OFFLINE with timeout: 30000ms
Finished reading information for table: githubComplexTypeEvents_OFFLINE
Finish processing 6/6 tables in task: SegmentStatusChecker
[TaskRequestId: auto] Finish running task: SegmentStatusChecker in 82ms
Getting Helix leader: 192.168.0.165_9000, Helix version: 1.0.4, mtime: 1668291356337
Starting SegmentRelocator with running frequency of 3600 seconds.
[TaskRequestId: auto] Start running task: SegmentRelocator
Processing 6 tables in task: SegmentRelocator
Finish processing 6/6 tables in task: SegmentRelocator
[TaskRequestId: auto] Finish running task: SegmentRelocator in 6ms
Starting OfflineSegmentIntervalChecker with running frequency of 3600 seconds.
[TaskRequestId: auto] Start running task: OfflineSegmentIntervalChecker
Processing 6 tables in task: OfflineSegmentIntervalChecker
Starting MinionInstancesCleanupTask with running frequency of 3600 seconds.
[TaskRequestId: auto] Start running task: MinionInstancesCleanupTask
[TaskRequestId: auto] Finish running task: MinionInstancesCleanupTask in 3ms
Finish processing 6/6 tables in task: OfflineSegmentIntervalChecker
[TaskRequestId: auto] Finish running task: OfflineSegmentIntervalChecker in 15ms
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/auth/info>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/cluster/info>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/cluster/configs>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/users>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables?type=realtime>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables?type=offline>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/airlineStats_OFFLINE/externalview>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/airlineStats_OFFLINE/idealstate>, content-type null status code 200 OK
Reading segment sizes from 1 servers for table: airlineStats_OFFLINE with timeout: 30000ms
Reading segment sizes from 1 servers for table: baseballStats_OFFLINE with timeout: 30000ms
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/baseballStats_OFFLINE/idealstate>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/dimBaseballTeams_OFFLINE/idealstate>, content-type null status code 200 OK
Finished reading information for table: baseballStats_OFFLINE
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/baseballStats_OFFLINE/size>, content-type null status code 200 OK
Finished reading information for table: airlineStats_OFFLINE
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/airlineStats_OFFLINE/size>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/dimBaseballTeams_OFFLINE/externalview>, content-type null status code 200 OK
Reading segment sizes from 1 servers for table: dimBaseballTeams_OFFLINE with timeout: 30000ms
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/baseballStats_OFFLINE/externalview>, content-type null status code 200 OK
Reading segment sizes from 1 servers for table: githubComplexTypeEvents_OFFLINE with timeout: 30000ms
Finished reading information for table: dimBaseballTeams_OFFLINE
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/dimBaseballTeams_OFFLINE/size>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/githubComplexTypeEvents_OFFLINE/idealstate>, content-type null status code 200 OK
Finished reading information for table: githubComplexTypeEvents_OFFLINE
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/githubComplexTypeEvents_OFFLINE/size>, content-type null status code 200 OK
Reading segment sizes from 1 servers for table: githubEvents_OFFLINE with timeout: 30000ms
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/githubEvents_OFFLINE/idealstate>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/githubComplexTypeEvents_OFFLINE/externalview>, content-type null status code 200 OK
Finished reading information for table: githubEvents_OFFLINE
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/githubEvents_OFFLINE/size>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/githubEvents_OFFLINE/externalview>, content-type null status code 200 OK
Reading segment sizes from 1 servers for table: starbucksStores_OFFLINE with timeout: 30000ms
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/starbucksStores_OFFLINE/idealstate>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/starbucksStores_OFFLINE/externalview>, content-type null status code 200 OK
Finished reading information for table: starbucksStores_OFFLINE
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/starbucksStores_OFFLINE/size>, content-type null status code 200 OK
...
...
[TaskRequestId: auto] Finish running task: RealtimeSegmentValidationManager in 2ms
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables>, content-type null status code 200 OK
Handled request from 0:0:0:0:0:0:0:1 GET <http://localhost:9000/tables/baseballStats/schema>, content-type null status code 200 OK
url string passed is : <http://192.168.0.165:8000/query/sql>
Processed
Getting Helix leader: 192.168.0.165_9000, Helix version: 1.0.4, mtime: 1668291356337
Handled request from 192.168.0.125 GET <http://192.168.0.165:9000/v2/brokers/tenants/DefaultTenant>, content-type application/json; charset=utf-8 status code 200 OK
Getting Helix leader: 192.168.0.165_9000, Helix version: 1.0.4, mtime: 1668291356337
Getting Helix leader: 192.168.0.165_9000, Helix version: 1.0.4, mtime: 1668291356337
Handled request from 192.168.0.125 GET <http://192.168.0.165:9000/v2/brokers/tenants/DefaultTenant>, content-type application/json; charset=utf-8 status code 200 OK
Getting Helix leader: 192.168.0.165_9000, Helix version: 1.0.4, mtime: 1668291356337
Getting Helix leader: 192.168.0.165_9000, Helix version: 1.0.4, mtime: 1668291356337
Starting TaskMetricsEmitter with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: TaskMetricsEmitter
[TaskRequestId: auto] Finish running task: TaskMetricsEmitter in 4ms
Starting SegmentStatusChecker with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: SegmentStatusChecker
Processing 6 tables in task: SegmentStatusChecker
Reading segment sizes from 1 servers for table: airlineStats_OFFLINE with timeout: 30000ms
Finished reading information for table: airlineStats_OFFLINE
Reading segment sizes from 1 servers for table: baseballStats_OFFLINE with timeout: 30000ms
Finished reading information for table: baseballStats_OFFLINE
Reading segment sizes from 1 servers for table: dimBaseballTeams_OFFLINE with timeout: 30000ms
Finished reading information for table: dimBaseballTeams_OFFLINE
Reading segment sizes from 1 servers for table: starbucksStores_OFFLINE with timeout: 30000ms
Finished reading information for table: starbucksStores_OFFLINE
Reading segment sizes from 1 servers for table: githubEvents_OFFLINE with timeout: 30000ms
Finished reading information for table: githubEvents_OFFLINE
Reading segment sizes from 1 servers for table: githubComplexTypeEvents_OFFLINE with timeout: 30000ms
Sumit Khaitan
11/13/2022, 8:28 AMno space left on the device
error on controller while trying to push the segment. When I checked the disk usage on controller, seems like /var/pinot/controller/data/<TABLE_NAME>
path is storing the segment files. Shouldn't segment be only stored on server and not on controller ?cheng
11/13/2022, 10:49 PMcheng
11/13/2022, 10:58 PMPratik Tibrewal
11/14/2022, 5:49 AMLong
in Pinot table and the corresponding kafka topic. Somehow the data in kafka topic seems to be fine but in the table, I am always getting -9223372036854775808
in the new column in Pinot. Any suggestions on what might be the cause for this?Mahesh babu
11/14/2022, 10:36 AMvmarchaud
11/14/2022, 5:18 PMSELECT groovy('{"returnType":"STRING","isSingleValue":true}', 'arg0.toList().join('';'')', JSONEXTRACTKEY("labels", '$.*')) FROM datasource_609bc534f46c000300b29dcf_REALTIME WHERE (("timestamp" >= 1667833781565)) AND "labels" != '{}' GROUP BY 1 LIMIT 0,100
sunny
11/15/2022, 1:33 AMjava.lang.NullPointerException: null value in entry: Server_pay-poc-pinot-w4.ay1.krane.9rum.cc_8001=null
at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
at com.google.common.collect.SingletonImmutableBiMap.<init>(SingletonImmutableBiMap.java:43)
at com.google.common.collect.ImmutableBiMap.of(ImmutableBiMap.java:81)
at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:128)
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:708)
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:686)
at io.trino.plugin.pinot.PinotSegmentPageSource.queryPinot(PinotSegmentPageSource.java:221)
at io.trino.plugin.pinot.PinotSegmentPageSource.fetchPinotData(PinotSegmentPageSource.java:182)
at io.trino.plugin.pinot.PinotSegmentPageSource.getNextPage(PinotSegmentPageSource.java:150)
at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:311)
at io.trino.operator.Driver.processInternal(Driver.java:410)
at io.trino.operator.Driver.lambda$process$10(Driver.java:313)
at io.trino.operator.Driver.tryWithLock(Driver.java:698)
at io.trino.operator.Driver.process(Driver.java:305)
at io.trino.operator.Driver.processForDuration(Driver.java:276)
at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1092)
at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
at io.trino.$gen.Trino_385____20221110_083442_2.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
For reference, the error does not occur in pinot version 0.10.0. In pinot version 0.10.0, the default value of pinot.server.instance.currentDataTableVersion is 2, so it is expected that there is no problem.
And I changed the pinot server configuration(pinot.server.instance.currentDataTableVersion=2) and the trino query succeeded.
I know that the settings in the commit below have disappeared from the helm chart settings. I think this commit is related to the above query error, but I don’t understand the commit message. Can you give me an explanation of what that means?
This is no longer needed as presto/trino side has upgraded the DataTable version
https://github.com/apache/pinot/pull/9255Nickel Fang
11/15/2022, 9:52 AM"segmentPartitionConfig": {
"columnPartitionMap": {
"trace_id": {
"functionName": "Murmur",
"numPartitions": 8
}
}
},
From the Kafka side, I set trace_id as key when producing an event message. And there are 8 partitions of the topic.
when I get the segment meta data, it seems not correspond to one partition.
"segment.partition.metadata": "{\"columnPartitionMap\":{\"trace_id\":{\"numPartitions\":8,\"partitions\":[0,2,4,6],\"functionName\":\"Murmur\",\"functionConfig\":null}}}",
Luis Fernandez
11/15/2022, 4:27 PMkurt
11/15/2022, 5:59 PM"dateTimeFieldSpecs": [{
"name": "day",
"dataType": "STRING",
"format" : "SIMPLE_DATE_FORMAT|yyyy-MM-dd",
"granularity": "1:DAYS"
}]
I get this error message: invalid datetime format: SIMPLE_DATE_FORMAT|yyyy-MM-dd
I also try the specific format examples given in the docs like SIMPLE_DATE_FORMAT|yyyy-MM-dd HH:mm:ss
and SIMPLE_DATE_FORMAT|yyyy-MM-dd|IST
and get the same invalid datetime format
error.
I believe I’m running the latest version of Apache Pinot. I just installed via the official Helm chart from the github repo. The Pinot pods are running apachepinot/pinot:latest-jdk11
. Is there any way I can confirm what version of Pinot I’m using with the admin tool or with the web GUI?Priyank Bagrecha
11/15/2022, 8:04 PMINFO
There are a couple of scenarios where segments in offline tables won't be purged:
* If the segment doesn't have an end time. This would happen if the segment doesn't contain a time column.
* If the segment's table has a segmentIngestionType of REFRESH.
In addition, segments will not be purged in real-time or offline tables if the retention period isn't specified.
In my case, I have an offline table which doesn't have a time column. The table has a segment ingestion type of refresh and there is NO retention period configured on the table. Is this a reason for segments not getting deleted from the disc when invoking delete all segments api?Ehsan Irshad
11/16/2022, 6:30 AMLoïc Mathieu
11/16/2022, 11:38 AMMathieu Alexandre
11/16/2022, 1:53 PMadls
as deepstore and image release-0.11.0
?
Caused by: com.azure.storage.file.datalake.models.DataLakeStorageException: Status code 412, "{"error":{"code":"ConditionNotMet","message":"The condition specified using HTTP conditional header(s) is not met.
Ralph Debusmann
11/16/2022, 2:35 PMRalph Debusmann
11/16/2022, 2:37 PMRalph Debusmann
11/16/2022, 2:38 PMRalph Debusmann
11/16/2022, 2:38 PMRalph Debusmann
11/16/2022, 2:38 PMRalph Debusmann
11/16/2022, 3:56 PMThomas Steinholz
11/16/2022, 6:12 PMorder by
clause using the time column of the table. The query ends up timing out for tables with more than hundreds of millions of rows when requesting more than a limit of 10s of records. For example, a query with a limit of 100 or 1000s will return with pinot server not-responded error. Are there any suggestions for improving the limit size of the order by
operator on the time column of the table?