This message was deleted.

Hi team, I'm experiencing some issues in pinot

0.10.0

trying to rebalance an offline table after some offline-servers reached

dead

state (and have been replaced with new, healthy nodes), was hoping to get some extra 👀 on it.

Hi team, I'm experiencing an issue in 0.11.0, it's regarding this github issue: https://github.com/apache/pinot/issues/9520, even if I use single quotes to quote the literal, the error still exists. Can someone help take a look at it? Thanks

I have a question about adding dedupConfig to a table, can it be added to an existing table, or do you have to drop and recreate the table for it to take effect? We have an existing table we've added that config to. In testing, if we start with the table config / schema in thread as a new table, everything works as expected (records are deduped based on event_id). However, if that dedupConfig section is added to the existing table config (through the pinot UI), it doesn't dedupe and seems to ignore the config. I have verified that after adding the section, it shows up and no errors are given in the UI.

Hey, I’m getting

OutOfMemoryException Java Heap Space

durning batch ingestion. I have the same configs as in the previous thread. The only difference is that I use an 18GB CSV file. What is the best strategy to investigate that? Any table configuration might help?

This message was deleted.

hey friends do you all have tips on building pinot? specially in terms of how long it takes to build? pro tips, workarounds?

I am seeing that most of the errors are captured in the brokerResp rather than err variable. err is mostly null for all scenarios i have tested (segment error, column name error, table name error, query datatype error etc) hence it never enters err != nil block to handle any of these scenarios. Is this expected, shouldn’t all the errors be captured in err variable. How do I capture all the error scenarios. Thank you.

query = "select * from pmm_metrics limit 10"
brokerResp, err := pinotClient.ExecuteSQL("fact_device_ux_logs", query)
if err != nil {
		fmt.Println("Suraj Pinot error", err)
	}

&{[] <nil> 0x1400008e7d0 [{305 null: 4 segments [pmm_metrics__9__10__20221005T1320Z, pmm_metrics__2__10__20221005T2125Z, pmm_metrics__17__9__20221004T2140Z, pmm_metrics__6__10__20221005T1315Z] unavailable}] map[] 14 14 28 0 0 28 0 0 0 false 0 5 0} <nil>

Hi team , I have a doubt in startree in which I give a filter column which is not in any of the dimensions or measures but still the star tree works , but The details given by the pinot doc is

All aggregation function + column pairs in the query must exist in the StarTree index.
All dimensions that appear in filter predicates and group-by should be StarTree dimensions.

So how does star tree work in this scenario?

Hi 😉 I’m facing a bit of trouble with tasks. When I check the API task seems to be well scheduled but still in “progressing” state. Minion is up and running and yes I’m working on purgeFor REALTIME (commited segments only)

{
  "taskState": "IN_PROGRESS",
  "subtaskCount": {
    "total": 1,
    "completed": 0,
    "running": 0,
    "waiting": 1,
    "error": 0,
    "unknown": 0
  },
  "startTime": "2022-10-07 15:12:17 CEST",
  "subtaskInfos": [
    {
      "taskId": "Task_PurgeTask_0ebaea8f-e5ab-4f05-ab2a-d9d6a284eb98_1665148337022_0",
      "taskConfig": {
        "configs": {
          "uploadURL": "<http://localhost:9000/segments>",
          "crc": "3793535843",
          "downloadURL": "<http://localhost:9000/segments/quotations/quotations__1__0__20220920T1554Z>",
          "segmentName": "quotations__1__0__20220920T1554Z",
          "TASK_ID": "Task_PurgeTask_0ebaea8f-e5ab-4f05-ab2a-d9d6a284eb98_1665148337022_0",
          "tableName": "quotations_REALTIME"
        },
        "taskType": "PurgeTask",
        "taskId": "Task_PurgeTask_0ebaea8f-e5ab-4f05-ab2a-d9d6a284eb98_1665148337022_0"
      }
    }
  ]
}

Any Idea ? I’m in V 0.11.0 and I also do not see minionUI ... maybe a misconfiguration in my minion ? Any ideas are welcomed .

Hello Pinot friends. Is there any way to tell how much memory is being used by the steam dedup bookeeping of the primary key and the segment reference? The size of the in-memory map is what I'm looking for. Here's the documentation for reference.

Hey team, if I change replica from 1 to 2, and then rebalance servers, is there downtime during rebalancing?

questions; are the servers supposed to have as many cores as partitions on a kafka topic (?), if that’s the case what’s the best way to scale up pinot setups, the more tables we add the more the servers will have to ingest and we will get into resource contention given the cores on a server, how do you all manage this?

Hi Team, I have included this in the OFFLINE table config, and wanted to test how tasks work. Though the table and schema are getting created, I’m not seeing any tasks in Minion Task Manager.

"ingestionConfig": {
  "batchIngestionConfig": {
    "segmentIngestionType": "APPEND",
    "segmentIngestionFrequency": "HOURLY",
    "batchConfigMaps": [
      {
        "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
        "input.fs.prop.region": "us-east-1",
        "inputDirURI": "<s3://masked-bucket/dataset-sample/year=2022/month=10/day=10/>",
        "includeFileNamePattern": "glob:**/*.parquet",
        "excludeFileNamePattern": "glob:**/*.tmp",
        "inputFormat": "parquet"
      }
    ]
  }
},
"tasks": {
  "taskTypeConfigsMap": {
    "SegmentGenerationAndPushTask": {
      "schedule": "0 * * * * ?"
    }
  }
},

This is what is passed to controller through values.yaml, where the task scheduler is enabled. Can you please help me how are the tasks enabled?

extra:
  configs: |-
    pinot.set.instance.id.to.hostname=true
    controller.task.scheduler.enabled=true

So, this has happened twice. I ran the offline spark ingestion job. The job was successful. The logs say the response was a successful upload

] Response for pushing table enriched_click segment enriched_click_OFFLINE_2022-09-26_2022-09-26_3 to location <https://pinot.internal.com.sovrn.startree.cloud> - 200: {"status":"Su
ccessfully uploaded segment: enriched_click_OFFLINE_2022-09-26_2022-09-26_3 of table: enriched_click_OFFLINE"}

But, the data actually is not returned in the query console.

what could be happening here? It happened for different tables.

Can you check if count (*) without predicates returns anything? Also if this is hybrid table, can you explicitly query offline table? Also select dt column and check what values are stored, pick one of them and query

count(*) without any perdicates has data.

Ok, check others that I mentioned

What does that mean?

I currently don't have any data in my realtime table and the retention is 30 DAYS in my REAL TIME and 730 days in my OFFLINE

Hello 👋 I am having the following error with a realtime table and I am wondering what's the best course for debugging?

[
  {
    "message": "null:\n6 segments [tag__2__0__20221011T1015Z, tag__4__0__20221011T1015Z, tag__5__0__20221011T1015Z, tag__3__0__20221011T1015Z, tag__1__0__20221011T1015Z, tag__0__0__20221011T1015Z] unavailable",
    "errorCode": 305
  }
]

Also when clicking on the Bad status button, I have this

"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098": "ERROR"

hey friends I wonder if anyone has issues like this, we have a table and we want to ingest data from the beginning of the topic, however in the cluster while we are ingesting this data the servers get really really busy and the p99 response metrics for other tables get impacted greatly, has anyone gotten across this, do you all know what the bottleneck is and why servers get so impacted in terms of response times? it’s really weird that a change adding one table will impact the cluster so negatively.

Trying to run the airlineStats offline ingestion example on our internal cluster. Getting this error.

Caused by: org.apache.avro.AvroRuntimeException: Not a valid schema field: $ts$WEEK
	at org.apache.avro.generic.GenericData$Record.get(GenericData.java:256)
	at org.apache.pinot.plugin.inputformat.avro.AvroRecordExtractor.extract(AvroRecordExtractor.java:76)
	at org.apache.pinot.plugin.inputformat.avro.AvroRecordReader.next(AvroRecordReader.java:74)
	at org.apache.pinot.segment.local.segment.creator.RecordReaderSegmentCreationDataSource.gatherStats(RecordReaderSegmentCreationDataSource.java:66)
	at org.apache.pinot.segment.local.segment.creator.RecordReaderSegmentCreationDataSource.gatherStats(RecordReaderSegmentCreationDataSource.java:37)
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.init(SegmentIndexCreationDriverImpl.java:178)
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.init(SegmentIndexCreationDriverImpl.java:152)

Hi team, I got a pinot table ingesting topic of which partition number is reduced from 6 to 3, and 3 segments stop consuming data. Then what are the right steps taken in order to ingest data for this table?

Hi team, we still had Pinot (0.10.0) consuming from Kafka and yesterday noticed that it marked the table as BAD with the message “Ideal segment count:8 does not match external segment count: 454”. Looking at the logs I found the following message:

2022/10/12 02:01:04.242 INFO [PeriodicTaskScheduler] [pool-8-thread-5] Starting RetentionManager with running frequency of 21600 seconds.
2022/10/12 02:01:04.242 INFO [BasePeriodicTask] [pool-8-thread-5] [TaskRequestId: auto] Start running task: RetentionManager
2022/10/12 02:01:04.244 INFO [ControllerPeriodicTask] [pool-8-thread-5] Processing 1 tables in task: RetentionManager
2022/10/12 02:01:04.251 INFO [RetentionManager] [pool-8-thread-5] Start managing retention for table: events_REALTIME
2022/10/12 02:01:05.369 WARN [TimeRetentionStrategy] [pool-8-thread-5] Segment: events__1__105__20220929T1204Z of table: events_REALTIME has invalid end time in millis: 9011824788000
2022/10/12 02:01:05.370 INFO [RetentionManager] [pool-8-thread-5] Deleting 449 segments from table: events_REALTIME

Could that invalid end time have something to do with the state mismatch?

Hi team, I’m trying to upload a sample file using

ingestFromURI

. I’m using the following fields: tableNameWithType: foo_OFFLINE batchConfigMapStr:

{
  "inputFormat": "recordio",
  "input.fs.className": "com.company.mlutils.pinot.plugin.filesystem.object_store.ObjectStorePinotFS"
}

sourceURIStr: os://DATA/day=2022-10-12/hour=10/partition-310_foo_1665569284.recordio It fails with a 500 error

{
  "code": 500,
  "error": "Caught exception when ingesting file into table: foo_OFFLINE. Could not create directory for downloading input file locally: s3:/pinot-deep-store/segments/upload_dir/working_dir_foo_OFFLINE_1665613948057/input_data_dir"
}

and I see the following in the controller log:

2022/10/12 22:32:28.057 INFO [FileIngestionHelper] [jersey-server-managed-async-executor-5] Starting ingestion of URI payload to table: foo_OFFLINE using working dir: /opt/pinot/s3:/pinot-deep-store/segments/upload_dir/working_dir_foo_OFFLINE_1665613948057
2022/10/12 22:32:28.058 ERROR [FileIngestionHelper] [jersey-server-managed-async-executor-5] Caught exception when ingesting file to table: foo_OFFLINE
java.lang.IllegalStateException: Could not create directory for downloading input file locally: s3:/pinot-deep-store/segments/upload_dir/working_dir_foo_OFFLINE_1665613948057/input_data_dir
        at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.pinot.controller.util.FileIngestionHelper.buildSegmentAndPush(FileIngestionHelper.java:102) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestData(PinotIngestionRestletResource.java:200) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestFromURI(PinotIngestionRestletResource.java:175) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.lambda$apply$0(ResourceMethodInvoker.java:381) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:819) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:814) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]

What could be wrong here? Note that the controller is able to upload data to the S3 deep-store on the realtime table

Hi Team, I’m trying to use Dropwizard to replace Yammer as the metrics factory, and I found that the MBean names of the two are different. Do we have JMX Prometheus YAML that matches the Dropwizard MBean name pattern?

while connecting to tableau from pinot. getting this error while extracting. any help?