Apache Pinot #troubleshooting

Varagini Karthik

12/15/2022, 5:50 PM

Hi Team, I'm loading data to Pinot from S3 using

ingestionConfig

and

task

in S3 data loaded is in Parquet files, i have around 13M records at first 1.8M records are loading into the Pinot ... After sometime records are reducing and new records are not loading I checked the controller logs .... Im seeing the following

Copy code

Workflow TaskQueue_SegmentGenerationAndPushTask or job TaskQueue_SegmentGenerationAndPushTask_Task_SegmentGenerationAndPushTask_1670619980015 is already failed or completed, workflow state (IN_PROGRESS), job state (COMPLETED), clean up job IS.

Varagini Karthik

12/15/2022, 5:50 PM

any thoughts on this

Varagini Karthik

12/15/2022, 5:50 PM

I dont have any timeStamp column

Varagini Karthik

12/15/2022, 5:51 PM

Copy code

"ingestionConfig": {
      "batchIngestionConfig": {
        "batchConfigMaps": [
          {
            "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
            "input.fs.prop.region": "us-east-1",
            "input.fs.prop.secretKey": "fPqIP75GVu+2BzQ+u8FWRMkEqRkE7KewNYGOT68C",
            "input.fs.prop.accessKey": "AKIAWAYAOHLSW7CB5FH3",
            "inputDirURI": "<s3a://XXXX/XXX/XXX/XXX/XXX/XXX/>",
            "includeFileNamePattern": "glob:**/*.parquet",
            "excludeFileNamePattern": "glob:**/*.tmp",
            "inputFormat": "parquet"
          }
        ],
        "segmentIngestionType": "APPEND",
        "segmentIngestionFrequency": "HOURLY"
      }
    },
"task": {
      "taskTypeConfigsMap": {
        "SegmentGenerationAndPushTask": {
          "schedule": "5 * * * * ?",
          "tableMaxNumTasks": "10"
        }
      }
    }

Caleb Shei

12/15/2022, 8:01 PM

I am not able to query a table whenever one of the segments of the table is

Bad

. Is there a way for Pinot to ignore any

Bad

segment? This can happen very often especially when a new segment is just added to the OFFLINE table and during that short period of time the table becomes unaccessible. How can we guarantee 24/7 availability for a table_OFFLINE which needs to be updated frequently?

Caleb Shei

12/15/2022, 8:08 PM

Another related question is - Does Pinot support table alias? This allows us to switch the underline OFFLINE table to a newly reloaded one within sub second without interrupting any applications that constantly query the table via table alias.

Padma Malladi

12/15/2022, 9:26 PM

Hi, did anyone have to change the size of the AWS nodes in an EC2 cluster in order to vertically scale up the pinot servers? If so, can you share the experience?

Grace Lu

12/16/2022, 2:01 AM

Hi team, I am at pinot 0.10.0 where groovy is not disabled by default. Right now I want to disable it for security reason and I set

pinot.broker.disable.query.groovy=true

Copy code

root@pinot-broker-0:/var/pinot/broker/config# cat pinot-broker.conf 
pinot.set.instance.id.to.hostname=true
pinot.broker.disable.query.groovy=true

but I don’t see the query behavior changes clearly. I wonder if here is any easy way to look at what’s the current cluster config and verify if groovy is disabled or not?

vivek

12/16/2022, 5:53 AM

Hi, we are using Pinot 0.10.0. While using a third party library to construct SQL queries, one of the queries generated (simplified here) was

Copy code

select * from mytable
where col = 'foo' AND NOT (col = 'bar' AND col = 'foobar')

(col is multi-valued) While the above is a valid SQL statement (not necessarily the optimal one), Pinot returns an error for the above syntax. Looking at the CalciteSqlParser code, it looks like only BETWEEN and LIKE can be negated - is our understanding correct ? If this is indeed the case, are there plans to support the above syntax ? Thanks.

eywek

12/16/2022, 10:53 AM

Hello, I was wondering if there is any mechanism in Pinot to avoid filling up the disk? Is there any way for the controller and/or server to avoid downloading/creating anymore segment on a given server if the disk is full? Thank you

Mathieu Alexandre

12/16/2022, 4:43 PM

Hello 👋 in my pinot setup on k8s and segment store on azure blob storage, i've found a path on controller like

~/adl2\:/****<http://dfs.core.windows.net/data/upload_dir|dfs.core.windows.net/data/upload_dir>

. It seems to work as expected but how can i custom this ?

Bala S

12/18/2022, 11:02 AM

Hi I am trying with realtime table in Pinot to load stream of messages from my Kafka which is in JSON format but the string is getting truncated when I try to query the table and pick a selected column from the JSON Array. Do I need to set any configuration for such JSON based strings/queries.

Mahesh babu

12/19/2022, 12:37 PM

Hi Team, Facing issue while loading data through spark way but working through standalone way. I'm using transform function to transform TIMESTAMP facing issue there Error: Failed to decode Pinot schema from json string Caused by: shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type

org.apache.pinot.spi.data.FieldSpec$DataType

from String "TIMESTAMP": not one of the values accepted for Enum class: [STRING, STRUCT, BOOLEAN, BYTES, FLOAT, LIST, INT, DOUBLE, MAP, LONG]

chandarasekaran m

12/20/2022, 5:45 AM

Hi team , can we read fields from Kafka header now ?? Is that change pushed to master ?? Cc: @Kishore G

Fizza Abid

12/20/2022, 8:24 AM

Hi Team, we are facing this issue in latest image of apache pinot, it was pushed 6 hours ago. UI is not getting open, something wrong in main.js. Can someone help with this? It was working till yesterday.

chandarasekaran m

12/20/2022, 4:34 PM

I want to retrieve few fields from header My transformation looks like below

Copy code

"ingestionConfig": {
    "transformConfigs": [
      {
        "columnName": "entity_change_action",
        "transformFunction": "__header$entityChangeAction"
      },
      {
        "columnName": "company_id",
        "transformFunction": "__header$companyId"
      }
    ]
  },

chandarasekaran m

12/20/2022, 4:35 PM

but i am not able to retrive data from header

chandarasekaran m

12/20/2022, 4:35 PM

any sample config? i have added

"metadata.populate" : "true",

chandarasekaran m

12/20/2022, 4:36 PM

@Seunghyun

Luis Fernandez

12/20/2022, 5:59 PM

hey friends long time!! I’m alerting on this metric

pinot_server_freshnessLagMs_99thPercentile

after we upgraded to pinot 11 it hasn’t come down to normal levels and we are wondering why, or maybe if it will take sometime for it to recover

Pratik Tibrewal

12/20/2022, 8:35 PM

Hey, I face this error while building Pinot:

Copy code

[INFO] [ERROR] Failed to execute goal org.apache.pinot:pinot-fmpp-maven-plugin:0.11.0:generate (generate-fmpp-sources) on project pinot-common: Failed to apply the value of the "data" setting.
[INFO] [ERROR] 
[INFO] [ERROR] Caused by:
[INFO] [ERROR] TDD error: Failed to evaluate function "tdd".
[INFO] [ERROR] Error location: line 1, column 141:
[INFO] [ERROR] ...-common/src/main/codegen/config.fmpp), default:tdd...
[INFO] [ERROR]                                                   ^
[INFO] [ERROR] 
[INFO] [ERROR] Caused by:
[INFO] [ERROR] java.io.FileNotFoundException (at java.io.FileInputStream.open0): /mnt/jenkins/workspace/pinot-release/pinot-common/target/codegen/default_config.fmpp (No such file or directory)

Any help on how to resolve this?

Mithun Vigneswar Gunasekaran

12/21/2022, 5:55 AM

Hi, I am using a REALTIME upsert table and see that the records which have been updated and became obsolete are still in the server storage segments. Is there a configuration by which we can clean-up these obsolete records which have been updated? We have a lot of updates coming in on a part of the data and cleaning up will help us recover this storage. If this is not possible OOB in Pinot, can you help figure out if there is a way by which we can recognise these obsolete records so we can write a minion job to clean this up? cc: @Mayank @Navina

Nickel Fang

12/21/2022, 6:10 AM

Hi, I am using Batch import function to import a csv file to an offline table. But I got the error like this

Copy code

Caused by: java.lang.IllegalArgumentException: Cannot read single-value from Object[]: [WriteOff Invoice for members with Payment Failures<4,  AutoPay and RetryStatus reset] for column: type_name

Can anyone help me?

chandarasekaran m

12/21/2022, 6:37 AM

<!here> I am getting below error when i am using preseto

Copy code

presto:default> show tables;
Query 20221221_063458_00004_7h3r8 failed: java.io.UncheckedIOException: java.net.UnknownHostException: pinot-controller: Name or service not known

chandarasekaran m

12/21/2022, 6:38 AM

Screenshot 2022-12-21 at 12.07.58 PM.png

Mathieu Alexandre

12/21/2022, 11:09 AM

Hello is there a way to force segment completion in a realtime table ? In our pinot 0.9.3, few old segments persists in CONSUMING state even if the table_config specify realtime.segment.flush.threshold.time to 24h. (ingestion status is healthy in debug api)

Becker Ewing

12/21/2022, 9:50 PM

👋 Wanted to check whether the Kafka 2.x connector in Pinot release-0.11.0 will work with kafka version 3.2.3? Is a new connector implementation required or would just bumping the

kafka.lib.version

to 3.2.3 in the kafka 2.0 connector pom.xml work (as described here)?

Alice

12/22/2022, 2:11 AM

Hi team, we got OOM killed issues recently, and could locate the root cause. Could you help give some suggestion on how to trace it?

Driss

12/22/2022, 5:50 AM

hello folks, I am trying to run a pinot-cluster locally with docker-compose, and my objective is to pass s3 config to controller, broker and server. and I try to pass a configuration for the controller and server following this documentation: https://docs.pinot.apache.org/users/tutorials/use-s3-and-pinot-in-docker#prepare-pinot-configuration-files, but I have an issue with

controller.local.temp.dir

parameter in

controller.config

Pinot-controller returns:

Copy code

manual-pinot-controller | 2022/12/21 14:02:34.577 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 7.104 since launch
manual-pinot-controller | java.lang.RuntimeException: Caught exception while initializing ControllerFilePathProvider
manual-pinot-controller |       at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:539) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:364) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:312) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at 
org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:537) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       ... 20 more
...
...
...
manual-pinot-controller | Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: S3, Status Code: 301, Request ID: PKAYJT6MNQWPPS9S, Extended Request ID: Yfc9wNlJc1ML54/mMBgVAYJskUI2Lrk1/eECqs/G+gtMLntfDAiXtoQgv3vjvsFhmUDVr78hVbA=)
manual-pinot-controller |       at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[pinot-s3-0.9.3-shaded.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106) ~[pinot-s3-0.9.3-shaded.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[pinot-s3-0.9.3-shaded.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[pinot-s3-0.9.3-shaded.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:5614) ~[pinot-s3-0.9.3-shaded.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at org.apache.pinot.plugin.filesystem.S3PinotFS.isDirectory(S3PinotFS.java:514) ~[pinot-s3-0.9.3-shaded.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at org.apache.pinot.plugin.filesystem.S3PinotFS.exists(S3PinotFS.java:407) ~[pinot-s3-0.9.3-shaded.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:71) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:537) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller |       ... 20 more

Rohit Anilkumar

12/22/2022, 9:05 AM

I am running this query from the sql console

Copy code

select "organization",count(*) as total_count,
DATETIMECONVERT(
"time",
'1:SECONDS:EPOCH',
'1:HOURS:SIMPLE_DATE_FORMAT:yyyy-MM-dd HH:mm',
'1:DAYS'
) AS convertedTime
from linkpage group by "organization","convertedTime" order by convertedTime desc

But I am getting 10 rows only. There are more than 30 days of data in the server.