Santosh
05/25/2021, 4:42 PMbin/pinot-admin.sh AddTenant -name Liquidation -role SERVER -instanceCount 2 -offlineInstanceCount 1 -realTimeInstanceCount 1 -exec
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/pinot/lib/pinot-all-0.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.7.1-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Anusha to Everyone (12:39 PM)
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.pinot.spi.plugin.PluginClassLoader (file:/opt/pinot/lib/pinot-all-0.7.1-jar-with-dependencies.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.pinot.spi.plugin.PluginClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Executing command: AddTenant -controllerProtocol http -controllerHost 10.222.86.148 -controllerPort 9000 -name Liquidation -role SERVER -instanceCount 2 -offlineInstanceCount 1 -realTimeInstanceCount 1 -exec
{"code":500,"error":"Failed to create tenant"}
{"code":500,"error":"Failed to create tenant"}
Surendra
05/25/2021, 5:05 PMWhen emitting an event to kafka, a user need to feed partitioning key and partition function for Kafka producer API
. Can someone give insights on how it works internally? How to configure schema registry for Kafka record keys ?Machhindra
05/26/2021, 3:41 AMMetric.avsc
===========
{
"namespace": "com.blah",
"name": "Metric",
"type": "record",
"fields": [{
"name": "product",
"type": ["string", "null"]
},{
"name": "productVersion",
"type": ["string", "null"]
},{
"name": "MetricSource",
"type": ["com.blah.MetricSource", "null"]
},{
"name": "metricPath",
"type":{
"type": "array",
"items": ["string", "null"]
}
}]
}
MetricSource.avsc
===========
{
"namespace": "com.blah",
"name": "MetricSource",
"type": "record",
"fields": [{
"name": "metric",
"type": ["string", "null"]
},{
"name": "metricValue",
"type": ["string", "null"]
},{
"name": "time",
"type": "long"
},{
"name": "timeOffset",
"type": "double"
},{
"name": "category",
"order": "ignore",
"type": ["null", "string"],
"default": null
},{
"name": "subCategory",
"order": "ignore",
"type": ["null", "string"],
"default": null
}
]
}
Jonathan Meyer
05/26/2021, 8:40 AMRK
05/26/2021, 4:56 PMhttps://apache-pinot.slack.com/files/U020N9ADN5D/F0233EVB6P5/img_20210526_174024.jpgโพ
Neil Teng
05/26/2021, 7:16 PM{
"name": "date",
"dataType": "STRING",
"format": "1:SIMPLE_DATE_FORMAT:yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
"granularity": "1:MILLISECONDS"
},
Sadim Nadeem
05/27/2021, 3:04 PMJonathan Meyer
05/27/2021, 3:35 PMShailesh Jha
05/27/2021, 3:54 PMERROR [PinotFSFactory] [Start a Pinot [SERVER]] Could not instantiate file system for class org.apache.pinot.plugin.filesystem.GcsPinotFS with scheme gs
@Mohamed SultanJonathan Meyer
05/27/2021, 5:04 PMMachhindra
05/27/2021, 6:23 PMSELECT DATETIMECONVERT(metricTime, '1:MILLISECONDS:EPOCH', '1:MILLISECONDS:EPOCH', '1:MINUTES'),
AVG(metricValue) AS "AVG_1"
FROM metric_v6.metric_v6
WHERE metricTime >= 1621555200000
AND metricTime < 1622160000000
AND metric = 'CECCP%'
GROUP BY DATETIMECONVERT(metricTime, '1:MILLISECONDS:EPOCH', '1:MILLISECONDS:EPOCH', '1:MINUTES')
LIMIT 10000
Screenshot of the Superset SQL Editor -Shailesh Jha
05/28/2021, 6:01 AMSEVERE: An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection closed
CC: @Mohamed Sultan @Sadim NadeemRK
05/28/2021, 8:00 AMSadim Nadeem
05/28/2021, 1:08 PMPedro Silva
05/31/2021, 1:09 PMdateTimeFieldSpecs[{
"name": "dateOfBirth",
"dataType": "STRING",
"format": "1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-dd'T'HH:mm:ss'Z'",
"granularity": "1:DAYS"
},...,]
Should you be able to apply datetime funcitons transformations at query time?
For example, retrieving the year of the field: select year("dateOfBirth") from ....
I'm getting parsing errors:
2021/05/31 12:52:50.536 ERROR [BaseCombineOperator] [pqw-1] Caught exception while executing operator of index: 0 (query: QueryContext{_tableName='HitExecutionView_REALTIME', _selectExpressions=[year(dateOfBirth)], _aliasList=[null], _filter=null, _groupByExpressions=null, _havingFilter=null, _orderByExpressions=null, _limit=10, _offset=0, _queryOptions={responseFormat=sql, groupByMode=sql, timeoutMs=9994}, _debugOptions=null, _brokerRequest=BrokerRequest(querySource:QuerySource(tableName:HitExecutionView_REALTIME), pinotQuery:PinotQuery(dataSource:DataSource(tableName:HitExecutionView_REALTIME), selectList:[Expression(type:FUNCTION, functionCall:Function(operator:YEAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:dateOfBirth))]))], orderByList:[], limit:10, queryOptions:{responseFormat=sql, groupByMode=sql, timeoutMs=9994}))})
java.lang.NumberFormatException: For input string: "1997-02-06T00:00:00Z"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_292]
at java.lang.Long.parseLong(Long.java:589) ~[?:1.8.0_292]
at java.lang.Long.parseLong(Long.java:631) ~[?:1.8.0_292]
Charles
06/01/2021, 9:10 AMcontroller.data.dir
, If there have some methods to delete controller segments with retention times, since we donโt have enough disk space to store in controller?Pedro Silva
06/01/2021, 10:50 AM{
"columnName": "dateOfBirthMs",
"transformFunction": "fromDateTime(dateOfBirth, 'yyyy-MM-dd''T''HH:mm:ss''Z')"
}
And schema definitions:
"dimensionFieldSpecs": [
,...,
{
"name": "dateOfBirth",
"dataType": "STRING"
},...,
],
"dateTimeFieldSpecs": [
...,
{
"name": "dateOfBirthMs",
"dataType": "LONG",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}
],
I get this exception:
java.lang.IllegalStateException: Caught exception while invoking method: public static long org.apache.pinot.common.function.scalar.DateTimeFunctions.fromDateTime(java.lang.String,java.lang.String) with arguments: [null, yyyy-MM-dd'T'HH:mm:ss'Z]
I was under the impression that Pinot would not apply the transformation if the input field is null or that the transformation itself would be resilient.
Is there any way around this?RK
06/02/2021, 11:26 AMMachhindra
06/02/2021, 8:03 PMKen Krugler
06/02/2021, 9:33 PMmultiValueDelimiter
Elon
06/03/2021, 4:26 AMpqw
threads in default scheduler) threadpool on one server being blocked. Would it help to make the worker thread pool use a cached threadpool while still keeping the query runner threadpool (pqr
threads in default scheduler) fixed? Or do you recommend using one of the other query schedulers? Here is the thread dump:RK
06/03/2021, 4:36 PMRK
06/04/2021, 8:10 AMJonathan Meyer
06/04/2021, 12:19 PM"segment.realtime.status": "IN_PROGRESS"
/ "CONSUMING"
Also, I'm not seeing any related logs
On the previous topic, consumption was OK โ๏ธJonathan Meyer
06/04/2021, 4:38 PMRK
06/07/2021, 5:14 AMKaushik Ranganath
06/07/2021, 5:19 AMJosh Highley
06/07/2021, 3:31 PM2021/06/07 15:24:27.918 INFO [Fetcher] [agent_daily__2__2__20210605T0819Z] [Consumer clientId=consumer-71, groupId=] Fetch offset 22 is out of range for partition agent_daily-2, resetting offset
2021/06/07 15:24:27.919 INFO [Fetcher] [agent_daily__2__2__20210605T0819Z] [Consumer clientId=consumer-71, groupId=] Resetting offset for partition agent_daily-2 to offset 5.
2021/06/07 15:24:27.938 INFO [Fetcher] [agent_daily__1__2__20210605T0819Z] [Consumer clientId=consumer-73, groupId=] Fetch offset 20 is out of range for partition agent_daily-1, resetting offset
2021/06/07 15:24:27.939 INFO [Fetcher] [agent_daily__1__2__20210605T0819Z] [Consumer clientId=consumer-73, groupId=] Resetting offset for partition agent_daily-1 to offset 0.
2021/06/07 15:24:27.942 INFO [Fetcher] [agent_daily__0__2__20210605T0819Z] [Consumer clientId=consumer-72, groupId=] Fetch offset 24 is out of range for partition agent_daily-0, resetting offset
2021/06/07 15:24:27.943 INFO [Fetcher] [agent_daily__0__2__20210605T0819Z] [Consumer clientId=consumer-72, groupId=] Resetting offset for partition agent_daily-0 to offset 1.
2021/06/07 15:24:33.018 INFO [Fetcher] [agent_daily__2__2__20210605T0819Z] [Consumer clientId=consumer-71, groupId=] Fetch offset 22 is out of range for partition agent_daily-2, resetting offset
2021/06/07 15:24:33.018 INFO [Fetcher] [agent_daily__2__2__20210605T0819Z] [Consumer clientId=consumer-71, groupId=] Resetting offset for partition agent_daily-2 to offset 5.
2021/06/07 15:24:33.038 INFO [Fetcher] [agent_daily__1__2__20210605T0819Z] [Consumer clientId=consumer-73, groupId=] Fetch offset 20 is out of range for partition agent_daily-1, resetting offset
2021/06/07 15:24:33.039 INFO [Fetcher] [agent_daily__1__2__20210605T0819Z] [Consumer clientId=consumer-73, groupId=] Resetting offset for partition agent_daily-1 to offset 0.
2021/06/07 15:24:33.042 INFO [Fetcher] [agent_daily__0__2__20210605T0819Z] [Consumer clientId=consumer-72, groupId=] Fetch offset 24 is out of range for partition agent_daily-0, resetting offset
2021/06/07 15:24:33.043 INFO [Fetcher] [agent_daily__0__2__20210605T0819Z] [Consumer clientId=consumer-72, groupId=] Resetting offset for partition agent_daily-0 to offset 1.
The 'reset' offsets of 5, 0, and 1 are correct: I created a new 'test' table for the same topic and it used those offsets with no issue. I've tried disabling/enabling the table but it resumes those error messages. Is there some other way to reset the table consumers?Sadim Nadeem
06/07/2021, 7:53 PMRK
06/08/2021, 6:39 AM