I dont think we can edit it

checking with ASF

Hi All, I need some details around pinot S3 integration. Please let me know if this is the right forum to ask my question

feel free to ask in troubleshooting

I created a PR (https://github.com/apache/pinot/pull/7400), but two of the tests fail. When I look at the logs, the failures seem unrelated to my change (one seems to be a ZkClient connection issue, the other is a timeout during an operation). Is this common for PR builds? What’s the right way to confirm my change didn’t cause the problem(s)?

I’ve been working on a logging issue when building segments using Hadoop - the expected logging output from the map task isn’t showing up. While poking around the jars on the classpath, I noticed that many of the plugins include logging implementations, which can mess with the Log4J-based Hadoop logging configuration. It seems odd that plugins would including logging classes in their builds, as the pinot-all jar has Log4J logging classes inside it. Thoughts on this?

yes, that has been a pain

please let us know if you can come up with a fix

Well, normally you’d have all code just use the

slf4j-api

, and that would be the only compile-time dependency. Then you’d have test dependencies on

log4j-slf4j-impl

everywhere, and only have a compile time dependency on

log4j-slf4j-impl

in stand-alone components like the server, broker, controller, etc.

we use slf4j everywhere

Please review: https://github.com/apache/pinot/issues/7519

I’m seeing the kinesis integration test failed, seems the localstack doesn’t start

Error:  org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.setUp  Time elapsed: 175.147 s  <<< FAILURE!
cloud.localstack.docker.exception.LocalstackDockerException: Could not start the localstack docker container.
	at cloud.localstack.Localstack.startup(Localstack.java:97)
	at org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.startKinesis(RealtimeKinesisIntegrationTest.java:224)
	at org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.setUp(RealtimeKinesisIntegrationTest.java:135)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:146)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:166)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)
	at org.testng.TestRunner.privateRun(TestRunner.java:744)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
	at org.testng.TestNG.runSuites(TestNG.java:1144)
	at org.testng.TestNG.run(TestNG.java:1115)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:136)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:145)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: java.lang.IllegalStateException: Could not find token: Ready\. in Docker logs: 2021-10-19T08:14:44:DEBUG:plugin.manager: instantiating plugin PluginSpec(localstack.aws.provider.sts:default = <function sts at 0x7fa537678e60>)

Hi, yes I also noticed that today. Looking into it.

It was working till yesterday so something changed in that time.

is there any change from github ci ? AlsoI cannot run it from my local laptop as well

Testing in local. Will update you in few minutes.

working fine in local. updating docker to latest version to see if that causes any issue.

I also updated the lib to 0.2.15, seems solved the issue

Hi ,can we remove old data(more than one week) from pinot table.if Yes how?

@User Please use the general or troubleshooting channel for such questions

Ok @User

Are there any in-progress discussions around adding big number data types? E.g. big decimal and big integer

we added biddecimal earlier this year

Is that available in 0.8.0? I can't find it in the documentation

I mean having a column of type Decimal(30, 4) for example. I've been skimming the source code and FieldType only define the types as per the docs. I've seen the PR that introduces

bytesToBigDecimal() but nothing to support BigDecimal as a data type

@User ^^

@User

BigDecimal

is not supported as a standard data type yet. Currently in order to use

BigDecimal

you need to store them as

BYTES

. You may use

BigDecimalUtils.serialize()

to get the serialized bytes

Yup, so my question as per the previous comment is whether there are any discussion about adding these data types at the moment or that isn't on the roadmap yet?

I think we have the primitives needed to support it as a standard datatype, so dont see a reason not to do it.

please file a github issue