https://pinot.apache.org/ logo
Join Slack
Powered by
# pinot-dev
  • k
    Hi devs - I saw https://github.com/apache/pinot/issues/7716, which caught my eye due to activity in the Flink project to remove Scala dependencies. I was curious why the 
    pinot-kafka
    (0.9 and 2.0) plugins needed Scala, since my use of Kafka clients never triggered Scala dependencies. From the issue above, it’s all about being able to start up a Kafka server (
    KafkaDataServerStartable
    ). And that only seems to be needed by test code, or the 
    tools
    subproject when starting a server for quickstart. In my experience bundling test/demo code into core jars often leads to this type of dependency problem, so it seems better to have this as a test dependency, with a separate demo jar that provides Kafka server startup for quickstart.
    👍 2
  • a
    Any challenges seen in building Pinot master on M1? I run into dependency issues where GRPC and Node dependencies are not found for the platform
  • o
    @User has left the channel
  • x
    seems the kinesis test doesn’t work again @User
    👍 1
  • x
    Tried upgrade localstack version, but not working this time. Maybe let’s not enable it by default?
  • k
    yeah we can disable the integration test for now can you send me the error log
  • x
    It's same as last time the startKinesis method does move on
  • x
    Copy code
    cloud.localstack.docker.exception.LocalstackDockerException: Could not start the localstack docker container.

	at cloud.localstack.Localstack.startup(Localstack.java:104)
	at org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.startKinesis(RealtimeKinesisIntegrationTest.java:224)
	at org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.setUp(RealtimeKinesisIntegrationTest.java:135)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:146)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:166)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)
	at org.testng.TestRunner.privateRun(TestRunner.java:744)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
	at org.testng.TestNG.runSuites(TestNG.java:1144)
	at org.testng.TestNG.run(TestNG.java:1115)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)
  • x
    I’ve merged the PR: https://github.com/apache/pinot/pull/7735 to disable KinesisTest for now. Please rebase your PRs if you are encountering the CI failure.
    thankyou 1
  • a
  • a
  • r
    @User can you merge this please?https://github.com/apache/pinot/pull/7889
  • s
    @User has left the channel
  • k
    Hi There's new reported RCE vulnerability in log4j which affects even 2.17.0. We need to upgrade to 2.17.1 to mitigate this
  • k
    Will raise PR in a while
  • k
  • m
    Thanks @User
  • a
  • k
    We do daily builds of offline segments using Hadoop, and then store the results in HDFS in the directory that is configured as our Pinot cluster’s deep store. Our build generates 35 new (or more typically updated) per-month segments each day, which we then deploy to our Pinot cluster via a metadata push. What this means is that we’ve got a deep store directory in HDFS with ≈ 1200 segments (representing 3 years of data) for a table. When we do the metadata push every segment is downloaded, metadata is extracted, and that metadata tarball is sent to the controller. This takes about 3 hours currently. But we only want to send the 35 new segments. It seems like a simple solution would be to support a new, optional 
    pushFileNamePattern
    parameter in the job conf, which could be used to filter down to only the segments we care about. The format could be the same as the existing 
    includeFileNamePattern
    (glob pattern). Thoughts on this approach? Worth a PR? Thanks!
    👍 1
  • m
    PR merge should have gone to docs automatically. @User
  • x
    We should keep modifying the docs
  • x
    And take snapshot on docs when we cut the release
  • r
    has anyone else's PRs started failing in quickstart?
  • s
    yes, mine is failing on qs. It is urgent so I think I will just merge it
  • h
    Any reason why it would start failing now? @User it looks like your patch was the first one to fail
  • r
    I'm not sure that's true - they all started failing independently at the same time but I had clean builds in between the last merge and the failures emerging
  • r
    I reverted the last thing I successfully built (https://github.com/apache/pinot/actions/runs/1811017850) and merged before this started happening, it fails https://github.com/apache/pinot/runs/5121243866?check_suite_focus=true
  • r
    I'm going to rerun some previously successful builds on merged branches because I think this is environmental, if it is they should fail
1...2627282930Latest