https://pinot.apache.org/ logo
Join Slack
Channels
Powered by
# pinot-dev
  • x
    hmm
  • x
    have you tried to build the pinot using maven
  • a
    Yes, I am able to build sucessfully through 
    mvn clean install package -DskipTests -Pbin-dist -DdownloadSources -DdownloadJavadocs
  • x
    ok
  • k
    @User ^^
  • m
    Hi Team, a heads up in case your dashboards don't see Pinot controller metrics: there was a recent change in naming conventions for Pinot Controller metrics (
    pinot_controller
    -> 
    pinot_controller_
    ).
    thankyou 1
  • k
    Hi all - I’m back to working on https://github.com/apache/incubator-pinot/issues/6293, where I was trying to put all of the code used for Pinot integration tests into a tests-qualified jar. But I’m wondering if it would work better to have a separate Pinot sub-project/jar that can be used to spin up a stand-alone “mini” Pinot cluster, and have that jar be used by pinot-integration-tests (and 3rd parties wanting to do the same). This would provide cleaner separation for what’s used internally by Pinot, versus what projects using Pinot would need to use. Thoughts?
    ➕ 1
  • k
    But I’m wondering if it would work better to have a separate Pinot sub-project/jar that can be used to spin up a stand-alone “mini” Pinot cluster, and have that jar be used by pinot-integration-tests (and 3rd parties wanting to do the same)
    + 100 for this
  • k
    this is the right thing to do
  • k
    infact our quickstart is very similar to this
  • k
    it just brings up everything in one process
  • n
    Quickstart isn’t working for me in Intellij after pulling recent changes. Anyone else seeing this?
    Copy code
    2021/03/11 20:28:36.874 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 10.357 since launch
java.lang.IllegalStateException: Failed to initialize PinotMetricsFactory. Please check if any pinot-metrics related jar is actually added to the classpath.
	at com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[guava-20.0.jar:?]
	at org.apache.pinot.common.metrics.PinotMetricUtils.initializePinotMetricsFactory(PinotMetricUtils.java:85) ~[classes/:?]
	at org.apache.pinot.common.metrics.PinotMetricUtils.init(PinotMetricUtils.java:57) ~[classes/:?]
	at org.apache.pinot.controller.ControllerStarter.initControllerMetrics(ControllerStarter.java:483) ~[classes/:?]
	at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:279) ~[classes/:?]
	at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:116) ~[classes/:?]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:91) ~[classes/:?]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:234) ~[classes/:?]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [classes/:?]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:233) [classes/:?]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:183) [classes/:?]
	at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [classes/:?]
	at org.apache.pinot.tools.admin.command.QuickstartRunner.startControllers(QuickstartRunner.java:124) [classes/:?]
	at org.apache.pinot.tools.admin.command.QuickstartRunner.startAll(QuickstartRunner.java:170) [classes/:?]
	at org.apache.pinot.tools.Quickstart.execute(Quickstart.java:169) [classes/:?]
	at org.apache.pinot.tools.admin.command.QuickStartCommand.execute(QuickStartCommand.java:78) [classes/:?]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164) [classes/:?]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184) [classes/:?]
  • n
    @User could this be related to the metrics registry changes ?
  • j
    How do you run the 
    StartServiceManagerCommand
    ?
  • a
    Hi we will be installing Pinot cluster in AWS on top of EKS. We know that in AWS EKS has Multi (Three) Availability Zone (AZ) based HA in a specific Region. So I would like to understand that whether the EKS based Pinot cluster will be by default Fault Tolerant & HA within the region in case of any AZ failure or not. I know that Pinot Server has Segment Replica and replica-group which provide HA within the cluster in case of server failure. But what will happen if the controller has issue in the cluster (on EKS) or multiple servers have been corrupted or the cluster (on EKS) as a whole goes down? Considering the fact that the server will have EBS as data serving file system (& EBS multi AZ replication/sync will be ON), will EKS by default bring up alternative node like Controller or Server (or even Broker)? Net-net can we expect 100% service availability in Pinot on EKS in any Region? Or do we need to setup another Pinot Cluster on EKS on another AZ i.e. minimum Two Pinot Cluster (On EKS) in Two AZ within a Region? Please suggest (edited)
  • x
    wanna start a session on this topic: https://github.com/apache/incubator-pinot/issues/6689
  • k
    Hi When a server is in dead state(update of deployment in kubernetes) pinot doesn't rebalance the segments among existing/new severs. Even manual rebalance is not helping(result: already balanced). Tried deleting the instance. It fails with error :"server is in ideal state of xyz table" . How do I resolve this?
  • n
    @User ^^
  • k
    Let’s repost this in troubleshooting..
  • g
    Hello! 👋 I am trying to write integration tests for my application that hits Pinot through the Java JDBC. When we do the same for elasticsearch, we use this elasticsearch test container to spin up a docker image to run the tests on. There is not a test container for Pinot. Does anyone have experience with this or has found an easy way to either make your own test container or spin up the docker image yourself? Thanks!
  • k
    we recently put together a test container for pinot as well @User ^^
  • k
    sharing it here as well, if anyone else needs it
  • k
    Copy code
    import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.containers.wait.strategy.WaitAllStrategy;
import org.testcontainers.utility.DockerImageName;

public class PinotContainer extends GenericContainer<PinotContainer> {

    public static final int BROKER_HTTP_PORT = 9000;
    public static final String DEFAULT_ENDPOINT = "/";
    private static final DockerImageName DEFAULT_IMAGE_NAME = DockerImageName.parse("apachepinot/pinot");
    private static final String DEFAULT_TAG = "0.6.0";

    public PinotContainer() {
        this(DEFAULT_IMAGE_NAME.withTag(DEFAULT_TAG));
    }

    public PinotContainer(String pinotVersion) {
        this(DEFAULT_IMAGE_NAME.withTag(pinotVersion));
    }

    public PinotContainer(final DockerImageName dockerImageName) {
        super(dockerImageName);
        dockerImageName.assertCompatibleWith(DockerImageName.parse("apachepinot/pinot"));
        withExposedPorts(BROKER_HTTP_PORT);
        withCommand("QuickStart", "-type", "batch");
        waitingFor(Wait.forHttp(DEFAULT_ENDPOINT).forStatusCode(200).forPort(BROKER_HTTP_PORT));
    }

    @Override
    protected void configure() {
        super.configure();
        withCreateContainerCmdModifier(createContainerCmd -> createContainerCmd.withName("pinot-quickstart"));
        withCommand("QuickStart", "-type", "batch");
        waitingFor(
            new WaitAllStrategy()
                    .withStrategy(waitStrategy)
                    .withStrategy(Wait.forListeningPort()));
    }

    public String getPinotBrokerUrl() {
        return String.format("<http://%s:%s>", getHost(), getBrokerPort());
    }

    public Integer getBrokerPort() {
        return getMappedPort(BROKER_HTTP_PORT);
    }
}
  • k
    we will publish it when we get a chance but for now you can just use this class
    🙌 2
    thankyou 1
  • n
    we are kinda blocked on it
  • s
    Hi, it looks that all the PRs are failing at the unit tests
    Copy code
    [INFO] ERROR in /home/runner/work/incubator-pinot/incubator-pinot/pinot-controller/src/main/resources/app/pages/Query.tsx
[INFO] ./app/pages/Query.tsx
[INFO] [tsl] ERROR in /home/runner/work/incubator-pinot/incubator-pinot/pinot-controller/src/main/resources/app/pages/Query.tsx(238,20)
[INFO]       TS2345: Argument of type '{ data: any[]; fileName: string; exportType: any; }' is not assignable to parameter of type 'IOption<void>'.
[INFO]   Property 'fields' is missing in type '{ data: any[]; fileName: string; exportType: any; }' but required in type 'IOption<void>'.
[INFO] Child HtmlWebpackCompiler:
[INFO]      1 asset
[INFO]     Entrypoint HtmlWebpackPlugin_0 = __child-HtmlWebpackPlugin_0
[INFO]     [0] ./node_modules/html-webpack-plugin/lib/loader.js!./app/index.html 1.42 KiB {0} [built]
Error:  npm ERR! code ELIFECYCLE
Error:  npm ERR! errno 2
Error:  npm ERR! pinot-controller-ui@1.0.0 build: `webpack --mode production`
Error:  npm ERR! Exit status 2
Error:  npm ERR! 
Error:  npm ERR! Failed at the pinot-controller-ui@1.0.0 build script.
Error:  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Error:  
Error:  npm ERR! A complete log of this run can be found in:
Error:  npm ERR!     /home/runner/.npm/_logs/2021-04-14T17_33_03_262Z-debug.log
  • m
    image.png
  • x
    yes
  • x
    Unit test fails due to recent github action change, there are some network interface changes
  • x
    so dns resolution has some issue
1...222324...30Latest