https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
b
How do you know it's getting the data? Could be many things, maybe coordinator http timeout settings, idk. Any details in overlord or coordinator logs if you search for the task name?
s
you are right, I don't know. So I created a supervisor spec using the wizard initially, and from that it connected to the kafka stream, got some data and I went followed the steps to create a supervisor specs... so yeah, I can't assume its getting the data, I think I can say the connectivity is there at least. So the supervisor is running, kicking off these peon tasks periodically.. and they have their own logs which seem to have no errors in. These are the indexing logs I am talking about. They do however have this warning...
Copy code
2022-12-22T14:36:42,721 WARN [main] org.apache.druid.java.util.metrics.AllocationMetricCollectors - Cannot initialize org.apache.druid.java.util.metrics.AllocationMetricCollector
java.lang.reflect.InaccessibleObjectException: Unable to make public long[] com.sun.management.internal.HotSpotThreadImpl.getThreadAllocatedBytes(long[]) accessible: module jdk.management does not "exports com.sun.management.internal" to unnamed module @27ae2fd0
        at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340) ~[?:?]
        at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280) ~[?:?]
        at java.lang.reflect.Method.checkCanSetAccessible(Method.java:198) ~[?:?]
        at java.lang.reflect.Method.setAccessible(Method.java:192) ~[?:?]
        at org.apache.druid.java.util.metrics.AllocationMetricCollectors.<clinit>(AllocationMetricCollectors.java:41) ~[druid-core-24.0.1.jar:24.0.1]
        at org.apache.druid.java.util.metrics.JvmMonitor.<init>(JvmMonitor.java:67) ~[druid-core-24.0.1.jar:24.0.1]
        at org.apache.druid.java.util.metrics.JvmMonitor.<init>(JvmMonitor.java:59) ~[druid-core-24.0.1.jar:24.0.1]
        at org.apache.druid.server.metrics.MetricsModule.getJvmMonitor(MetricsModule.java:148) ~[druid-server-24.0.1.jar:24.0.1]
        at org.apache.druid.server.metrics.MetricsModule$$FastClassByGuice$$99ddce1b.invoke(<generated>) ~[druid-server-24.0.1.jar:24.0.1]
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) ~[guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) ~[guice-4.1.0.jar:?]
there are however these errors I've not managed to get to the bottom of...
Copy code
2022-12-22T14:38:52,228 ERROR [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - Invalid config event received: {server.2=w3-druid-zk-03-preprod:2888:3888:participant, server.1=w3-druid-zk-02-preprod:2888:3888:participant, server.0=0.0.0.0:2888:3888:participant, version=0}
2022-12-22T14:38:52,229 INFO [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - New config event received: {server.2=w3-druid-zk-03-preprod:2888:3888:participant, server.1=w3-druid-zk-02-preprod:2888:3888:participant, server.0=0.0.0.0:2888:3888:participant, version=0}
2022-12-22T14:38:52,229 ERROR [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - Invalid config event received: {server.2=w3-druid-zk-03-preprod:2888:3888:participant, server.1=w3-druid-zk-02-preprod:2888:3888:participant, server.0=0.0.0.0:2888:3888:participant, version=0}
I think the zookeeper config is supposed to have 0.0.0.0 for the current node... I don't think the cluster forms if its not set to that.
b
Interesting. If you follow the wizard, does the working config match this:
Copy code
{server.2=w3-druid-zk-03-preprod:2888:3888:participant, server.1=w3-druid-zk-02-preprod:2888:3888:participant, server.0=0.0.0.0:2888:3888:participant, version=0}
v
Is this cluster running on a single machine?
s
@Vadim its on 5 machines, each of them redhat openshift vms. The druid doctor returns successfully. @Ben Krug not sure what exactly you mean. I am thinking the zookeeper connection is correct mainly based of the fact that the cluster can form at all.
I see this in the task logs if that helps
Copy code
{
  "id": "index_kafka_sn-content-events_f6e229d741032ab_cifepkmi",
  "groupId": "index_kafka_sn-content-events",
  "type": "index_kafka",
  "createdTime": "2022-12-23T23:04:52.620Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "RUNNING",
  "duration": -1,
  "location": {
    "host": "w3-druid-data-01-preprod",
    "port": 8100,
    "tlsPort": -1
  },
  "dataSource": "sn-content-events",
  "errorMsg": "Task [index_kafka_sn-content-events_f6e229d741032ab_cifepkmi] failed to return start time, killing t..."
}
g
Getting to this after 2 months (I saw a link to the thread recently), so I'm not sure if this is still relevant đŸ™‚
But… I wonder what version you're using? In 25.0 the code that checks start times (and all other RPC stuff) was rewritten to be more robust (https://github.com/apache/druid/pull/13354)
So, if you're using an earlier version, I would suggest upgrading If it's still happening on 25.0+, then we should look into it more deeply
Open in Slack
PreviousNext
Powered by