https://pinot.apache.org/ logo
Join Slack
Powered by
Hi all, I have created a realtime Pinot table and ...
# troubleshooting
k
Hi all, I have created a realtime Pinot table and after the initial snapshot it's not streaming any any new records. I queried the source table and it has new data in it. I'm using debezium connector and it's up and running. Any suggestions/ideas on troubleshooting?
table config
Copy code
{
  "tableName": "profiles",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "schemaName": "profiles",
    "replication": 1,
    "timeColumnName": "created_at",
    "timeType": "MILLISECONDS",
    "replicasPerPartition": "1",
    "allowNullTimeValue": false,
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "1",
    "segmentPushType": "APPEND",
    "completionConfig": {
      "completionMode": "DOWNLOAD"
    }
  },
  "ingestionConfig": {
    "transformConfigs": [
      {
        "columnName": "dob",
        "transformFunction": "Groovy({date_of_birth != 'null' ? new Date(date_of_birth).format('yyyy-MM-dd'):'1900-01-01'},date_of_birth)"
      }
    ]
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "nullHandlingEnabled": true,
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "LowLevel",
      "stream.kafka.topic.name": "profiles",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
      "stream.kafka.isolation.level": "read_committed",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "kafka:9092",
      "stream.kafka.zk.broker.url": "zookeeper:2181",
      "realtime.segment.flush.threshold.size": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.desired.size": "200M"
    }
  },
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  },
  "upsertConfig": {
    "mode": "FULL"
  },
  "metadata": {}
}
n
likely some issue with Segment build. Can you check server logs and controller logs? might help to look for the exact segment name that was stuck
m
You can also check the 
/debug
endpoint in Swagger
k
The controller log has below Warnings:
Copy code
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Copy code
WARNING: The following warnings have been detected: WARNING: HK2 service reification failed for [javax.servlet.ServletConfig] with an exception:
MultiException stack 1 of 2
java.lang.NoSuchMethodException: Could not find a suitable constructor in javax.servlet.ServletConfig class.
	at org.glassfish.jersey.inject.hk2.JerseyClassAnalyzer.getConstructor(JerseyClassAnalyzer.java:168)
	at org.jvnet.hk2.internal.Utilities.getConstructor(Utilities.java:156)
	at org.jvnet.hk2.internal.ClazzCreator.initialize(ClazzCreator.java:105)
	at org.jvnet.hk2.internal.ClazzCreator.initialize(ClazzCreator.java:156)
	at org.jvnet.hk2.internal.SystemDescriptor.internalReify(SystemDescriptor.java:716)
	at org.jvnet.hk2.internal.SystemDescriptor.reify(SystemDescriptor.java:670)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:441)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2287)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.igdCacheCompute(ServiceLocatorImpl.java:1163)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.access$400(ServiceLocatorImpl.java:105)
	at org.jvnet.hk2.internal.ServiceLocatorImpl$8.compute(ServiceLocatorImpl.java:1157)
	at org.jvnet.hk2.internal.ServiceLocatorImpl$8.compute(ServiceLocatorImpl.java:1154)
	at org.glassfish.hk2.utilities.cache.internal.WeakCARCacheImpl.compute(WeakCARCacheImpl.java:105)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(ServiceLocatorImpl.java:1237)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetInjecteeDescriptor(ServiceLocatorImpl.java:558)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getInjecteeDescriptor(ServiceLocatorImpl.java:567)
	at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.lambda$new$0(ContextInjectionResolverImpl.java:81)
	at org.glassfish.jersey.internal.util.collection.Cache$OriginThreadAwareFuture.lambda$new$0(Cache.java:169)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.glassfish.jersey.internal.util.collection.Cache$OriginThreadAwareFuture.run(Cache.java:225)
	at org.glassfish.jersey.internal.util.collection.Cache.apply(Cache.java:77)
	at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.resolve(ContextInjectionResolverImpl.java:95)
	at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.resolve(ContextInjectionResolverImpl.java:121)
	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueParamProvider.lambda$getValueProvider$0(DelegatedInjectionValueParamProvider.java:67)
	at org.glassfish.jersey.server.spi.internal.ParamValueFactoryWithSource.apply(ParamValueFactoryWithSource.java:50)
	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:64)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:109)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:353)
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
	at java.base/java.lang.Thread.run(Thread.java:829)
MultiException stack 2 of 2 java.lang.IllegalArgumentException: Errors were discovered while reifying SystemDescriptor(
	implementation=javax.servlet.ServletConfig
	contracts={javax.servlet.ServletConfig}
	scope=org.glassfish.jersey.process.internal.RequestScoped
	qualifiers={}
	descriptorType=CLASS
	descriptorVisibility=NORMAL
	metadata=
	rank=0
	loader=null
	proxiable=null
	proxyForSameScope=null
	analysisName=null
	id=184
	locatorId=0
	identityHashCode=38173719
	reified=false)
	at org.jvnet.hk2.internal.SystemDescriptor.reify(SystemDescriptor.java:681)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:441)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2287)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.igdCacheCompute(ServiceLocatorImpl.java:1163)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.access$400(ServiceLocatorImpl.java:105)
	at org.jvnet.hk2.internal.ServiceLocatorImpl$8.compute(ServiceLocatorImpl.java:1157)
	at org.jvnet.hk2.internal.ServiceLocatorImpl$8.compute(ServiceLocatorImpl.java:1154)
	at org.glassfish.hk2.utilities.cache.internal.WeakCARCacheImpl.compute(WeakCARCacheImpl.java:105)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(ServiceLocatorImpl.java:1237)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetInjecteeDescriptor(ServiceLocatorImpl.java:558)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getInjecteeDescriptor(ServiceLocatorImpl.java:567)
	at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.lambda$new$0(ContextInjectionResolverImpl.java:81)
	at org.glassfish.jersey.internal.util.collection.Cache$OriginThreadAwareFuture.lambda$new$0(Cache.java:169)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.glassfish.jersey.internal.util.collection.Cache$OriginThreadAwareFuture.run(Cache.java:225)
	at org.glassfish.jersey.internal.util.collection.Cache.apply(Cache.java:77)
	at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.resolve(ContextInjectionResolverImpl.java:95)
	at org.glassfish.jersey.inject.hk2.ContextInjectionResolverImpl.resolve(ContextInjectionResolverImpl.java:121)
	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueParamProvider.lambda$getValueProvider$0(DelegatedInjectionValueParamProvider.java:67)
	at org.glassfish.jersey.server.spi.internal.ParamValueFactoryWithSource.apply(ParamValueFactoryWithSource.java:50)
	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:64)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:109)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
Nothing related to segment build though
n
what about server?
k
Just these warnings
Copy code
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-0.8.0-jar-with-dependencies.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
I've posted on debezium google group as well. Checking the kafka topic I'm not seeing new messages but the connector status is up and running.
Open in Slack
PreviousNext
Powered by