This message was deleted.
# troubleshootings
Slackbot
09/27/2022, 9:09 PMThis message was deleted.
j
Julian Reyes
09/27/2022, 9:12 PMin coordinator logs:
Copy code
2022-09-27T21:11:04,730 WARN [KinesisSupervisor-test-Reporting-0] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Exception while getting current/latest sequences │
│ org.apache.druid.indexing.seekablestream.common.StreamException: com.amazonaws.AbortedException: │
│ at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.wrapExceptions(KinesisRecordSupplier.java:120) ~[?:?] │
│ at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.getPartitionTimeLag(KinesisRecordSupplier.java:991) ~[?:?] │
│ at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.getPartitionsTimeLag(KinesisRecordSupplier.java:825) ~[?:?] │
│ at org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisor.updatePartitionLagFromStream(KinesisSupervisor.java:340) ~[?:?] │
│ at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.updateCurrentAndLatestOffsets(SeekableStreamSupervisor.java:3590) ~[druid-indexing-service-24.0.0.jar:24.0.0] │
│ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_275] │
│ at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_275] │
│ at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_275] │
│ at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_275] │
│ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_275] │
│ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_275] │
│ at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275] │
│ Caused by: com.amazonaws.AbortedException: │
│ at com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:61) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.internal.SdkFilterInputStream.markSupported(SdkFilterInputStream.java:125) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.RepeatableInputStreamRequestEntity.isRepeatable(RepeatableInputStreamRequestEntity.java:139) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.shouldRetry(AmazonHttpClient.java:1640) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1223) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[aws-java-sdk-core-1.12.264.jar:?] │
│ at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2980) ~[?:?] │
│ at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2947) ~[?:?] │
│ at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2936) ~[?:?] │
│ at com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetShardIterator(AmazonKinesisClient.java:1436) ~[?:?] │
│ at com.amazonaws.services.kinesis.AmazonKinesisClient.getShardIterator(AmazonKinesisClient.java:1405) ~[?:?] │
│ at com.amazonaws.services.kinesis.AmazonKinesisClient.getShardIterator(AmazonKinesisClient.java:1453) ~[?:?] │
│ at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.getRecordsForLag(KinesisRecordSupplier.java:1024) ~[?:?] │
│ at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.lambda$getPartitionTimeLag$18(KinesisRecordSupplier.java:1008) ~[?:?] │
│ at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.wrapExceptions(KinesisRecordSupplier.java:117) ~[?:?] │
│ ... 11 moreg
Gian Merlino
09/28/2022, 1:04 AMis it possible your kinesis stream isn't receiving any data? sometimes, via kinesis api, that can cause timeouts while reading from it
j
Julian Reyes
09/28/2022, 5:49 AMThat's unlikely, same kinesis stream is being read by another druid cluster just fine (0.23.0) and the stream receives over 6B records per day
Julian Reyes
09/28/2022, 10:53 AMAnother messages I am seeing in the current task running are:
Copy code
Kinesis records are being processed slower than they are fetched. OrderedPartitionableRecord buffer full, storing iterator and retrying in [5000ms]Julian Reyes
09/28/2022, 11:00 AMnot sure if that might be related to some of the below tuning conf for kinesis
Copy code
"httpTimeout": "PT10S",
"shutdownTimeout": "PT80S",
"recordBufferSize": 10000,
"recordBufferOfferTimeout": 5000,
"recordBufferFullWait": 5000,Julian Reyes
09/28/2022, 11:43 AMI think the issue is somehow to the fact that our k8s cluster lost connectivity somehow, well, not all the pods, but for example one coordinator did not have internet access whereas other had, similarly with some middleManagers .. All broker/router/historical did have connectivity tho, so it’s weird