Hi, I have a problem with a task that is 100% busy...
# troubleshooting
m
Hi, I have a problem with a task that is 100% busy and the watermarks are stuck few hours ago. I looked at the backpressure and I see single subtask which is 100% This subtask is also much further behind on watermarks. Any suggestions how to debug the problem?
a
Hi, Have you identified what this subtask is doing? For this you can check flamegraph/thread dumps and see if there is a bottleneck in your operator. Do you have any data skew? Is this subtask processing more records than others?
m
It's doing events session window aggregation based on user id, as I understand there isn't a reason to have skewness in the operator when the user id is kind of random.
I also have join with a lookup table (postgres), which might be a bottleneck, but the task isn't just slow, it getting completely stuck
a
One thing that I’ve seen previously with postgres is that it can become stuck on socketRead if no socker timeout is configured. But it is hard to understand what is actually happening without data - thread dump or flamegraph should help in understanding that, both can be gathered from within Flink dashboard.
m
This is the relevant task dump: looks like it really is stuck on the socketread
Copy code
"klaviyo: Aggregate session stats -> klaviyo_sessions_for_join[6] -> Calc[7] -> LookupJoin[8] -> Calc[9] -> ConstraintEnforcer[10] -> TableToDataSteam -> session_stats_klavyio[28] -> Calc[29] -> (ConstraintEnforcer[30] -> Sink: session_stats_historical[30], ConstraintEnforcer[31] -> Sink: session_stats_live[31]) (1/6)#0" Id=98 RUNNABLE (in native)
	at java.base@11.0.18/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base@11.0.18/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
	at java.base@11.0.18/java.net.SocketInputStream.read(SocketInputStream.java:168)
	at java.base@11.0.18/java.net.SocketInputStream.read(SocketInputStream.java:140)
	at java.base@11.0.18/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
	at java.base@11.0.18/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
	at java.base@11.0.18/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
	at java.base@11.0.18/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1455)
	at java.base@11.0.18/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
	at org.postgresql.core.PGStream.receiveChar(PGStream.java:465)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2154)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:367)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:134)
	at org.apache.flink.connector.jdbc.statement.FieldNamedPreparedStatementImpl.executeQuery(FieldNamedPreparedStatementImpl.java:55)
	at org.apache.flink.connector.jdbc.table.JdbcRowDataLookupFunction.lookup(JdbcRowDataLookupFunction.java:151)
	at app//org.apache.flink.table.runtime.functions.table.lookup.CachingLookupFunction.lookupByDelegate(CachingLookupFunction.java:160)
	at app//org.apache.flink.table.runtime.functions.table.lookup.CachingLookupFunction.lookup(CachingLookupFunction.java:126)
	at app//org.apache.flink.table.functions.LookupFunction.eval(LookupFunction.java:52)
	at LookupFunction$123.flatMap(Unknown Source)
	at app//org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.doFetch(LookupJoinRunner.java:92)
	at app//org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:79)
	at app//org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:34)
	at app//org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
	at app//org.apache.flink.streaming.runtime.io.RecordProcessorUtils$$Lambda$1075/0x0000000800c72c40.accept(Unknown Source)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
	at StreamExecCalc$118.processElement_split29(Unknown Source)
	at StreamExecCalc$118.processElement(Unknown Source)
	at app//org.apache.flink.streaming.runtime.io.RecordProcessorUtils$$Lambda$1075/0x0000000800c72c40.accept(Unknown Source)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
	at app//org.apache.flink.table.runtime.operators.source.InputConversionOperator.processElement(InputConversionOperator.java:144)
	at app//org.apache.flink.streaming.runtime.io.RecordProcessorUtils$$Lambda$1075/0x0000000800c72c40.accept(Unknown Source)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
	at app//org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
	at app//org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
	at app//org.apache.flink.streaming.api.functions.windowing.PassThroughWindowFunction.apply(PassThroughWindowFunction.java:35)
	at app//org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process(InternalSingleValueWindowFunction.java:48)
	at app//org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents(WindowOperator.java:554)
	at app//org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:378)
	...

	Number of locked synchronizers = 3
	- java.util.concurrent.locks.ReentrantLock$NonfairSync@5bd546b3
	- java.util.concurrent.locks.ReentrantLock$NonfairSync@1bee9f9e
	- java.util.concurrent.locks.ReentrantLock$NonfairSync@7dc68444
d
DId you find a solution to your stuck query? Mines not stuck but it is very slow, i'm having a hard time tracking down why and fixing it.
m
If it's just slow I'm not sure it's related to the problem I experienced here. I solved it by adding socketTimeout to the jdbc url of the postgres connection
jdbc:postgresql://...?socketTimeout=30