hi guys, I have huge problem with doing stream job...
# troubleshooting
m
hi guys, I have huge problem with doing stream job from cassandra, I need to process whole table ( basically for each row I need based on it fields create record in another table ) so I do:
Copy code
ClusterBuilder clusterBuilder = buildCassandraCluster();

        CassandraSource<HistoryReadout> historyReadoutCS = new CassandraSource<>(
                clusterBuilder,
                HistoryReadout.class,
                String.format("SELECT * FROM %s.data;", keyspace),
                () -> new Mapper.Option[]{Mapper.Option.saveNullFields(true)}
        );
then:
Copy code
DataStreamSource<HistoryReadout> cassHistReadoutSrc =
                env.fromSource(historyReadoutCS, WatermarkStrategy.noWatermarks(), "CassHistReadoutSrc");
and:
Copy code
SingleOutputStreamOperator<Tuple2<String, HistoryReadout>> processedStream = cassHistReadoutSrc.process(new ProcessFunction<HistoryReadout, Tuple2<String, HistoryReadout>>() {
            @Override
            public void processElement(HistoryReadout readout, ProcessFunction<HistoryReadout, Tuple2<String, HistoryReadout>>.Context ctx, Collector<Tuple2<String, HistoryReadout>> out) throws Exception {
                System.out.println("In process element" + readout);
                try {
                    String tableName = HistoryReadoutCassandraServiceUtil.getRowTableName(keyspace, readout);
                    System.out.println(tableName);
                    out.collect(new Tuple2<>(tableName, readout));
                } catch (Exception e) {
                    System.err.printf("Error generating table name for record: %s, Error: %s%n", readout, e.getMessage());
                }
            }
        });
at the end:
Copy code
processedStream.addSink(new SinkFunction<>() {
            private transient Session cassandraSession;

            @Override
            public void invoke(Tuple2<String, HistoryReadout> value, Context context) {
                String tableName = value.f0;
                HistoryReadout readout = value.f1;
                if (cassandraSession == null) {
                    cassandraSession = createSinkCassandraSession();
                }

                try {
                    System.out.printf("Inserting into table: %s, Record: %s%n", tableName, getLogStr(readout));
                    HistoryReadoutCassandraServiceUtil.processRecordInsert(cassandraSession, tableName, readout);
                } catch (Exception e) {
                    System.err.printf("Error processing record insert: %s, Error: %s%n", readout, e.getMessage());
                }
            }

            private Session createSinkCassandraSession() {
                ClusterBuilder clusterBuilder = buildCassandraCluster();
                return clusterBuilder.getCluster().connect();
            }


            private String getLogStr(HistoryReadout readout) {
                return readout.tenantid + "-" + readout.iodeviceid + "-" + readout.discriminator + "-" + readout.createdat + "-" + readout.iodevicetype + "-" + readout.serializerid;
            }
        });

        env.execute("Flink Cassandra Connector Job");
    }
and this code works if table is small, but for bigger tables I see only:
Copy code
com.datastax.driver.core.SessionManager$State@57666b8
com.datastax.driver.core.SessionManager$State@74ca0805
com.datastax.driver.core.SessionManager$State@228b352f
com.datastax.driver.core.SessionManager$State@642b3042
com.datastax.driver.core.SessionManager$State@1a5f3752
and nothing happens for a couple of minutes I wonder why, after few minutes I see stream is running, but I am testing on dev environment, in prod I will need to mirgate 2 TB table I need to bne sure that this code is good and does not for example fetch whole table into memory or smth