Howdy! I'm looking to traverse state using the st...
# troubleshooting
b
Howdy! I'm looking to traverse state using the state processor API, however my flink jobs are often a mix of datastream and Table/SQL APIs. I could potentially manually register state descriptors for a job on the datastream portions, follow best practices such as consistent UUIDs on operators, etc.. but for Table/SQL, since that is all generated via the planner, I'm not sure if there is a path forward. Does a methodology exist for interrogating a job graph and building a mapping of state descriptors, etc...?
a
Hi, to my knowledge, Table API jobs do not support UUIDs in the graphs and their state easy to break, so that only thing you can do is to throw away job state and build new state from scratch. I remember there are some FLINK tickets to bring some UIDs for SQL jobs.
a
I have done this (state decoding for enumerator state) using the flink state api for the ddb connector. Feel free to reach out to me for sample code
b
@Abhi Gupta - that sounds really cool, and sounds like a learning opportunity for me either way! Any reference code would be much appreciated 🙂
a
Yeah i can put the code here. Gimme a bit. Its in my intellij somewhere ill have to find it
🙌 1
Copy code
public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment see = StreamExecutionEnvironment.getExecutionEnvironment();
        String basePath = "<some path>";
        String savepointPath = basePath;

        SavepointReader sr = SavepointReader.read(see, savepointPath + "/<savepointpath>");

        // Create Field object
        Field privateField
                = SavepointReader.class.getDeclaredField("metadata");
        privateField.setAccessible(true);
        SavepointMetadataV2 metadata = (SavepointMetadataV2) privateField.get(sr);

        for (OperatorState state : metadata.getExistingOperators()) {
            if (state.getCoordinatorState() != null) {
                saveState(metadata, OperatorIdentifier.forUidHash(state.getOperatorID().toString()), basePath);
            }
        }

    }
    public static void saveState(SavepointMetadataV2 metadata, OperatorIdentifier operatorIdentifier, String basePath) throws Exception {
        String uid = operatorIdentifier.toString();

        DynamoDbStreamsShardSplitSerializer splitSerializer = new DynamoDbStreamsShardSplitSerializer();
        DynamoDbStreamsSourceEnumeratorStateSerializer enumeratorStateSerializer = new DynamoDbStreamsSourceEnumeratorStateSerializer(splitSerializer);

        OperatorState state = metadata.getOperatorState(operatorIdentifier);
        byte[] coordinatorState = state.getCoordinatorState().getData();
        byte[] enumeratorBytes = Arrays.copyOfRange(coordinatorState, 12, coordinatorState.length);
        Map<String, DynamoDBStreamsShardSplitWithAssignmentStatus> knownSplits = new HashMap<>();
        Map<String, DynamoDBStreamsShardSplitWithAssignmentStatus> allKnownSplits = new HashMap<>();
        FileOutputStream out_known = new FileOutputStream(basePath + "/known_splits_" + uid + ".txt");

        DynamoDbStreamsSourceEnumeratorState enumerator = enumeratorStateSerializer.deserialize(0, enumeratorBytes);
        for (DynamoDBStreamsShardSplitWithAssignmentStatus s : enumerator.getKnownSplits()) {
            if(s.assignmentStatus() == SplitAssignmentStatus.ASSIGNED) {
                knownSplits.put(s.split().getShardId(), s);
            }
            allKnownSplits.put(s.split().getShardId(), s);
            out_known.write((s + "\n").getBytes());
        }
        out_known.close();


        Map<String, DynamoDbStreamsShardSplit> assignedSplits = new HashMap<>();
        FileOutputStream out = new FileOutputStream(basePath + "/splits_" + uid + ".txt");
        for (OperatorSubtaskState os : state.getSubtaskStates().values()) {
            for (OperatorStateHandle osh : os.getManagedOperatorState()) {
                byte[] stateBytes;
                stateBytes = osh.asBytesIfInMemory().orElseGet(() -> {
                    Path path = Paths.get(basePath, ((RelativeFileStateHandle) osh.getDelegateStateHandle()).getRelativePath());
                    try {
                        return Files.readAllBytes(path);
                    } catch (IOException e) {
                        throw new RuntimeException(e);
                    }
                });
//                byte[] stateBytes = osh.asBytesIfInMemory().get();
                for (OperatorStateHandle.StateMetaInfo smi : osh.getStateNameToPartitionOffsets().values()) {
                    long[] offsets = smi.getOffsets();
                    for (int i = 0; i < offsets.length; i++) {
                        int start = (int) offsets[i] + 12;
                        int end = 0;
                        if (i < offsets.length - 1) {
                            end = (int) offsets[i + 1];
                        } else {
                            end = stateBytes.length;
                        }
                        byte[] bytes = Arrays.copyOfRange(stateBytes, start, end);
                        DynamoDbStreamsShardSplit split = splitSerializer.deserialize(0, bytes);
                        assignedSplits.put(split.getShardId(), split);
                        if (split.isFinished()) {
                            System.out.println("Finished split: " + split + " uid: " + uid);
                        }
                        out.write((split + "\n").getBytes());
                    }
                }
            }
        }
        out.flush();
        out.close();

        knownSplits.keySet().removeAll(assignedSplits.keySet());
        System.out.println("Missing splits in " + operatorIdentifier + ": " + knownSplits.keySet());
    }
🙇‍♂️ 1
🤯 1
this is decoding the ddb connector specific code, but the idea remains the same essentially @Brad Murry
in this particular case, i knew that my specific state started at byte 12, for your specific job, you'll have to initially manually look at the state to find which specific point your particular state is starting 😛
😮 1
you can use the unix command "less" to look at your flink job savepoint / checkpoint to understand the way the checkpoint would be written
b
This is a great starting point, really appreciate the share, @Abhi Gupta!
🙌 1
a
but i'd still want someone with more knowledge on flink to understand if theres a better way, since the script is really hacky and isnt generalized
d
An option to consider is that with ProcessTableFunctions (PTFs) you have explicit control of the state and the UUIDs. So for any SQL operators you are willing to replace with PTFs, you can use the state processor API to manipulate the state.