Hi All, I am working on a flink project and I want...
# troubleshooting
r
Hi All, I am working on a flink project and I want to maximize utilization od resources. Currently I have 10 Task Managers in my cluster, each having 1 task slot. I am producing data with 10 distinct keys and I expect to use all the 10 task managers, but I end up using only 6 task managers. How do I maximize utilization. I have tried using murmurhash code to find the keys that should be directed to different parallel instances of a subtask, but nothing seems to work.
Copy code
import org.apache.flink.runtime.state.KeyGroupRangeAssignment;

public class FlinkKeyGroupMapper {

    public static void main(String[] args) {
        final int maxParallelism = 128;  // Flink maxParallelism
        final int parallelism = 10;      // Number of subtasks
        final int keysNeeded = parallelism;

        // Map: subtask index -> key found
        java.util.Map<Integer, String> subtaskToKey = new java.util.HashMap<>();

        int i = 100;
        int configId = 20003;

        while (subtaskToKey.size() < keysNeeded) {
            String key = configId + "_" + i;

            // Compute key group for this key (Flink does MurmurHash inside)
            int keyGroup = KeyGroupRangeAssignment.assignToKeyGroup(key, maxParallelism);

            // Find which operator subtask the key group maps to
            int subtaskIndex = KeyGroupRangeAssignment.computeOperatorIndexForKeyGroup(
                    maxParallelism, parallelism, keyGroup);

            // If this subtask index not assigned a key, assign it
            if (!subtaskToKey.containsKey(subtaskIndex)) {
                subtaskToKey.put(subtaskIndex, key);
                System.out.println("Subtask " + subtaskIndex + " -> Key: " + key);
            }

            i++;
        }
    }
}
this is how I have generated the keys, and my key selector in flink also uses the same key. Can anyone guide me in how can I determine which parallel instance of subtask my current key should be redirected to? Thanks
f
You probably should use one of
shuffle()
,
rebalance()
or
rescale()
rather than generate keys in such a weird way