Hi everyone, I'm trying out Flink CDC 1.20.3 in K...
# troubleshooting
a
Hi everyone, I'm trying out Flink CDC 1.20.3 in K8s and running into an issue while using S3 for checkpoints. I have added the flink-s3-fs-presto/1.20.3 plugin to my image. While running a simple postgres to postgres cdc job, I see this error in the cdc session pod. I am using S3 compatible storage Versity gateway as storage for checkpoints. Error:
com.amazonaws.services.s3.model.AmazonS3Exception: " (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: "; S3 Extended Request ID: "; Proxy: null)
Session Pod Configuration:
Copy code
spec:
  flinkConfiguration:
    classloader.resolve-order: parent-first
    state.checkpoints.dir: 'file:///tmp/checkpoints'
    state.savepoints.dir: 'file:///tmp/savepoints'
    execution.checkpointing.interval: '30s'
    sql-gateway.endpoint.rest.address: 0.0.0.0
    sql-gateway.endpoint.rest.port: 8083
    execution.checkpointing.storage: "filesystem"
    execution.checkpointing.dir: "<s3p://flink/flink-checkpoints/flink-green>"
    execution.checkpointing.savepoint-dir: "<s3p://flink/flink-checkpoints/flink-green>"
    execution.checkpointing.interval: "30s"
    execution.checkpointing.incremental: true

    s3.endpoint: "<https://s3.xxx>"
    s3.path.style.access: "true"
    s3.access.key: "flink"
    s3.secret.key: "xxxxxxxxxxxx"
    s3.ssl.enabled: "true"
    s3.ssl.verification.enabled: "true"
    s3.use-path-style-access: "true"
Full log file is attached. Any ideas on what could be the issue?
For anyone who comes across this issue, it seems like the presto plugin automatically takes the region from the endpoint URL. So if you have a custom S3 url like https://s3.dev.com the "dev" part will get sent as the region even if we explicitly specify region separately. This is why we get bad request error on S3 side. So if you have custom s3 its better to use a url that doesnt start with "s3"