Bumping this again for visibility - I’ve been struggling with a recent upgrade of an Elasticsearch client to 8.x and now I’m in a stuck situation. I’ve tried several attempts to work around it but I’m not sure what the best course of action to take is. I don’t know if it’s possible to override the underlying version of the RHLC used in the Flink bits to support some of the REST API Compatibility mode headers (for interop between 7.x and 8.x) or if I’m in a situation where I need to fully fork the connector itself. As mentioned in earlier, related posts, there don’t appear to be any publicly available version(s) for the ES 8.x connector in repositories like Maven, so I’m not sure if the general guidance is to fork it directly.

It’s possible that forking and modifying is necessary but there are a few things to check first. One is if there are any patches or snapshot release and check for issues under both the connector & flink. In fact have you raised the issue under the repo for Flink connector?

Thanks for the response @D. Draco O'Brien, I think the entire forking discussion is only temporary because there isn’t yet any artifacts in Maven for the ES 8.x connector. I don’t really want to fork anything for the maintenance issues that you mentioned, but I don’t see how else others are currently using the new connector bits without doing so.

yeah so the setting up the intermediate cluster might be the best option for now. Wish there were better options

Last time I worked with this was last year so there may be better solutions. At the time there was no update of the ES sink for 8. This is due to the depreciation of the high level rest client with the new Java API. It is just better, but it is completely different making a straight upgrade of the existing sink unrealistic.

Yup, in my situation I eventually just forked the existing ES 7.x connector and used the updated version of the underlying REST client to support compatibility mode and passed the headers to the sinks which seems to work without any major code changes to the jobs themselves. I’m hoping by the time I don’t have to be concerned with maintaining the ES 7.x clusters that the official connector is available by then.

I wrote my own 8 connector unfortunately it was for my previous company and I couldn't open source it