Hey, can someone from the datahub team clarify your stance on openserach/elasticsearch? With EOL for elasticsearch 7.x coming whenever ES 9.0 releases, the people from our organisation are pushing to upgrade our on premises deployments to ES 8, but with datahub I#m getting quite a few errors when trying ES 8. I wanted to test the compatability mode for the restHighLevelClient and this is when i noticed that datahub is using the opensearch clients. So what are the plans regarding search, will there ever be an update for datahub which works with ES 8?

@brainy-tent-14503 would love your thoughts here!

We currently don’t have ES 8 support on the road map. The change to opensearch client allowed for supporting ES 7.x, OS 1.x, OS 2.x (new) which was an easier update and also requested by the community. Moving to a newer 7.x or 8.x client requires a large refactor since the restHighLevelClient is deprecated (7.15) and likely we would have to maintain two implementations OS 3.x and ES8.x going forward. The compatibility mode I believe was only viable on newer 7.x clients, we were stuck at 7.10 to be able to continue supporting OS instances. In short, there is going to need to be a major refactor to support both OS and ES unless there is some common client that I don’t know of yet. That effort is not planned at the moment.

Thank you David, then i will probably have to look into deploying it with opensearch instead of elastic

it would be good if the team make an advisory on continued ES7 usage, @little-megabyte-1074

@brainy-tent-14503 would it be possible to continue using restHighLevelClient with a newer 7 major version(even if it is deprecated) to enable compatibility with ES 8.x? See https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html. Is there an independent requirement to stay on 7.10? Does supporting OS instances strictly depend on 7.10 or the existance of restHighLevelClient?

Its not clear whether the Opensearch Java Rest High level client 2.x would be compatible with ES 8.x in compatibility mode. I haven’t tested it, but feel free to give it a try once the next release v0.12.0 is released.