Hello I moved our POC datahub deployment to AWS managed serv DataHub #all-things-deployment

Hello, I moved our POC datahub deployment to AWS-m...

nutritious-tailor-97292

01/24/2024, 3:26 PM

Hello, I moved our POC datahub deployment to AWS-managed services (RDS, Opensearch, MSK) But I'm getting the following error inside the upgrade job:

Copy code

org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [datahubpolicyindex_v2]]

I'm running datahub

v0.12.0

Any ideas? I've seen this error before but in those cases it was resolved by the datahub-update routine, not originating there.

nutritious-tailor-97292

01/24/2024, 3:27 PM

datahub-upgrade.log

nutritious-tailor-97292

01/24/2024, 3:28 PM

I also made sure I had

USE_AWS_ELASTICSEARCH=true

set on the elasticsearch job

elasticsearch-setup.log

gray-ocean-32209

01/24/2024, 5:44 PM

what is your deployment framework for datahub? have copied your elastic search data to the new deployment (OpenSearch) if not you may have run the restore indices job once https://datahubproject.io/docs/how/restore-indices/ with Kubernets https://datahubproject.io/docs/how/restore-indices/#all-things-deployment

thanks bear 1

nutritious-tailor-97292

01/24/2024, 6:24 PM

We're using helm on k8s. Since we are in POC I was just starting things with fresh DBs. I just tried the restore indices and it does not solve the issue. My understanding was that this restored the content of the indices by republishing MAE events. It doesn't seem to ensure the indices exist

gray-ocean-32209

01/24/2024, 6:32 PM

Yes, restoring indices won’t create indices. you may have to run the elasticsearch-setup, kafka-setup and mysql-setup

nutritious-tailor-97292

01/24/2024, 6:33 PM

Those all run successfully but the indices aren't created. I posted the elasticsearch-setup log above

gray-ocean-32209

01/24/2024, 6:33 PM

with Helm deployment enable these respective sections https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml

gray-ocean-32209

01/24/2024, 6:39 PM

how about

datahub-system-update-job

job ?

nutritious-tailor-97292

01/24/2024, 6:39 PM

That is the first error log

gray-ocean-32209

01/24/2024, 7:03 PM

@nutritious-tailor-97292 you have shared

datahubUpgrade

logs (first one), not

datahub-system-update-job

datahub-system-update-job is where elasticsearch indices are created https://github.com/acryldata/datahub-helm/blob/f04bcdeced701edca2e7fe5bfc5a8095415bab6e/charts/datahub/values.yaml#L351

nutritious-tailor-97292

01/24/2024, 7:12 PM

Sorry, the first log I posted is actually from the system update job. I named it based off of the docker image before I realized that multiple jobs used that image. I just re-ran my deployment and the

datahub-system-update-job

failed with that same error

teamwork 1

🆗 1

nutritious-tailor-97292

01/24/2024, 7:17 PM

Here is the full log. Pretty much the same as last time. When I

/_cat/indices/

on my opensearch cluster only 2 indices are there:

Copy code

green open graph_service_v1           GhweujZxQh6wlNWUzf-87A 1 1 0 0 416b 208b
green open .kibana_1                  Eh6BSDbYSbemo6VLfpVaYQ 1 1 0 0 416b 208b
green open datahub_usage_event-000001 aJmlTza_Tr6mOQgDsiXihw 5 1 0 0  2kb  1kb

datahub-system-update-job.log

gray-ocean-32209

01/24/2024, 7:18 PM

Yeah, somehow indices are not created, checking ..

gray-ocean-32209

01/25/2024, 6:23 AM

@nutritious-tailor-97292 It looks like the reason for the actual failure is

Copy code

2024-01-24 19:11:29,607 [main] ERROR c.l.d.u.s.e.steps.BuildIndicesStep:39 - BuildIndicesStep failed.
java.lang.RuntimeException: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.EntityIndexBuilders.reindexAll(EntityIndexBuilders.java:34)
	at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.configure(ElasticSearchService.java:45)
	at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.reindexAll(ElasticSearchService.java:55)
	at com.linkedin.datahub.upgrade.system.elasticsearch.steps.BuildIndicesStep.lambda$executable$0(BuildIndicesStep.java:36)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeStepInternal(DefaultUpgradeManager.java:110)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:68)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:42)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:33)
	at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.java:80)
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164)
	at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:23)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:936)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)
	at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1901)
	at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1865)
	at org.opensearch.client.IndicesClient.create(IndicesClient.java:159)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilder.createIndex(ESIndexBuilder.java:495)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilder.buildIndex(ESIndexBuilder.java:191)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.EntityIndexBuilders.reindexAll(EntityIndexBuilders.java:32)
	... 21 common frames omitted

it indicates ElasticSearch is not responding to create index requests, you must be having some problem in ES, please take a look at ES logs when you running this deployment

nutritious-tailor-97292

01/25/2024, 2:55 PM

That doesn't fully make sense since the

graph_service_v1

index is successfully created somehow. I'm also able to

curl -XPUT <eshost>/test_index

from the GMS pod to create a test index

nutritious-tailor-97292

01/25/2024, 6:26 PM

Managed to pull this log

Opensearch logs

billions-tailor-78534

01/26/2024, 12:18 AM

What Opensearch version are you using?

billions-tailor-78534

01/26/2024, 12:19 AM

Support for Openserach 2.x was only added recently, would check if that's in availaable in

v0.12.0

nutritious-tailor-97292

01/26/2024, 2:06 PM

@billions-tailor-78534 I'm using the

Elasticsearch_7.10

version on AWS Opensearch. Do you know if the

USE_AWS_ELASTICSEARCH

flag should still be set in that case?

billions-tailor-78534

01/26/2024, 5:26 PM

@nutritious-tailor-97292 yeah that flag should be set. from your original datahub-upgrade.log file what happened is that there was an attempt to access an indice before they were built. i suspect it's mostly harmless. what is the actual problem? did the upgrade job fail? the log ends at the exception. Also the relevant parts from the log:

2024-01-24 151823,875 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - Starting upgrade with id SystemUpdate...

2024-01-24 151823,876 [main] INFO c.l.d.u.impl.DefaultUpgradeReport:16 - Executing Step 1/6: BuildIndicesPreStep...

2024-01-24 151825,997 [pool-13-thread-1] ERROR c.l.m.s.e.query.ESSearchDAO:105 - Search query failed

org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [datahubpolicyindex_v2]]

nutritious-tailor-97292

01/26/2024, 5:33 PM

The job seemed to fail as a result of that. I attached a similar run with the full log above: https://datahubspace.slack.com/archives/CV2UVAPPG/p1706123825176009?thread_ts=1706110001.974929&cid=CV2UVAPPG

billions-tailor-78534

01/26/2024, 6:41 PM

Copy code

2024-01-24 19:10:58,794 [main] INFO  c.l.m.g.e.ElasticSearchGraphService:319 - Setting up elastic graph index
2024-01-24 19:10:58,802 [main] INFO  c.l.m.s.e.i.ESIndexBuilder:491 - Index graph_service_v1 does not exist. Creating
2024-01-24 19:10:58,969 [main] INFO  c.l.m.s.e.i.ESIndexBuilder:496 - Created index graph_service_v1
2024-01-24 19:10:59,519 [main] INFO  c.l.m.s.e.i.ESIndexBuilder:491 - Index containerindex_v2 does not exist. Creating
[...]
2024-01-24 19:11:29,607 [main] ERROR c.l.d.u.s.e.steps.BuildIndicesStep:39 - BuildIndicesStep failed.
java.lang.RuntimeException: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.EntityIndexBuilders.reindexAll(EntityIndexBuilders.java:34)
	at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.configure(ElasticSearchService.java:45)
	at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.reindexAll(ElasticSearchService.java:55)
	at com.linkedin.datahub.upgrade.system.elasticsearch.steps.BuildIndicesStep.lambda$executable$0(BuildIndicesStep.java:36)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeStepInternal(DefaultUpgradeManager.java:110)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:68)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:42)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:33)
	at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.java:80)
2024-01-24 19:11:29,608 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Failed Step 2/6: BuildIndicesStep. Failed after 0 retries.
2024-01-24 19:11:29,608 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Exiting upgrade SystemUpdate with failure.
2024-01-24 19:11:29,609 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Upgrade SystemUpdate completed with result FAILED. Exiting...

billions-tailor-78534

01/26/2024, 6:41 PM

did you try running the job again? it's basically timing out from ES on creating the indices

billions-tailor-78534

01/26/2024, 6:46 PM

also maybe try creating an index manually to see whether it really takes +30s https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-example

nutritious-tailor-97292

01/26/2024, 7:17 PM

Yeah the job fails every attempt. I was able to curl the Opensearch cluster from the GMS pod and create an index essentially instantly. I know misconfigured security groups will result in timeouts, but that seems unlikely since I can curl the cluster successfully and some indices are created by datahub jobs. I'll keep digging

billions-tailor-78534

01/26/2024, 9:36 PM

Yeah that's odd.

billions-tailor-78534

01/26/2024, 9:40 PM

and it was able to create the previous index. wondering if it's an issue with the opensearch client that we use

billions-tailor-78534

01/26/2024, 9:44 PM

Can you check if there's an idle timeout setting in Elasticsearch that is 30 secs?

billions-tailor-78534

01/26/2024, 9:45 PM

or maybe there's a load balancer in front of ES?

nutritious-tailor-97292

01/30/2024, 8:20 PM

Just to confirm. On AWS should I be running the

Elasticsearch_7.10

engine or an

Opensearch_1.3

nutritious-tailor-97292

02/02/2024, 6:50 PM

To wrap up here. The culprit seemed to be something with the network routing in AWS. Our k8s was in a different cluster than the AWS clusters (RDS, MSK, Opensearch) which worked for everything except those socket timeouts. Putting them in the same VPC fixed it

gray-ocean-32209

02/02/2024, 6:56 PM

Good to hear you are able find the issue @nutritious-tailor-97292, it was good learning, Thanks informing back

3 Views

Open in Slack

Previous Next