https://datahubproject.io logo
Join Slack
Powered by
Hi! I have just upgraded from `0.8.15` to `0.8.16`...
# troubleshoot
w
Hi! I have just upgraded from 
0.8.15
to 
0.8.16
and GMS is not able to start. Found a couple of exceptions suggesting there is an issue with ES indexes. Any idea how we can overcome this?
Copy code
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'metadataAuditEventsProcessor' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-244284873505208431.dir/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/MetadataAuditEventsProcessor.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.kafka.MetadataAuditEventsProcessor]: Constructor threw exception; nested exception is java.lang.RuntimeException: Reindex from datasetindex_v2 to datasetindex_v2_1635345340490 failed
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:314)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:295)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1358)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
and
Copy code
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.kafka.MetadataAuditEventsProcessor]: Constructor threw exception; nested exception is java.lang.RuntimeException: Reindex from datasetindex_v2 to datasetindex_v2_1635345340490 failed
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:216)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:310)
	... 41 common frames omitted
Caused by: java.lang.RuntimeException: Reindex from datasetindex_v2 to datasetindex_v2_1635345340490 failed
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.IndexBuilder.buildIndex(IndexBuilder.java:112)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.EntityIndexBuilder.buildIndex(EntityIndexBuilder.java:24)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilders.buildAll(ESIndexBuilders.java:22)
s
One way would be to go and drop the ES index. As this is dataset related everything should be in mysql. You can go through the guide for re-indexing. Weird I didn't see it while upgrading from 
0.8.14
to 
0.8.16
.
w
you mean delete 
datasetindex_v2
and then run 
RestoreIndices
https://datahubproject.io/docs/docker/datahub-upgrade/ ?
s
yes. You would probably need to re-run the elastic setup job before restore Indices.
delete index -> run job to setup index -> restore indices
this if you are stuck. If you can rollback then maybe wait for a team member to find the cause and fix that
w
Copy code
/ $ curl -X GET "https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_cat/indices"
yellow open datahub_usage_event-000015                            roJIasnqQgWrgfWiRaxzyg 5 1   151    0   382kb   382kb
yellow open datahub_usage_event-000014                            c5CqgSnGSVOKxMlbFc9iqQ 5 1    99    0 308.4kb 308.4kb
yellow open datahub_usage_event-000017                            ae0pUrBTQUCQfrAF6X7cEQ 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000016                            A2-hPLjYSp2IRPXnkuAQng 5 1   235    0 691.2kb 691.2kb
yellow open corpuserindex_v2_1635341699800                        ZsGKi2fhRzGeUmBNsyRtjw 5 1  9097    0   3.9mb   3.9mb
yellow open datahub_usage_event-000019                            tTs2KLaNRQui6TueBz5M1w 5 1   317    0 788.3kb 788.3kb
yellow open glossarytermindex_v2_1633440697713                    2oyaSW5fTVaqNw51Ej9SCQ 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000018                            DbEIuS5jQw6WDeNBh5YYhw 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000051                            0-ygiJDkS12wvEM6zHQ7Nw 5 1   115    0 242.7kb 242.7kb
yellow open datahub_usage_event-000050                            SQpc_Oq1SKSWA38eCkMK_g 5 1     1    0  13.4kb  13.4kb
yellow open datahub_usage_event-000053                            Q1-zySySTiiDg63uyOHK2g 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000052                            oI129IA0RIieVBzYinb4-g 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000011                            6mB6HmX6QZSTZUwGn3ZkrQ 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000055                            Up5yYvCeSCyYyI0esDUrYg 5 1    54    0 247.6kb 247.6kb
yellow open datahub_usage_event-000054                            u4_VTxtBRnW0SGqbq4OBvA 5 1   187    0 514.4kb 514.4kb
yellow open datahub_usage_event-000010                            nll_ym8FTR6HXwPhRHL_yg 5 1     6    0  30.9kb  30.9kb
yellow open datahub_usage_event-000013                            iFys_vblRe-aoDExZ_8GjQ 5 1    46    0   194kb   194kb
yellow open datahub_usage_event-000056                            km6COUVZRrONycWoVSJaEw 5 1    45    0 333.3kb 333.3kb
yellow open datahub_usage_event-000012                            5B8FukbETwGOHWNt7rhXbA 5 1   246    0 593.8kb 593.8kb
yellow open dataplatformindex_v2_1633440683829                    gAtgdtgLR0WxeRq0DSCwCg 5 1     0    0     1kb     1kb
yellow open tagindex_v2_1633440696141                             zQ3un9iUSYaEwSbiVsD0CA 5 1     6    5  37.5kb  37.5kb
yellow open datajobindex_v2_1635341713296                         iMbI8EaCS9SXWaYeE3kRqg 5 1   303   88 399.8kb 399.8kb
yellow open datahub_usage_event-000026                            Ya8uRfmlSkCUC2h48IkVFA 5 1    68    0 219.2kb 219.2kb
yellow open datahub_usage_event-000025                            cOJCH8rEQcmUNzgnKNt5VA 5 1     0    0     1kb     1kb
yellow open dataset_datasetprofileaspect_v1_1633440704720         1nfm0J_yRUCJtxUQYdbtIw 5 1  3823    0   4.6mb   4.6mb
yellow open datahub_usage_event-000028                            pjW3jb9IThelLhbbGs4H-g 5 1     0    0     1kb     1kb
yellow open mlmodelindex_v2_1635341707160                         Y9VdtwXFS6OSeEhGwGivDA 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000027                            9BCvP6WhQz6dAe8eTEIRAA 5 1    47    0 109.2kb 109.2kb
yellow open datahub_usage_event-000029                            uQsirTzmQj6BRXmhbALirw 5 1    61    0 263.1kb 263.1kb
yellow open datahub_usage_event-000020                            mS6gVM9CRsSEgoAtoYqINQ 5 1    30    0 121.6kb 121.6kb
yellow open graph_service_v1                                      SFnV_KzKQiGmJhcuvxk3nA 5 1 19677 1720   2.2mb   2.2mb
yellow open datahub_usage_event-000022                            VDcmRppwS3CUZdLHow_eQw 5 1     0    0     1kb     1kb
yellow open system_metadata_service_v1                            vpZsnJW5T4-d7fv6crb0Qg 5 1 91291   52  11.9mb  11.9mb
yellow open datahub_usage_event-000021                            MVWb6Oi6TLa_DhunLP8jug 5 1    20    0    92kb    92kb
yellow open datahub_usage_event-000024                            Sx8AOK3qSLaJs1DBeceEqg 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000023                            ZT6mi4nOT063OflMdS9Fww 5 1   106    0 315.7kb 315.7kb
yellow open schemafieldindex_v2                                   ExeYbGJcR2CZS60GU-uKLA 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000037                            VrmXiGcETpe1fkeRENZxiA 5 1   365    0 647.4kb 647.4kb
yellow open datahub_usage_event-000036                            6hu9LPLSRtmnd9uxHNOQcQ 5 1   610    0     1mb     1mb
yellow open mlfeatureindex_v2_1635341710392                       Kzg36HJtTbuYLpPW_n-FIw 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000039                            ZchNhsQ8SCiMTQHTQrpXfA 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000038                            KhbVEuocSWe9f8qFVuisag 5 1     0    0     1kb     1kb
yellow open corpgroupindex_v2_1635341706070                       zgFPSD0SRSmrbBms72QgQg 5 1  1391    0     1mb     1mb
yellow open mlmodeldeploymentindex_v2_1635341698979               zVDrPCJCSYCgH2XGGsLwmw 5 1     0    0     1kb     1kb
yellow open glossarynodeindex_v2_1633440690988                    _GXkA8K6R7qm5h_7KKPQnA 5 1     0    0     1kb     1kb
yellow open usagestats_v1                                         xXOGb2PGTWeY42MEU8i5pQ 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000031                            VA-GPY1gTNuWm1R28fH3GA 5 1   205    0 839.7kb 839.7kb
yellow open datahub_usage_event-000030                            AYR4nD2tR5C-2BLS8Mb0iQ 5 1   150    0 623.5kb 623.5kb
yellow open datahub_usage_event-000033                            Swy_zeJ_ROaCCe9XmnRErQ 5 1    81    0 366.4kb 366.4kb
yellow open datahub_usage_event-000032                            HDi2fTVoQICf0jw0BR_DYw 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000035                            rGZoF18wR2StW0dO79z0iA 5 1    84    0 383.7kb 383.7kb
yellow open datahub_usage_event-000034                            73vS6ki3QneMLPFa1oib-w 5 1    73    0 411.8kb 411.8kb
yellow open mlfeaturetableindex_v2_1635341708317                  c0IB5-KpQoCgNbnnH8mcgw 5 1     0    0     1kb     1kb
yellow open chartindex_v2_1635345372236                           T4-aKSvkRJiYbp0UJ6S5Pg 5 1     0    0     1kb     1kb
yellow open dataprocessindex_v2_1633440689327                     _yXumie3SAiZpyLo8IwkeA 5 1     0    0     1kb     1kb
yellow open datahubpolicyindex_v2_1633440684377                   UiEPE26tTwmnWG20W0FD-A 5 1     0    0     1kb     1kb
yellow open dataset_datasetusagestatisticsaspect_v1_1635345373494 qodIFBqQQreFfDmKrZ0qKw 5 1     0    0     1kb     1kb
yellow open dataflowindex_v2_1635341711316                        TzqR4QYZTo29WvBjpInNCA 5 1   303  113 342.3kb 342.3kb
yellow open datahub_usage_event-000048                            1GceJe7XTKq5qN_OL8ppKw 5 1    44    0   254kb   254kb
yellow open datahub_usage_event-000004                            RbdWPn_LSY25h-DApME9_Q 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000047                            FovcbEB-S5mQBAPr-ne-nQ 5 1   186    0 625.2kb 625.2kb
yellow open datasetindex_v2_1635345366366                         KkeiLKpHQZ-Kx9lcEa2FhQ 5 1 10125  404  16.9mb  16.9mb
yellow open datahub_usage_event-000003                            deJI8soBRfCj4wPoqBzdUA 5 1    22    0 150.2kb 150.2kb
yellow open mlprimarykeyindex_v2_1635341715465                    U_IGQ8ayQTWU0gOTTCIy0w 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000006                            zjO4ZMVjRKiEyKkVVqifnw 5 1   165    0 558.4kb 558.4kb
yellow open datahub_usage_event-000005                            ceJ46PriRpShEBYTgGpPyw 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000049                            7JMC3xLyTYa5vTxCeVOw_Q 5 1   114    0 182.8kb 182.8kb
yellow open datahub_usage_event-000008                            zvmyjygeStWJmSqQV7-TAw 5 1    81    0 234.4kb 234.4kb
yellow open datahub_usage_event-000007                            bxToJrNUQZCBpjns36Dxeg 5 1   190    0 516.5kb 516.5kb
yellow open datahub_usage_event-000009                            h6jxctYwQZ6S1Q4oJbwTHQ 5 1    16    0  80.3kb  80.3kb
yellow open datahub_usage_event-000040                            wF7UXOmBRb26Kzia_W0ZiA 5 1    65    0 282.5kb 282.5kb
yellow open datahub_usage_event-000042                            0tk5CZiUTLi7rPDzrEdFQg 5 1   196    0 519.5kb 519.5kb
yellow open datahub_usage_event-000041                            eypXOxKgTAqjXlassXpwdQ 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000044                            YZK5fHUvTN2AkDSltZPNRw 5 1   190    0 426.6kb 426.6kb
yellow open datahub_usage_event-000043                            JACy-LxeT7KbplAmA4QaMA 5 1   198    0 584.8kb 584.8kb
green  open .kibana_1                                             HjIPsKLATw-o2u_YxZs45g 1 0     1    0     5kb     5kb
yellow open datahub_usage_event-000046                            vtk5-B3YQ5aXsrM9sFkL2A 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000002                            Ipz513zoRueBNOlsZ6Sogw 5 1   226    0 656.8kb 656.8kb
yellow open datahub_usage_event-000045                            hglmchIJSZuFv2DhjpIUCQ 5 1     0    0     1kb     1kb
yellow open datahub_usage_event-000001                            Qr5dZo41QCi8BfOjeB5szA 5 1   213    0 665.6kb 665.6kb
yellow open dashboardindex_v2_1635345372792                       uURmxQjNSumm3uEFR8w5_g 5 1     0    0     1kb     1kb
yellow open mlmodelgroupindex_v2_1635341709305                    a7RkBS10SCC-OjYD7JuhmA 5 1     0    0     1kb     1kb
yellow open .opendistro-job-scheduler-lock                        jNRaBsh0QbS_9UgK7OOrNA 5 1    56   14 198.8kb 198.8kb
which one should I delete? or all of them? 🤔
1. 
curl -X DELETE "https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_all
2. run elastic setup job 3. 
docker pull acryldata/datahub-upgrade:head && docker run --env-file docker.env acryldata/datahub-upgrade:head -u RestoreIndices
4. 🤞 Is that the recipe? 🙂
e
@witty-butcher-82399 does gms have permissions to reindex (or create new indices) ?
every time there is a change to the underlying data model and it requires a change in the index definition, it does an automatic reindexing of existing entity indices (thigns that end in index_v2)
these are not backward incompatible so the reindexing should never fail unless it lacks the permissions to do so
If this all doesn’t work, yes you could delete all indices that end in index_v2 so *index_v2
and then restart gms
FYI now indices are created during gms (or mae consumer if run separately) start up. setup job just sets up the datahub_usage_event index
But let’s make sure gms has proper permissions to create indices
If you do end up going the deleting index route, you will have to restore indices by following this guide! https://datahubproject.io/docs/how/restore-indices/#docker-compose
b
Hi everybody, I work together with Sergio on the same datahub initiative. In the 
dev
environment where we attempted the upgrade, Elasticsearch is not protected behind auth so I would exclude a problem related to permissions. Our deployment is based on the helm charts that you provide and I noticed that as part of the 
datahub-upgrade
templates there is a job that seems to do exactly the operations you suggested to perform manually: < https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml | restore indices job > Now my question is: is this a valid alternative to execute in order to automate the restore of the indices (my idea would be to use it as a disaster recovery tool as well), or is it only meant as a one-off tool for the no-code update? In the latter case I think it would be great to release it as a general purpose utility. Btw, after examining the code I decided to give it a go and it fixed our deployment. However it raised an exception while executing:
Copy code
Sending MAE from local DB...
Found 96275 latest aspects in aspects table
Reading rows 0 through 1000 from the aspects table.
Caught exception during attempt 0 of Step with id SendMAEStep: java.util.IllegalFormatConversionException: a != java.lang.String
Retrying 0 more times...
Failed Step 4/4: SendMAEStep. Failed after 0 retries.
Exiting upgrade RestoreIndices with failure.
Upgrade RestoreIndices completed with result FAILED. Exiting...
e
It is a valid way to restore indices! The command you ran is exactly how we intended for it to work as mentioned in the kubernetes section https://datahubproject.io/docs/how/restore-indices
Seems like there was an invalid aspect in MySQL. Did the job still go through and restore all other aspects? Or did it just fail
b
Seems like there was an invalid aspect in MySQL. Did the job still go through and restore all other aspects? Or did it just fail
It was reported as failed, however it went through and restored most of the aspects (this is what we observed). Not sure how to debug it further though
p
b
We investigated this further and we got to the cause of the problem. I opened an issue to describe it: https://github.com/linkedin/datahub/issues/3625 I also opened a PR: https://github.com/linkedin/datahub/pull/3626 but I’m not sure on/if adding tests for regression
2 Views
Open in Slack
PreviousNext
Powered by