Hi. I’m running update from 0.9.5 to 0.10.1 (deplo...
# troubleshoot
i
Hi. I’m running update from 0.9.5 to 0.10.1 (deployed in EKS with Helm). It looks like, reindex job does not close ELK indices. I’m trying to complete update in DEV and this reindex is never ending, because there is always mismach in document counts due to open indices. It end-up spawning 1000 shards on EKS cluster and I gave up restoring indices from snapshot. Please find more details below (this is live log, update-job is running, but indices are open). Am I doing something wrong? Job datahub-datahub-system-update-job:
Copy code
2023-04-08 01:15:49.352  INFO 1 --- [           main] c.l.m.s.e.indexbuilder.ESIndexBuilder    : Created index dataprocessinstanceindex_v2_1680916549224
2023-04-08 01:16:49.405  INFO 1 --- [           main] c.l.m.s.e.indexbuilder.ESIndexBuilder    : Task: yLvZIFxaQ8Km_Dg8_0vdHg:464294584 - Reindexing from dataprocessinstanceindex_v2 to dataprocessinstanceindex_v2_1680916549224 in progress...
2023-04-08 01:17:49.441  WARN 1 --- [           main] c.l.m.s.e.indexbuilder.ESIndexBuilder    : Task: yLvZIFxaQ8Km_Dg8_0vdHg:464294584 - Document counts do not match 3036994 != 79135. Complete: 2.6057014%
2023-04-08 01:17:50.441  INFO 1 --- [           main] c.l.m.s.e.indexbuilder.ESIndexBuilder    : Task: yLvZIFxaQ8Km_Dg8_0vdHg:464294584 - Reindexing from dataprocessinstanceindex_v2 to dataprocessinstanceindex_v2_1680916549224 in progress...
2023-04-08 01:18:50.489  WARN 1 --- [           main] c.l.m.s.e.indexbuilder.ESIndexBuilder    : Task: yLvZIFxaQ8Km_Dg8_0vdHg:464294584 - Document counts do not match 3037042 != 123981. Complete: 4.0822945%
2023-04-08 01:18:52.492  INFO 1 --- [           main] c.l.m.s.e.indexbuilder.ESIndexBuilder    : Task: yLvZIFxaQ8Km_Dg8_0vdHg:464294584 - Reindexing from dataprocessinstanceindex_v2 to dataprocessinstanceindex_v2_1680916549224 in progress...
indices:
Copy code
$ curl -XGET https://redacted:443/_cat/indices?s=index:asc
green  open .kibana_1                                                _v7DnFkWTfiNG9bsgVBmbg 1 0       4     3   22.9kb   22.9kb
yellow open .opendistro-job-scheduler-lock                           gXdKjsz8TmyVxQOZO7JcVA 5 1      58   896    405kb    405kb
green  open .opensearch-observability                                ne6d_vqlS7azmrr7rs8MmA 1 0       0     0     208b     208b
green  open .tasks                                                   JmDYsKzQSByAOe9_4FnXdw 1 0       2     0   13.8kb   13.8kb
yellow open assertion_assertionruneventaspect_v1                     E4Oy5K5LSMuw9sCoBdG3wg 1 1       0     0     208b     208b
yellow open assertionindex_v2_1680916341451                          MeSGfdE2TDCT3P8NzLJVow 1 1       0     0     208b     208b
yellow open assertionindex_v2_clone_1680916209740                    SZ5x1BmoRb-XQog-ldzVLg 1 1       0     0     208b     208b
yellow open chart_chartusagestatisticsaspect_v1                      6sK_wp9qTf6YfbTNHU3cvA 1 1       0     0     208b     208b
yellow open chartindex_v2_1680916508743                              FX-Q3t6VQgel_29jwYQwcA 1 1       0     0     208b     208b
yellow open chartindex_v2_clone_1680916215081                        EPWNO6eORKmlyvvU4LdzpA 1 1       0     0     208b     208b
yellow open containerindex_v2_1680916217544                          GdHa6mUWQJCnNchf8kc5pg 1 1     295     0  445.4kb  445.4kb
yellow open containerindex_v2_clone_1680916205327                    aZ9Y6FpuTLq-C0nHUPQLgw 1 1     295     1  240.2kb  240.2kb
yellow open corpgroupindex_v2_1680916298952                          c0k6PZlLThGxolcV2jcCuw 1 1    9039     0   33.5mb   33.5mb
yellow open corpgroupindex_v2_clone_1680916207464                    lOwXCdCrQba0R52yqQ1UoQ 1 1    9039     3      7mb      7mb
yellow open corpuserindex_v2_1680916385514                           J7Pm9vamT7ChRHZH58-szQ 1 1   54570     0  108.2mb  108.2mb
yellow open corpuserindex_v2_clone_1680916212508                     sbBpUSPfTPOXGhlIgwSXbA 1 1   54570     1   30.1mb   30.1mb
yellow open dashboard_dashboardusagestatisticsaspect_v1              j4zayejpSwSxbwa_zT7umw 1 1       0     0     208b     208b
yellow open dashboardindex_v2_1680916383282                          IUviBU6qSz2p-ixrJ3_a5Q 1 1       0     0     208b     208b
yellow open dashboardindex_v2_clone_1680916211075                    6fPmGrUHQKmD3IVaGaNjOw 1 1       0     0     208b     208b
yellow open dataflowindex_v2_1680916447029                           moxh5o89Ruugr0Jp4snkpA 1 1     351     0  880.8kb  880.8kb
yellow open dataflowindex_v2_clone_1680916213195                     lU3789SoR5q6El2ILp6vew 1 1     351     8  469.6kb  469.6kb
yellow open datahubaccesstokenindex_v2_1680916258241                 icRmUSt-TXqbUj1DS6dDEg 1 1       2     0   16.3kb   16.3kb
yellow open datahubaccesstokenindex_v2_clone_1680916206271           6j4VQAnMQ0CVwGoJJYWT1g 1 1       2     0   14.8kb   14.8kb
yellow open datahubexecutionrequestindex_v2_1680916320699            soLegwj1Q4Oi4YeDgcWfVQ 1 1       0     0     208b     208b
yellow open datahubexecutionrequestindex_v2_clone_1680916209016      TPcHVGCORfyxspzgJwlbzw 1 1       0     0     208b     208b
yellow open datahubingestionsourceindex_v2_1680916487702             VSCr-ptJQhOz8at3xv7Hyg 1 1       0     0     208b     208b
yellow open datahubingestionsourceindex_v2_clone_1680916213912       gweGm5jMTIqpVDCihNV3Yg 1 1       0     0     208b     208b
yellow open datahubpolicyindex_v2_1680916238023                      BuJtg-s_Sjq-1fp5wAhxRA 1 1      15     0     43kb     43kb
yellow open datahubpolicyindex_v2_clone_1680916205715                8ms_cGW_R8KYrUrcaA1UkQ 1 1      15     2   50.3kb   50.3kb
yellow open datahubretentionindex_v2_1680916385221                   -g4aYQilSOiRwLWnJOyhEA 1 1       0     0     208b     208b
yellow open datahubretentionindex_v2_clone_1680916212282             qkmMdoB_T8OojE-KEXwHwQ 1 1       0     0     208b     208b
yellow open datahubroleindex_v2_1680916278714                        wPxRPENxQdS46Y-L_eW2AQ 1 1       3     0   11.6kb   11.6kb
yellow open datahubroleindex_v2_clone_1680916207180                  3F9I5cySTtaJQcRaGbRdyA 1 1       3     0    7.7kb    7.7kb
yellow open datajob_datahubingestioncheckpointaspect_v1              CHuumlxhTqW2BJbiDk5xhQ 1 1     356     0    6.2mb    6.2mb
yellow open datajob_datahubingestionrunsummaryaspect_v1              pzAyIz42Rbif9dZOnOXmCQ 1 1       0     0     208b     208b
yellow open datajobindex_v2_1680916321205                            cmO1uwNJR3O967kXcD_67Q 1 1    3547    26    6.3mb    6.3mb
yellow open datajobindex_v2_clone_1680916209481                      j-V2kltTQw-a1Cbssg_Ztw 1 1    3547   469    3.5mb    3.5mb
yellow open dataprocessinstance_dataprocessinstanceruneventaspect_v1 iT2WND21QaWcaBm6DDOA5Q 1 1 6076340     0      1gb      1gb
yellow open dataprocessinstanceindex_v2                              MuN9IKTLQgS-YvD2YiheoQ 1 1 3037068 18854    2.5gb    2.5gb
yellow open dataprocessinstanceindex_v2_1680916549224                6jd2Ykf5RzG3tjSGijOyAA 1 1  176529     0  296.5mb  296.5mb
yellow open dataprocessinstanceindex_v2_clone_1680916215556          ZAKuf8iwT7m8t8lysazAyQ 1 1 3036957 19447    2.5gb    2.5gb
yellow open dataset_datasetprofileaspect_v1                          pzQ-y-U0TO-ZR70ZNIJ6Pg 1 1       0     0     208b     208b
yellow open dataset_datasetusagestatisticsaspect_v1                  UWg2cP78T6uVPNr6ThRloQ 1 1  124827  7465  136.8mb  136.8mb
yellow open dataset_operationaspect_v1                               S5Dw0GKlTbGECeGcKqvzlQ 1 1 6758731  1416 1018.3mb 1018.3mb
yellow open datasetindex_v2_1680916508969                            S-F_HiBgTXaqQAi8mfK67Q 1 1    6560     0   22.4mb   22.4mb
yellow open datasetindex_v2_clone_1680916215296                      nrzWuXqjSoeN0HCMZ5RFgw 1 1    6560   100   17.1mb   17.1mb
yellow open domainindex_v2_1680916487907                             4izOUkg8TeiGOvp7ggb8Gg 1 1      20     0   30.8kb   30.8kb
yellow open domainindex_v2_clone_1680916214148                       Fj5-lqmZRn62u4y879PAwQ 1 1      20     8   46.3kb   46.3kb
yellow open globalsettingsindex_v2_1680916319428                     7svGEVJmT6-n5XvGP-UfOw 1 1       0     0     208b     208b
yellow open globalsettingsindex_v2_clone_1680916207901               -SFPy2AGRMyTBIFY7DI1wQ 1 1       0     0     208b     208b
yellow open glossarynodeindex_v2_1680916467488                       O-4-C2RHRCCgz9EBYiIaDA 1 1       1     0   38.4kb   38.4kb
yellow open glossarynodeindex_v2_clone_1680916213672                 zx8Yb_ZJSQGzPpW3uzwKLw 1 1       1     0   32.6kb   32.6kb
yellow open glossarytermindex_v2_1680916362427                       D2bBNT6FT9-2IrSNssfw0w 1 1       1     0   40.4kb   40.4kb
yellow open glossarytermindex_v2_clone_1680916210607                 VrgvdzxvTtmktHCTGq1_6A 1 1       1     0   34.7kb   34.7kb
yellow open graph_service_v1                                         W_Bcz6tuTwW6nH2NWSZ1uQ 1 1 3079242 15035  604.1mb  604.1mb
yellow open system_metadata_service_v1                               -ljiiLMTSeidyP0yqTZ7Bw 1 1 9408521 73010    1.1gb    1.1gb
yellow open tagindex_v2_1680916341913                                raqGV84UQAqJ0P3GdkLYtg 1 1       5     0   25.7kb   25.7kb
yellow open tagindex_v2_clone_1680916210179                          LdLr2zOWQzy8LnlClzlqtw 1 1       5     0   12.3kb   12.3kb
yellow open telemetryindex_v2_1680916508365                          6Nh947jPRLW4RqQcuk_JlA 1 1       0     0     208b     208b
yellow open telemetryindex_v2_clone_1680916214654                    yjhPnNPNQuOFjUUBz9jbSA 1 1       0     0     208b     208b
yellow open testindex_v2_1680916445849                               BP18OC6lTZezTEjNPQm68Q 1 1       0     0     208b     208b
yellow open testindex_v2_clone_1680916212733                         UY6leBs9QDur6ENpOwg1Rg 1 1       0     0     208b     208b
📖 1
🔍 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
a
How many entities do you have? It can take quite a long time
w
I too have many cloned indices/aliases. Should I clean them manually ?
a
That would probably be best, a reindex may help too!
i
We have about 7k entities. I was able to complete reindex after restoring es indexes from snapshot and stopping gms and consumers.
b
hello! @important-night-50346 I have encountered the same problem as you right now. I would like to know more about how you solved it. It may have been some time ago, but I would appreciate any help. For example, I would like to know what “restore es from snapshot” means and how you stopped GMS. I am a beginner and know little about the operation of ES.