Dears, i`m analyzing the dockerfile of elasticsearch setup and there is a script which is responsible of indieces creation.

<https://github.com/datahub-project/datahub/blob/master/docker/elasticsearch-setup/create-indices.sh>

There is a function which is checking if there is a defined INDEX_PREFIX of elasticsearch indieces. How to define this prefix ? I was trying to define this with dockerfile env as: INDEX_PREFIX=mytestprefix, but nothing is working or even was trying using ELASTIC_INDEX_PREFIX which is defined under application.yml for datahub-frontend-react which might be correlated with that settings. But still my indices are following

yellow open datajobindex_v2                                          ldJo3uE4QJewBhNzr9XdXA 1 1 0 0   208b   208b
yellow open datahubsecretindex_v2                                    haJzqxCHSYu58IMnaSduSQ 1 1 0 0   208b   208b
yellow open dataset_datasetprofileaspect_v1                          Y2qj3-drTWqutplPcLvvhQ 1 1 0 0   208b   208b
yellow open datahubexecutionrequestindex_v2                          mrgBox2jRtKQeMPtHLaT5w 1 1 0 0   208b   208b
yellow open dataflowindex_v2                                         0_D7Q1iGRW-z2bFgHC0fZg 1 1 0 0   208b   208b
yellow open mlmodelgroupindex_v2                                     Vnah4hvCQhK61JdkwlqnPw 1 1 0 0   208b   208b
yellow open mlmodelindex_v2                                          PZKoGxrsTvmGk25VNNjHLQ 1 1 0 0   208b   208b
yellow open datahubpolicyindex_v2                                    brZRjdiNTryRDNRBUSsKyQ 1 1 5 0 10.3kb 10.3kb
yellow open assertionindex_v2                                        QozAdKuxS364gVmIek_pqw 1 1 0 0   208b   208b
yellow open corpuserindex_v2                                         XPPJPMjyR3OJIV4GUSy7sg 1 1 0 0   208b   208b
yellow open dataprocessindex_v2                                      CPSGwCMvSmiJYsnvUb8s3w 1 1 0 0   208b   208b
yellow open chartindex_v2                                            ra5VG-JfSMKwcN2-VpEQmQ 1 1 0 0   208b   208b
yellow open tagindex_v2                                              AXOqaMg8SJWXXnyhD2p_KQ 1 1 0 0   208b   208b
yellow open mlmodeldeploymentindex_v2                                sJt2p1YvSQWW1ewRKoKuEA 1 1 0 0   208b   208b
yellow open datajob_datahubingestioncheckpointaspect_v1              LTmBTaAJTbmd3tktC66wiQ 1 1 0 0   208b   208b
yellow open dataplatforminstanceindex_v2                             SfvFJfC-SZOnDYBL44x4Eg 1 1 0 0   208b   208b
yellow open assertion_assertionruneventaspect_v1                     js_KYeqPSEix2g5EsRHaMg 1 1 0 0   208b   208b
yellow open dashboardindex_v2                                        J4sC64YlRVmSmDyJ6G4T2A 1 1 0 0   208b   208b
yellow open telemetryindex_v2                                        EFpZSHf_S6Whj-8bVvOswQ 1 1 0 0   208b   208b
yellow open datasetindex_v2                                          FT8o_fmdQvCcqhTxJlYVAg 1 1 0 0   208b   208b
yellow open mlfeatureindex_v2                                        vCLwtkvvQ9ugeec6pVjR4Q 1 1 0 0   208b   208b
yellow open dashboard_dashboardusagestatisticsaspect_v1              sygR_ZRDSJG5nHZvAwC5_g 1 1 0 0   208b   208b
yellow open dataplatformindex_v2                                     QhyOxlslRdKzMLpP5evHTg 1 1 0 0   208b   208b
yellow open datajob_datahubingestionrunsummaryaspect_v1              4ky9JVTvT5WleFMBtbZRhw 1 1 0 0   208b   208b
yellow open dataprocessinstanceindex_v2                              enwXRh_zQaW8tiYUwNCB3Q 1 1 0 0   208b   208b
yellow open glossarynodeindex_v2                                     HmHkL_o-RMGQsALQxz6vgQ 1 1 0 0   208b   208b
yellow open datahubingestionsourceindex_v2                           5WShz3TgTtyM5E5kslJ_tA 1 1 0 0   208b   208b
yellow open invitetokenindex_v2                                      ZJ75LBP8QY-UpHDpZXTZ_g 1 1 0 0   208b   208b
yellow open datahubretentionindex_v2                                 nu642knUTYqQ1L6pM9lYIQ 1 1 0 0   208b   208b
yellow open graph_service_v1                                         fVLYqPbOTOmLSJ5-zDkmqg 1 1 0 0   208b   208b
yellow open dataprocessinstance_dataprocessinstanceruneventaspect_v1 loO3jJUDTB6Bw_wxU1DI6w 1 1 0 0   208b   208b
yellow open dataset_operationaspect_v1                               9nwFViX-SI6WWd8SKLjPuQ 1 1 0 0   208b   208b
yellow open system_metadata_service_v1                               21XSV1HfSxagi0Irbew1uA 1 1 0 0   208b   208b
yellow open datahubaccesstokenindex_v2                               JLquhnmlQgea7q0hUsjkpA 1 1 0 0   208b   208b
yellow open containerindex_v2                                        Zzeucw97Q8Cl9VnVMdcoXQ 1 1 0 0   208b   208b
yellow open schemafieldindex_v2                                      TLPPap0ORYS1ej-dOiVxjA 1 1 0 0   208b   208b
yellow open domainindex_v2                                           EXmwmFX5Qg6VMssb-oTiXQ 1 1 0 0   208b   208b
yellow open testindex_v2                                             gkkE8iP_T2GTHmYtS7Qgeg 1 1 0 0   208b   208b
yellow open mlfeaturetableindex_v2                                   -WW5qco8R560Z7rADnYwLA 1 1 0 0   208b   208b
yellow open notebookindex_v2                                         AHsVbrz7RhuQ51nVV9Qtfw 1 1 0 0   208b   208b
yellow open datahubupgradeindex_v2                                   6eXzTZLCR_KkBHfveZbJgw 1 1 0 0   208b   208b
yellow open glossarytermindex_v2                                     nGMaf6RQS5KXkhtQ-ghABQ 1 1 0 0   208b   208b
yellow open mlprimarykeyindex_v2                                     g4_WJniiR2SOPuVPRWFyEQ 1 1 0 0   208b   208b
yellow open corpgroupindex_v2                                        vfHg9BD0QyWiUl_2HIbq6A 1 1 0 0   208b   208b
yellow open dataset_datasetusagestatisticsaspect_v1                  jBuQor2tRRe2CGsWC2GCFg 1 1 0 0   208b   208b

As i would like to use external elasticsearch and would like to have possibility to much more control on the name of the indieces, just in case i would need to do the cleanup. It would be good to understood where to define this prefix. Of course if it is possible at all.

have you tried adding environment variable here - https://github.com/datahub-project/datahub/blob/master/docker/elasticsearch-setup/env/docker.env

@bumpy-needle-3184 in the docker-compose.yml where i have a part for elasticsearch i were doing it as follow:

elasticsearch-setup:
  container_name: elasticsearch-setup
  depends_on:
  - elasticsearch
  environment:
  - ELASTICSEARCH_HOST=elasticsearch
  - ELASTICSEARCH_PORT=9200
  - ELASTICSEARCH_PROTOCOL=http
  - INDEX_PREFIX=test1

Teoretically i see in the log that INDEX_PREFIX has been applied, but only to the naming of policy and to the lifecycle_names. But it is not applied to the name of every index. And based on the other file which is under datahub/metadata-ingestion/tests/unit It is not even implemented because test_elasticsearch_source.py containing static name of indexes

creating datahub_usage_event_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "7d"
}
}
},
"delete": {
"min_age": "60d",
"actions": {
"delete": {}
}
}
}
}
}{"acknowledged":true}
creating datahub_usage_event_index_template
{
"index_patterns": ["*test1_datahub_usage_event*"],
"data_stream": { },
"priority": 500,
"template": {
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"type": {
"type": "keyword"
},
"timestamp": {
"type": "date"
},
"userAgent": {
"type": "keyword"
},
"browserId": {
"type": "keyword"
}
}
},
"settings": {
"index.lifecycle.name": "test1_datahub_usage_event_policy"
}
}
}{"acknowledged":true}
datahub_usage_event_policy exists

datahub_usage_event_index_template exists

datahub_usage_event_policy exists

datahub_usage_event_index_template exists

datahub_usage_event_policy exists

datahub_usage_event_index_template exists

And after that output of localhost:9200/_cat/indieces

yellow open datahubsecretindex_v2                                    GeHeEQafRZi_EePD4iWpVg 1 1 0 0 208b 208b
yellow open dataset_datasetprofileaspect_v1                          ln3lMcGCR_CRoJF6Iww18Q 1 1 0 0 208b 208b
yellow open datahubexecutionrequestindex_v2                          vI1Qcbg3QFCZ8bSJnNwHRQ 1 1 0 0 208b 208b
yellow open datajobindex_v2                                          HKte2taLRMaxJxaSm5xsQQ 1 1 0 0 208b 208b
yellow open mlmodelindex_v2                                          ZFgz9aCjQpedjNA-k5G42g 1 1 0 0 208b 208b
yellow open mlmodelgroupindex_v2                                     rlbE3JP8SNuvuVtSzTOLew 1 1 0 0 208b 208b
yellow open dataflowindex_v2                                         IUiFeDYFS9iqyyvLw3c7hw 1 1 0 0 208b 208b
yellow open assertionindex_v2                                        ed19GizFRrO0slh77chkNQ 1 1 0 0 208b 208b
yellow open datahubpolicyindex_v2                                    osRylwQuS0mO9EqMnh26Bg 1 1 0 0 208b 208b
yellow open corpuserindex_v2                                         uu0yY78iS9i-ljDuDUVqlQ 1 1 0 0 208b 208b
yellow open dataprocessindex_v2                                      uZupHdzST8ysPZrapOm2BA 1 1 0 0 208b 208b
yellow open chartindex_v2                                            gBraayAsR0Gr1K7f1a21cA 1 1 0 0 208b 208b
yellow open tagindex_v2                                              SUuZp3jZToaT_Fw9fSu9rQ 1 1 0 0 208b 208b
yellow open mlmodeldeploymentindex_v2                                d5ksREn9QtOrjLbUJwENmQ 1 1 0 0 208b 208b
yellow open datajob_datahubingestioncheckpointaspect_v1              u83kEqvbRMe45hE5SQZQsA 1 1 0 0 208b 208b
yellow open dataplatforminstanceindex_v2                             jjbxGIhUT9S4qOD__tlyfg 1 1 0 0 208b 208b
yellow open assertion_assertionruneventaspect_v1                     VD7vsP--RlaqGM0A9Ecqhg 1 1 0 0 208b 208b
yellow open dashboardindex_v2                                        i0Xbgc54QzCs7gCtns6fVg 1 1 0 0 208b 208b
yellow open datasetindex_v2                                          5axezrtyRBuOid23PwUwDw 1 1 0 0 208b 208b
yellow open telemetryindex_v2                                        nLNly8p6Ssak3MJ9nIBSfg 1 1 0 0 208b 208b
yellow open mlfeatureindex_v2                                        jw6ywz8-SCOQAnnNT5waBA 1 1 0 0 208b 208b
yellow open datajob_datahubingestionrunsummaryaspect_v1              Qty3J8TGQWmTRi8MXGKqzg 1 1 0 0 208b 208b
yellow open dashboard_dashboardusagestatisticsaspect_v1              Qnh-sGyLSmeQrt-qGiB04w 1 1 0 0 208b 208b
yellow open dataplatformindex_v2                                     SZeXrWfZQEu9_3BYcf8YBg 1 1 0 0 208b 208b
yellow open dataprocessinstanceindex_v2                              p30xGMVqTn687HKDfUZ3Qw 1 1 0 0 208b 208b
yellow open glossarynodeindex_v2                                     CoLlyfoCQxCXmiGzIgfn6Q 1 1 0 0 208b 208b
yellow open datahubingestionsourceindex_v2                           vvMZHWDmSdW7p1CHAY-KeA 1 1 0 0 208b 208b
yellow open invitetokenindex_v2                                      u_D4AR8yTQODOamXBvUe8g 1 1 0 0 208b 208b
yellow open datahubretentionindex_v2                                 0nsJ8W8ySLCE_HgSsWWQhw 1 1 0 0 208b 208b
yellow open graph_service_v1                                         333pcpaSQaCks0kwayIoIQ 1 1 0 0 208b 208b
yellow open dataprocessinstance_dataprocessinstanceruneventaspect_v1 le6YhayoTqa2RzOHBxLtfw 1 1 0 0 208b 208b
yellow open system_metadata_service_v1                               LmosQUSISnax6iUGWOeI1w 1 1 0 0 208b 208b
yellow open dataset_operationaspect_v1                               vwUxinrRRjm1sRhoagRWAg 1 1 0 0 208b 208b
yellow open datahubaccesstokenindex_v2                               JDOz1CarRRqBpZ33dYXo3A 1 1 0 0 208b 208b
yellow open containerindex_v2                                        GhNLDk2dT3KQFV4YjsV8FQ 1 1 0 0 208b 208b
yellow open schemafieldindex_v2                                      xlDLSrQbSuCHYjp9EVNzcQ 1 1 0 0 208b 208b
yellow open domainindex_v2                                           htySCyhLRuKN2YLMiPDJWA 1 1 0 0 208b 208b
yellow open testindex_v2                                             N5Xbn4o2RiSFshRVbnXcrw 1 1 0 0 208b 208b
yellow open mlfeaturetableindex_v2                                   ZF6tLJOQSsqWXx4qApHWDg 1 1 0 0 208b 208b
yellow open notebookindex_v2                                         PvbH3NlFQKCpBwCsjQAZNQ 1 1 0 0 208b 208b
yellow open datahubupgradeindex_v2                                   3WuXou_OQ6Sl95oKYfCnGA 1 1 0 0 208b 208b
yellow open glossarytermindex_v2                                     ZJqKf3vGSEmAZq3FMypJ4g 1 1 0 0 208b 208b
yellow open mlprimarykeyindex_v2                                     bbjk2oErQ9SomWmNlR0UOQ 1 1 0 0 208b 208b
yellow open corpgroupindex_v2                                        pJeD8bA4Taqb2kHlohwBGw 1 1 0 0 208b 208b
yellow open dataset_datasetusagestatisticsaspect_v1                  4_GlitVKQreF3-DhDzJ6Sw 1 1 0 0 208b 208b

So would be good to request this as a feature to name every index for example as datahub_* or allow to define this variable globally.

create-indices.script.sh is used for inserting indices for datahub_usage_event so the prefix may not work with other index

Have you set this variable on gms also?

INDEX_PREFIX

? You can see it in application.yml file of gms

@square-activity-64562 sure were trying as well on both places.