https://datahubproject.io logo
Join Slack
Powered by
Hi! So `create-indices.sh` on `ealsticsearch-setup...
# all-things-deployment
s
Hi! So 
create-indices.sh
on 
ealsticsearch-setup
job creates a number of indexes starting with underscore 
_
Copy code
#   1. ILM policy
  create_if_not_exists "_ilm/policy/${PREFIX}datahub_usage_event_policy" policy.json
  #   2. index template
  create_if_not_exists "_index_template/${PREFIX}datahub_usage_event_index_template" index_template.json
  #   3. although indexing request creates the data stream, it's not queryable before creation, causing GMS to throw exceptions
  create_if_not_exists "_data_stream/${PREFIX}datahub_usage_event" "datahub_usage_event"
turns out names are forbidden to start with 
_
on my ES instance (Aiven). If I change the script to use another name… what other components would I need to change? or would I just be better off starting my own ES on GKE?
🔍 1
📖 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
a
We don’t work with Aiven too often- @brainy-tent-14503 do you know if there’s a way to forbid certain index prefixes?
s
I extended the 
datahub-elasticsearch-setup
docker image with my own 
create-indices.sh
to rename the indexes (no 
_
) and now i’m trying to run datahub in local dev mode to see whetehr I need to change the code that uses the indexes, I’m a bit lost on that, seems ES indexes are called using mustache template? i.e. there is no property specifying the name of the indexes, am I right?
Issue I’m having now when running 
./gradlew quickstartDebug
is 
datahub-upgrade
fails, apaprently, because of ES
Copy code
ERROR c.l.d.u.s.e.steps.BuildIndicesStep:39 - BuildIndicesStep failed.
2023-05-17 11:03:10 java.lang.RuntimeException: [containerindex_v2_1684314190072] ElasticsearchStatusException[Elasticsearch exception [type=index_not_found_exception, reason=no such index [containerindex_v2_1684314190072]]]
notice this is the a fork from the current master (I haven’t modified anything)
and ES seems to fail because
Copy code
2023-05-17 10:56:28 {"type": "server", "timestamp": "2023-05-17T08:56:28,176Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "high disk watermark [90%] exceeded on [9djfJpP5QiaVKgoYdNHxLw][elasticsearch][/usr/share/elasticsearch/data/nodes/0] free: 3.8gb[6.6%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete", "cluster.uuid": "Gg1ASe8_SSi8v8qPlQM00Q", "node.id": "9djfJpP5QiaVKgoYdNHxLw"  }
got a bunch of these, isn’t there a way to skip the upgrade?
b
Those underscore url paths are not our convention, but elasticsearch see 1, 2, 3. Those are not names but the elasticsearch’s API for configuring those kinds of resources. The actual application/datahub indices are configurable with an index prefix (4), however by default this is an empty string and no indices or aliases are being created starting with underscore. If using helm this can be configured at 
elasticsearch.indexPrefix
here and converted to the linked env here.
The error indicates that there is not enough disk space on your Elasticsearch instance and therefore Elasticsearch is denying setting up the indices.
If the indices are not created, then datahub will not work unfortunately.
s
Yah, but these logs , about not enough space, are from using 
./gradlew
quickstart` and 
quickstartDebug
so I'm a bit surprised I have to tune them
b
Quickstart is using a local elasticsearch instance running in docker. You need more local space then.
You can typically run cleanup commands to make more space, remove old images, etc.
Here is some docs on that process https://docs.docker.com/config/pruning/
s
Thanks for the tips, will try it out
Hi @brainy-tent-14503, I found the problem is Aiven is NOT using ElasticSearch but OpenSearch, so ILM becomes ISM, etc
I suppose some simple changes should fix this
a
Opensearch 1.x is support, but its referred to as AWS - 
USE_AWS_ELASTICSEARCH=true
- here
4 Views
Open in Slack
PreviousNext
Powered by