Hi, I'm trying to add Elasticsearch as an ingestio...
# getting-startedb
broad-summer-18536
10/20/2023, 2:36 AMHi, I'm trying to add Elasticsearch as an ingestion source, but it seems can only fetch indexes but can't see any index in a datastream. Is there any chance would support Elaticsearch datastream? thx~
r
rich-barista-93413
10/24/2023, 7:35 AM@gray-shoe-75895 Any idea on this?
👍 1
b
broad-summer-18536
10/25/2023, 2:01 AMThanks @delightful-ram-75848 ~
I've allow all index-pattern like ["^.*"], but it still can only see those indexes which doesn't belongs to any Elasticsearch datastream.
g
gray-shoe-75895
10/25/2023, 6:13 AMI'm not 100% sure. Our code definitely has handling for datastreams (https://github.com/datahub-project/datahub/blob/c38bb91519e0005f6e8a0a2d648bd1c77f[…]tadata-ingestion/src/datahub/ingestion/source/elastic_search.py). In the ingestion report, does it show anything as "dropped"?
b
broad-summer-18536
10/25/2023, 10:15 AM@gray-shoe-75895 thanks for your response!
I've set the pattern
index_pattern:allow: ["^\\.*"]broad-summer-18536
10/25/2023, 10:18 AMMy datastream named
logstashindex_pattern:allow: ["logstash"]broad-summer-18536
10/25/2023, 10:19 AMIt still not show any index when I run the ingest commnad
datahub ingest -c elasticsearch.dhub.yamlbroad-summer-18536
10/25/2023, 10:21 AMMy elasticsearch version is 7.17.0.
a
aloof-gpu-11378
10/27/2023, 4:45 AMA few hypotheses for what's happening here
1. a permissions issue is preventing you from seeing those indexes
2. for some reason (possibly pagination?), the call to indices.get_alias() is simply not returning the target index https://github.com/datahub-project/datahub/blob/c38bb91519e0005f6e8a0a2d648bd1c77f[…]tadata-ingestion/src/datahub/ingestion/source/elastic_search.py
3. there's a bug in our logic that silently drops that index
Could you run with
datahub --debug ingest ...👍 1
b
broad-summer-18536
10/27/2023, 6:30 AMHi @gray-shoe-75895
We doesn't set any permission on this ELK stack since this is used for internal.
After I ran tthe ingest command, I still can only saw other index excluding those indexes in datastream in the debug log.
broad-summer-18536
10/27/2023, 6:39 AMMy index config is just like following, is there anything else I forget to config?
# Options
url_prefix: "" # optional url_prefix
env: "PROD"
index_pattern:
allow: ["^.*"]
broad-summer-18536
10/27/2023, 7:10 AMAfter trying using python shell for testing,
it seems the elasticsearch python lib can't get the index in data stream....
root@datahub-test-by-walter:~/.datahub# python
Python 3.10.12 (main, Jun 11 2023, 052628) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from elasticsearch import Elasticsearch
>>> client = Elasticsearch("XXXXXXXX", use_ssl=False)
>>> client.indices.get_alias("*")
broad-summer-18536
10/27/2023, 7:46 AMOK, seems that indices in my data stream don't have alias, so it's not the Datahub's problem
After tested, I found that data stream backing indices don't support aliases, so I tried to set the alias of my data stream named
logstashlogsallow: ["logs"]logstash.ds-logstash-2023.10.27-000014.ds-logstash-2023.10.27-000015raw_index_metadata = raw_index[index]allow: ["logs"]raw_index_metadata = raw_index["logstash"]r
rich-barista-93413
11/07/2023, 6:41 PMHonestly I think I'm a bit out of my depth given my understanding ElasticSearch - if you can figure out a tweak to DataHub's usage of the ElasticSearch sdks to make it work, we'd definitely want to collaborate around that
👍 1
b
broad-summer-18536
11/09/2023, 1:43 AMIt's my pleasure! I would take time to figure out how to improve the skd.
👍 1