Hello, I'm getting the following error when trying...
# ingestionb
broad-thailand-41358
03/08/2022, 5:29 PMHello, I'm getting the following error when trying to ingest elasticsearch:
hosthost contains bad characters, found <http://master1.search.dfw.wordpress.com> (type=assertion_error) Copy code
source:
type: "elasticsearch"
config:
# Coordinates
host: <http://master1.search.dfw.wordpress.com:9203>s
square-activity-64562
03/08/2022, 5:54 PMTry running without the
http://b
broad-thailand-41358
03/08/2022, 6:10 PMhey @square-activity-64562 when I try that, it gives me this error:
ValueError: URL must include a 'scheme', 'host', and 'port' component (ie '<https://localhost:9200>')s
square-activity-64562
03/08/2022, 6:12 PMWill ask someone else to help
thank you 1
g
green-football-43791
03/08/2022, 7:35 PMHey @broad-thailand-41358 - i tried parsing your config locally on my machine and it was successful- no validation errors.
green-football-43791
03/08/2022, 7:35 PMI also noticed some changes to the validation logic happened recently:
green-football-43791
03/08/2022, 7:35 PMgreen-football-43791
03/08/2022, 7:35 PMWhat version are you on?
green-football-43791
03/08/2022, 7:35 PMcan you try updating to the latest cli version?
green-football-43791
03/09/2022, 3:58 PMHmm… I’m not very familiar with your setup. What variables would you need to pass? This may just require a small contribution to the elasticsearch source.
b
broad-thailand-41358
03/09/2022, 5:09 PMmy company runs a proxy on port 8080 for authentication, I had to edit the environment variable SOCKS_PROXY=127.0.0.1:8080 for the trino plugin to work, I was wondering if there was something similar I had to do with elasticsearch
g
green-football-43791
03/09/2022, 9:01 PMI am not sure, I think we would have to look into the elasticsearch python client specifically
green-football-43791
03/09/2022, 9:02 PMhmm looks like it is possible: https://github.com/elastic/elasticsearch-py/issues/275#issuecomment-143781969
green-football-43791
03/09/2022, 9:03 PMwould you be open to adding a contribution to support this in https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/elastic_search.py?
b
broad-thailand-41358
03/09/2022, 10:00 PMI would love to @green-football-43791, but I honestly doubt my Python skills are up to par 😄
Let me take a look and see what I can do. I'm demoing DataHub to my company tomorrow and will take a look after that
g
green-football-43791
03/09/2022, 10:42 PMnp! I’d be happy to pair on it if you want 🙂
thank you 1
green-football-43791
03/09/2022, 10:43 PMit would be helpful to get some together to try running things against your instance so we can see if it works
green-football-43791
03/09/2022, 10:43 PMgood luck on your demo tomorrow!! do you think have enough to go off w/o the elastic proxy set up?
green-football-43791
03/09/2022, 10:43 PMif you want, we can try to get something together this afternoon. I’m free at 4pm PST if you want to pair then
green-football-43791
03/09/2022, 10:44 PMshould just take an hour or so
b
broad-thailand-41358
03/10/2022, 5:21 PMhey @green-football-43791 I should be free at that time!
broad-thailand-41358
03/10/2022, 5:22 PMshoot me an invite slam@automattic.com
g
green-football-43791
03/10/2022, 5:56 PMsure!
green-football-43791
03/10/2022, 5:57 PMjust sent for this afternoon 🙃
3 Views