Hi everyone, I'm trying to use airflow 2.4.3 with meltano's datahub plugin, but everytime I try to start up airflow I'm now getting an error that says:

Invalid version: '[<current_datetime>] {_plugin.py:349} INFO - Patching datahub policy 2.4.3'

I'm trying to use

apache-airflow==2.4.3

,

apache-airflow-providers-amazon==6.0.0

,

acryl-datahub-airflow-plugin==0.9.6.2

. I've configured the

gms_host

in the datahub utility, but I haven't set the connection on the airflow side because I would normally do this through the UI. Is there a way for me to set the configuration in my

meltano.yml

? I've tried setting

datahub_conn_id

under

config.core

but no luck. Any idea what policy this error is referring to?

hey Jam! i'm asking around to find the right person to answer your question, we'll get back to you before too long 🙂 appreciate your patience

Thank you so much, Chris!

okay good to know! also our airflow expert isn't around right now but said he will help you out first thing tomorrow morning if that's alright?

That works for me. Thank you!

@bulky-jackal-3422 hey, Unfortunately, I don’t know much about Meltano, but the config should be the Airflow config, where you can set those configuration properties. I don’t know how you can change it when you use Meltano? Do you have direct access to Airflow, or Meltano has an embedded Airflow? How is this working? What is the stack trace in the above screenshot? Is it fail there? As the stderr is empty there and the stdout shows a log message which is expected (it is the message about it trying to patch all the airflow dag/tasks to add success/failure hooks automatically)

Hi @dazzling-judge-80093! 1. Meltano passes the config I put in the

meltano.yml

to the

airflow.cfg

. 2. I'm invoking Airflow through Meltano. I can find out how to access Airflow directly. When I normally run Airflow via Meltano and I do

docker ps

, I don't see any containers running, so I'll have to find another way. 3. The stacktrace is via Meltano's debug log. I screenshotted that specific part, because it was the only place I could find the mention of

version

and since the stacktrace I posted initially mentions

Invalid version

it piqued my interest. Is there any interaction between

acryl-datahub-airflow-plugin

and the version of Airflow?

ahh, I think I know what is the problem

Okay, that's what I thought was going on too, but I haven't heard back from the Meltano folks to see why it's handled this way

If you can change the logger, then maybe you can silence the DataHub plugin or in the long term, we can as well change this logline to debug level if it helps or provides a way to silence it

I'll try changing the log level to see if I can move past this issue!

ahh, cool

Changing my log-level didn't seem to make an impact, though

ok, I think we can change as well this to debug to make sure it won’t break Meltano

Awesome, I really appreciate the help!