hello! I’m trying to ingest validation with Great Expectation within an Airflow pipeline, but I get a strange error:

[2022-03-17, 13:35:39 CET] {local_task_job.py:154} INFO - Task exited with return code Negsignal.SIGKILL

Note that the part GE - Airflow works good, i.e. if I deactivate the action of sending stuff to DataHub everything works fine. I also set the logging level to debug, but nothing interesting is printed out. Any hint?

Usually airflow tasks getting killed means memory issue if running in Kubernetes. Not sure if that is the case for you

no kubernates, but running within containers. Do you think it can apply too?

That error is shown by Airflow that it is being killed. Usually K8s kills when memory issues happen not CPU. Not sure what you are using. But could be a possibility which should be easy to cross-check.

@hundreds-photographer-13496 ^

Tried to increase memory from 10 to 25 GB, same issue

Is it possible in GE to enable debug logging? Or some of your monitoring can show how much memory is being used? Maybe I am wrong and this is not a memory issue which debug logging would show us. Is that container containing a lot of expectations? I am not sure how many expectations @big-carpet-38439 tested it out with. Do you have a single GE container running everything? If not, maybe try running with another which has smaller number of expectations.

Actually everything works fine until I activate the action of ‘sending expectations to DataHub’, within the checkpoint configuration file. This, together with the failed test of increasing memory from 10 to 25 GB I think is sufficient to exclude the memory issue. Unless there is some bug which make the thing eat enormous amounts of memory

Yes 10 GB to 25 GB should be large enough. Can we enable debug logging on this?

Already did, but nothing related has been shown

@big-carpet-38439 any ideas?

+ @dazzling-judge-80093

Do you have

graceful_exceptions

disabled? How the action is set up?

running the checkpoint from the command line, whithin the same Docker container which runs the Airflow DAG, works good. Tied also both true and false for AIRFLOW_VAR_GRACEFUL_EXCEPTIONS. Is there any way I can see more logs of what happens ?

So if you run outside of the container it is pushing results to DataHub?

actually if I run inside the container, but in a terminal instead that from the airflow task, it runs good. So it should be an issue of the action ‘publish to datahub’, but only when the checkpoint is run by the airflow operator

Oh wait - I think I know

All the other interactions with DataHub (lineage calls, other metadata ingestsion, etc.) works by setting the GMS to http://datahub-gms:8080 , so this should be too… or not?

Yes it should

Hey @stale-jewelry-2440, have you tried this ? does this work for you ? https://github.com/datahub-project/datahub/issues/4531#issuecomment-1083148595

Hej! I’ll try that, thank you!

Wow very strange