👋 HI team! I have a pretty vague question.. and I would like to make it as concise as possible... I have a Redshift ingestion coded as Airflow DAG, which runs a pipeline that looks like this

pipeline = Pipeline.create(
        {
            "source": {
                "type": source,
                "config": {
                    "username": f"{conn.login}",
                    "password": f"{conn.password}",
                    "database": f"{conn.schema}",
                    "host_port": f"{conn.host}:{conn.port}",
                    "database_alias": alias,
                    "env": "PROD",
                    "schema_pattern": {
                        "deny": deny_schemas
                    },
                },
            },
            "transformers": transformers,
            "sink": {
                "type": "datahub-rest",
                "config": {"server": f"{datahub.host}"},
            },
        }
    )
    pipeline.run()
    pipeline.raise_from_status()

The thing is the DAG ends with

{local_task_job.py:154} INFO - Task exited with return code Negsignal.SIGKILL

It's the only thing that is actually logged.... it seems it fails as soon as the process starts. At first, I thought it was a memory issue, we increase the pods memory, and now we are pretty far from the memory limit. It even fails when I ran dray_run. Locally it's working fine. Locally I'm using Airflow 2.2.2 while in production I'm using Airflow 2.2.1. I would really appreciate any suggestion...

Hi @nice-country-99675, Have you tried setting the logging level to logging.DEBUG? It might help in narrowing down the root-cause.

Thanks Ravindra! will do that. In the meantime I just changed my DAG to use a BashOperator and call datahub ingest from cli...

I also encountered this issue. It seems like datahub ingestion pipeline consumes all memory in very short time and be killed. If I run

airflow dags test {DAG_ID} {EXEC_DATE}

it succeeded. However, if I run DAG in standard way, then it is killed everytime. @helpful-optician-78938 @nice-country-99675

cc @dazzling-judge-80093, is this a known issue?

I didn't know about this

by the way, in 0.29.1 client version, I had a very similar issue to @nice-country-99675. All memory is taken and I get the Negsignal.SIGKILL failure. I appreciated very much his idea to switch to BashOperator as this did the trick away from using the Pipeline class.