one more question my “postgresql task with profiling” run more than 84 hrs and not yet complete how to tuning ? my database info: 17 table 50-100 column per table 1-9m rows per table ingest config

{
    "source": {
        "type": "postgres",
        "config": {
            "username": login,
            "password": password,
            "database": "user_activity",
            "host_port": host,
            "schema_pattern": {
                "deny": ["information_schema"]
            }
        },
    },
    "sink": {
        "type": "datahub-kafka",
        "config": {
            "connection": {
                "bootstrap": "prerequisites-kafka.datahub:9092",
                "schema_registry_url": "<http://prerequisites-cp-schema-registry.datahub:8081>"
            }
        },
    },
}

Hi @millions-soccer-98440, you can limit the number of documents profiled by using the config variables here -> https://datahubproject.io/docs/metadata-ingestion/source_docs/sql_profiles#config-details

@helpful-optician-78938 is also looking into understanding where the bottlenecks are

Hi @witty-state-99511 “profiling.limit” in RDBMS meaning number of DB rows or not? If I need to profiling all document there are other ways to tuning or not?

yes it is about limiting the number of rows in the table. We will get back to you about tuning soon cc @helpful-optician-78938