Hello while trying to injest with the datahub kafka sink I keep getting this error

datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (bigquery): Missing provider configuration.

This is what the recipe looks like

pipeline_name: analytics
source:
    type: bigquery
    config:
        env: DEV
        include_table_lineage: true
        include_usage_statistics: true
        include_tables: true
        include_views: true
        profiling:
            enabled: true
            profile_table_level_only: false
        stateful_ingestion:
            enabled: true
        credential:
            project_id: <redacted>
            private_key: <redacted>
            private_key_id: <redacted>
            client_email: <redacted>
            client_id: <redacted>
sink:
    type: datahub-kafka
    config:
        connection:
            bootstrap: 'datahub-prerequisites-kafka:9092'
            schema_registry_url: '<http://datahub-prerequisites-cp-schema-registry:8081>'

Are you trying to ingest from Kafka to Datahub ?

No I'm trying to ingest from bigquery, but sink to the DB kafka sink for a higher throughput because the rest endpoint is getting hammered pretty hard, and I don't want to throttle rest API requests

did you install the bigquery ingest plugin ? If not, you can run the below command.

pip install 'acryl-datahub[bigquery]'

That all works already... this error only comes up when I specify datahub-kafka as a sink... And I have also checked that I have the

'acryl-datahub[kafka]'

package installed

Hi, @lemon-scooter-69730 you should disable stateful_ingestion according to https://datahubspace.slack.com/archives/CUMUWQU66/p1667461540670879?thread_ts=1667460445.308079&cid=CUMUWQU66