Hello! I'm having problems with ingestion on K8S deployment.

'[2022-06-20 16:55:51,002] INFO     {datahub.cli.ingest_cli:99} - DataHub CLI version: 0.8.38\n'
           '[2022-06-20 16:55:54,017] INFO     {datahub.cli.ingest_cli:115} - Starting metadata ingestion\n'
           '[2022-06-20 16:55:54,017] INFO     {datahub.ingestion.source.sql.bigquery:367} - Populating lineage info via GCP audit logs\n'
           "[2022-06-20 16:55:54,021] ERROR    {datahub.ingestion.source.sql.bigquery:505} - lineage-gcp-logs => Error was ('Failed to load service "
           "account credentials from /tmp/tmp01jy7wi8', ValueError('Could not deserialize key data. The data may be in an incorrect format, it may "
           "be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', "
           "[_OpenSSLErrorWithText(code=503841036, lib=60, reason=524556, reason_text=b'error:1E08010C:DECODER routines::unsupported')]))\n"

This is related to the creation of the encryption key?

Hello Patrick, Is your BigQuery instance using SSL? It looks like the DataHub recipe is not configured correctly to communicate with your BigQuery instance? SSL configs not passed or maybe incorrectly?

I'm using a service account that has the permissions needed by the ingestion. I was using the same service account in a recipe created on the quick-start (docker-compose), but this recipe dont work on the K8S deployment. Note: The ingestion secrets were created via UI.

Could you share the docker-compose recipe and the k8s one?

Yes! The K8S and the docker-comose file are the same:

source:
    type: bigquery
    config:
        project_id: bi-data-science
        credential:
            project_id: '${DATAHUB__BIGQUERY_PROJECT_ID}'
            private_key_id: '${DATAHUB__BIGQUERY_PRIVATE_KEY_ID}'
            private_key: '${DATAHUB__BIGQUERY_PRIVATE_KEY}'
            client_email: '${DATAHUB__BIGQUERY_CLIENT_EMAIL}'
            client_id: '${DATAHUB__BIGQUERY_CLIENT_ID}'
        include_tables: true
        include_views: true
        include_table_lineage: true
        use_exported_bigquery_audit_metadata: false
        start_time: '2022-05-28T00:00:00Z'
        end_time: '2050-01-01T00:00:00Z'
        schema_pattern:
            allow:
                - firebase
        profiling:
            enabled: true
            bigquery_temp_table_schema: bi-data-science.dsc_temporary_tables
            turn_off_expensive_profiling_metrics: true
            query_combiner_enabled: true
            max_number_of_fields_to_profile: 10
            max_workers: 10
            profile_table_level_only: false
            include_field_null_count: true
            include_field_min_value: true
            include_field_max_value: true
            include_field_mean_value: true
            include_field_median_value: true
            include_field_stddev_value: false
            include_field_quantiles: true
            include_field_distinct_value_frequencies: false
            include_field_histogram: false
            include_field_sample_values: false
sink:
    type: datahub-rest
    config:
        server: host
        max_threads: 4

Can you let us know how you created the secrets? Also, can you check and let us know if the image tag is same in both cases for actions container?

I decided to use Airflow as my ingestion backend because I can have more control over executions and apply version control for recipes. Even though, thanks a lot for your time.