Hi team I hope you are all doing great I am working on datah DataHub #troubleshoot

Hi team, I hope you are all doing great! I am wor...

microscopic-application-63745

03/07/2023, 2:56 PM

Hi team, I hope you are all doing great! I am working on datahub 0.9.5 and I am trying to run an S3 Data Lake custom recipe and according to the documentation I can use the config property

verify_ssl

but whenever I add it I get the following error:

Copy code

[2023-03-07 15:10:23,049] ERROR    {logger:26} - Please set env variable SPARK_VERSION
[2023-03-07 15:10:23,543] ERROR    {datahub.ingestion.run.pipeline:127} - 1 validation error for DataLakeSourceConfig
verify_ssl
  extra fields not permitted (type=value_error.extra)

Please note that without the

verify_ssl

the recipe ingests just fine.

✅ 2

microscopic-application-63745

03/07/2023, 3:04 PM

The recipe is very similar to this one (I modified a few things for security concerns):

Copy code

transformers:
    -
        type: simple_add_dataset_tags
        config:
            tag_urns:
                - 'urn:li:tag:forest '
    -
        type: simple_add_dataset_terms
        config:
            term_urns:
                - 'urn:li:glossaryTerm:1_2_3'
sink:
    type: datahub-rest
    config:
        server: '<http://datahub-datahub-gms:8080>'
source:
    type: s3
    config:
        profiling:
            enabled: false
        use_s3_object_tags: false
        use_s3_bucket_tags: false
        path_specs:
            -
                include: 's3://<SOME_PATH>/*.*'
        env: PROD
        aws_config:
            aws_access_key_id: <SOME_KEY>
            aws_region: <SOME_REGION>
            aws_secret_access_key: <SOME_SECRET_KEY>
        verify_ssl: true

microscopic-application-63745

03/07/2023, 4:31 PM

nvm the solution was to change the CLI to 0.9.5

Open in Slack

Previous Next