Hi all. I am creating a destination that can write parquet files to Azure Data Lake Gen2 folders as those are way easier to process in Synapse. Also adding support for SAS keys as Storage Accoutn Keys are discouraged. However, I am stuck on the connector acceptance tests as it can’t find the destination_config.json during the startup of the connector.

2023-03-14T16:14:49.249+0100 INFO Preparing command: docker run --rm --init -i -w /data/job --log-driver none --name destination-azure-data-lake-storage-gen2-write-0-0-yqfcb --network host -v /tmp/airbyte_tests/test9476898619965199338:/data -v /tmp/airbyte_tests/output3489329161128183704:/local -e USE_STREAM_CAPABLE_STATE=false -e FIELD_SELECTION_WORKSPACES= -e STRICT_COMPARISON_NORMALIZATION_WORKSPACES= -e APPLY_FIELD_SELECTION=false -e WORKER_CONNECTOR_IMAGE=airbyte/destination-azure-data-lake-storage-gen2:dev -e WORKER_JOB_ATTEMPT=0 -e AUTO_DETECT_SCHEMA=true -e WORKER_JOB_ID=0 -e STRICT_COMPARISON_NORMALIZATION_TAG=strict_comparison airbyte/destination-azure-data-lake-storage-gen2:dev write --config destination_config.json --catalog destination_catalog.json
2023-03-14T16:14:51.295+0100 INFO 2023-03-14T15:14:50.847+0000 INFO integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2023-03-14T16:14:51.300+0100 INFO 2023-03-14T15:14:50.856+0000 INFO Running integration: io.airbyte.integrations.destination.azure_data_lake_storage_gen2.AzureDataLakeStorageGen2Destination
2023-03-14T16:14:51.300+0100 INFO 2023-03-14T15:14:50.857+0000 INFO Command: WRITE
2023-03-14T16:14:51.308+0100 INFO 2023-03-14T15:14:50.857+0000 INFO Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2023-03-14T16:14:51.335+0100 INFO 2023-03-14T15:14:50.878+0000 ERROR Something went wrong in the connector. See the logs for more details

Is there a trick to use the secrets/conf.json for running the connector acceptance tests during local developmenmt?

Are you planning on pushing this back to Airbyte? Was planning on developing the same thing but just adding a Parquet Option to existing Azure Blob (which supports Gen 2 just fine)

Of course I will create a PR. I also had a working version for Azure Blob, but due to changing of configuraiton properties and backwards compatability, it’s easier to create a new one and add new features like base paths and additional authentication methods. But it would be nice to figure out how to get authentication tests done correctly

Got it. Makes sense. We wanted to add variable paths as well (mostly to support the timestamp container structure that's supposed to work already but doesn't).

anyone any tips to get the acceptance tests running?