Checking the Airbyte documentatation I have found a guide how to create a secret to store the State and Logging into an Azure Blob storage but the documentation says that only AWS and GCS supported. I would like to confirm if Azure Blob storage is supported for this purpose?

Yes, Azure Blob Storage is supported for storing State and Logging in Airbyte. The documentation provides instructions on how to set up Azure Blob Storage for this purpose. You need to add external storage details to your

values.yaml

file. This will disable the default internal Minio instance (

airbyte/minio

). Airbyte recommends using a single bucket across all three values. The configuration for Azure Blob Storage in the

values.yaml

file looks like this:

yaml global:     storage:       type: "Azure"       storageSecretName: airbyte-config-secrets # Name of your Kubernetes secret.       bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.         log: airbyte-bucket         state: airbyte-bucket         workloadOutput: airbyte-bucket       azure:         connectionStringSecretKey: azure-blob-store-connection-string

You also need to create a Kubernetes secret containing the Azure Blob Storage connection string. The secret should look like this:

yaml apiVersion: v1   kind: Secret   metadata:     name: airbyte-config-secrets   type: Opaque   stringData:     # Azure Secrets     azure-blob-store-connection-string: ## DefaultEndpointsProtocol=https;AccountName=azureintegration;AccountKey=wJalrXUtnFEMI/wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY/wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY==;EndpointSuffix=<http://core.windows.net|core.windows.net>

Please note that you need to replace the placeholders with your actual Azure Blob Storage details. For more information, please refer to the following documentation: - [State and Logging Storage](https://docs.airbyte.com/deploying-airbyte/integrations/storage) - [Azure Blob Storage](https://docs.airbyte.com/integrations/destinations/azure-blob-storage#getting-started)

@kapa.ai I am facing the following error after adding the azure details about the storage:

Error: INSTALLATION FAILED: YAML parse error on airbyte/charts/server/templates/deployment.yaml: error converting YAML to JSON: yaml: line 220: could not find expected ':'

The error message you're seeing indicates a problem with the YAML syntax in your configuration file. YAML is very sensitive to correct indentation and formatting. The error message is pointing to a specific line in your YAML file where it expected to find a colon (':') but didn't. Unfortunately, without seeing the actual YAML file, it's hard to provide a specific solution. However, you can start by checking the mentioned line in your YAML file (line 220 in

airbyte/charts/server/templates/deployment.yaml

) and making sure that it follows the correct YAML syntax. Here is an example of how to specify Azure storage details in a YAML file, as per the [Airbyte documentation](https://docs.airbyte.com/deploying-airbyte/integrations/storage#values):

yaml global:     storage:       type: "Azure"       storageSecretName: airbyte-config-secrets # Name of your Kubernetes secret.       bucket: # S3 bucket names that you've created. We recommend storing the following all in one bucket.         log: airbyte-bucket         state: airbyte-bucket         workloadOutput: airbyte-bucket       azure:         connectionStringSecretKey: azure-blob-store-connection-string

Please ensure that your YAML file follows the correct structure and indentation. If the problem persists, you might want to validate your YAML file using an online YAML validator to catch any syntax errors.

@kapa.ai I have copied and paste the same code and it is still producing the same error.

A message from kapa.ai

I'm sorry to hear that you're still experiencing the issue. YAML files can be quite sensitive to formatting and small errors can sometimes be hard to spot. Here are a few general tips to troubleshoot YAML files: 1. Indentation: YAML uses indentation (spaces, not tabs) to determine the hierarchy of the data structure. Make sure your indentation is consistent and correctly represents the structure you intend. 2. Colons: In YAML, colons separate keys from values (like

key: value

). Make sure each key-value pair has a colon. 3. Dashes: Dashes are used to create bullet lists. If you're using dashes, make sure they're formatted correctly. 4. Quotes: Sometimes, values need to be wrapped in quotes. If a value includes special characters, or looks like a data type (like

true

,

false

,

null

, numbers), you might need to wrap it in quotes. If you've checked all of these and are still having issues, you might find it helpful to use a YAML validator tool. These tools can often provide more specific feedback about what's wrong with your YAML file. If the problem persists, it might be helpful to reach out to the Airbyte community on their [discussion forum](https://discuss.airbyte.io/) or [GitHub](https://github.com/airbytehq/airbyte) for more specific assistance.

global:
  ...
  storage:
    type: azure
    secretName: ${secret_name}
    bucket:
      # -- Provide one azure blob storage bucket for all
      log: ${namespace}-core
      state: ${namespace}-core
      workloadOutput: ${namespace}-core
    azure:
      connectionStringSecretKey: azure-blob-store-connection-string # From airbyte-config-secrets
  ...

@Matheus Dantas This worked for me for azure blob storage