Hi everyone! We are developing a tool that downloads Airbyte's data (sources, connections and destinations mainly), to store it inside a Git repo for future use This tool allows then to use this information (in json format) to load and restore (or configure a new Airbyte instance) as it was at "extraction time". It's quite like the Export-Import stuff inside Settings in the UI We acknowledge certain fields are hidden behind ****** in the API response (we use Git Secrets to load them when needed), but some others (like S3) are exposing critical information, i.e

aws_secret_access_key

Is there anywhere in the docs, or Airbyte's repos, where we could find a list of the hidden (and non-hidden) fields for the different connectors? Finding them out by trial and error, taking into account the vast number of connectors available, is extremely difficult.

Hi @Bruno, each source and destinations specs contains the

airbyte_secret

boolean attribute that will help you identify these fields.

Are you interacting with our API to download Airbyte's configurations?

Yes, I do. I'm searching in the docs and can't find those specs you mention

Example for postgres source definition:

curl -X POST "<http://localhost:8000/api/v1/source_definition_specifications/get>" \
 -H "Accept: application/json" \
 -H "Content-Type: application/json" \
 -d '{"sourceDefinitionId":"decd338e-5647-4c0b-adf4-da0e75f5a750"}'

Sample of the response:

"password": {
        "type": "string",
        "order": 4,
        "title": "Password",
        "description": "Password associated with the username.",
        "airbyte_secret": true
      }

@[DEPRECATED] Augustin Lafanechere thank you for your response. Those fields/properties with

airbyte_secret: true

are only for UI 'hiding'? Doesn't the API hide the information too? In some sources/destinations (like Postgres) we get the '*******' values (as expected for those with airbyte_secret enabled), but others like S3 won't do so. I've read a bit about Octavia and it looks really promising.

Ok thanks for these details, I thought first that you wanted to find a way to identify a secret fields. Let me dig into this and I'll come back to you.

Great. So it's supposed that

airbyte_secret:true

fields should be

********

in the API response? I was about to create an issue in Github

HI @Bruno I think what you are pointing is related to this already known issue

We have a

maskSecrets

method that need to be improved in this file:

airbyte-config/persistence/src/main/java/io/airbyte/config/persistence/split_secrets/JsonSecretsProcessor.java

Could you tell me on which connector you have this exposed aws key?

Good morning @[DEPRECATED] Augustin Lafanechere! Thanks for the last message, I was a bit lost when reading

JsonSecretsProcessor

. That issue states the following: As of #6114 we will have a more complete template for parsing JsonSchema configs/objects which could likely be adapted. And that #6114 was merged into 0.29.22-alpha #6450

We are having these problems with S3 connectors. In its fields it's clear:

"aws_access_key_id": {
                        "title": "Aws Access Key Id",
                        "description": "In order to [...],
                        "airbyte_secret": true,
                        "type": "string"
                    }
 "aws_secret_access_key": {
                        "title": "Aws Secret Access Key",
                        "description": "In order to [...]",
                        "airbyte_secret": true,
                        "type": "string"
                    },

But when retrieving a S3 source from the API, those fields are plain-text 😞

I think https://github.com/airbytehq/airbyte/issues/6393 is the specific issue.

Oh I linked it in the original comment.