Got a question, how are you all handling secrets? Because we can’t really commit these recipes to git with a password in full text 😄 So I patched the yaml config for now to fetch from ssm (we use AWS) whenever a path is prefixed with “ssm;//” . Not sure if anybody has a better way or idea? Seemed a bit cleaner than rewriting them on the fly. (As the ingest entrypoint expects a file and not a loaded config object.)

@calm-sunset-28996: great question... I was actually looking around to see what are the best practices for secrets in yaml etc .. and I didn't turn up much in my research.

Tl,dr: Currently using Shirshanka's first suggestion. In my case the databases I want crawled require kerberos authentication. To provide that authentication I use custom script that is executed prior to the crawler which populates certain environment variables. I'm using kubernetes so this login script is actually a secret resource (so the payload is encrypted) used in an init container. All sensitive information (like creds) in the custom script are provided by a key vault in my CI/CD pipeline that installs my helm charts and K8s resources.