Random question on Friday so I bet this is the right place. You need to build CICD pipeline to compile, test, dockerize, sign and push to storage from scratch, no questions asked. It has to be generic to support multiple coding langs and their ecosystems in a single monorepo. Approach is to solve it like a Software Engineer - you want to run CICID, compile, test and change locally, like any SWE would do, and then run it on GH Actions, GitLab or whatever, treating the SVC platform as a shim or mere adapter to your CICD at best. What would you do? Am I insane nothing like it exists?

Lol, you just described atmos 😃

Well 💩 I have a feeling that will be a hard sell within my teams.

It would be helpful for us to understand what makes atmos a hard sell? Is it documentation, learning curve, YAML, or something else?

To be honest, personal (not mine) taste and lack of technical depth. I've started poc-ing with atmos ci to see it in action.

The reason I ask is I have heard the push back before, but struggle to ever get the constructive criticism we can use to make it better. Our entire goal is provide a supreme, best-in-class DX, while also not limiting what power users can do with it.

Yeah I get that. I'm gonna build with it and then in case I got constructive criticisms I will share. To me not being able to develop and test CICD code locally drives me nuts and that will be my main angle to sharpen my knives.

Yes, would definitely accept the GCP support

Great, 'cause most likely I gonna need that quite soon.

We've just rounded out the Azure support with feature parity. GCP is mostly there, but not with the native CI pieces. The good news is is it's just an interface that gets extended, so it's very easy to add. Main reason for not supporting it is just testing.

Great, then I'm gonna start familiarise myself with contribution guides.

(actually, I think @Andriy Knysh (Cloud Posse) will be doing some more work on the Azure integration with some work he is doing now)

Thanks, let me wire those into my factory.

@toka thank you for looking into it. We are constantly improving Atmos and we release every week (AWS, Azure, GCP, Helm, toolchain, UI/UX, workflows, CI, etc.). Your contributions will be reviewed fast. Also, let us know if you need help with any of what you want to implement

I'm using OpenSepc daily. Here it's Spec Kit? Asking if I get it right.

We have't formally adopted speckit in atmos, but some features have gone out that way. We have slightly less formal process, it's its in

docs/prds

.

@Erik Osterman (Cloud Posse) we need to review both and implement some guidance github.com/github/spec-kit/discussions/1536

@Igor Rodionov was also looking into switching to openspec

From my teams experience I can share, we have started with SpecKit and moved to OpenSpec for good. OpenSpec proven itself to be easier to maintain over time, especially when design or implementation details change over time.

thanks for the info, very useful

In general I'm looking to wire CICD for monorepo into ArgoCD and more.

Have you seen this demo? atmos.tools/examples/local-gitops

Looks cool; I'm not sure how/if making commits fits into my case here. How would you wire cicd for apps, backends etc? So far the way I've found is through extending Atmos with custom commands to build, test etc.

Looks cool; I'm not sure how/if making commits fits into my case here.

With ArgoCD, do you use the deployment-repo pattern? E.g. a repo that represents the desired state of all manifests?

With ArgoCD, do you use the deployment-repo pattern? E.g. a repo that represents the desired state of all manifests?

Partly. I have a deployment repo that Argo CD watches, but it holds config only. Argo CD renders the Helm chart from an OCI registry at sync. Do you mean rendered manifests in the repo or does config-only count as the pattern for you?

If there's a specific gap you encounter, please bring it up and we'll see about closing it

I'm still in the process of finding way that suites us best and I'm looking for ways to move away from GH Actions and inline/bash scripts to actually build, compile, test APIs/backends and so on - that's why I'm looking into having a single interface to execute pipeline code and post PR comments and Summaries with templates, or for example how to wire OPA policy checks for those (and again, post output into a comment or summary) instead of wiring PR statuses and checks the usual way.

The following should be possible in 1.131.0 or 1.132.0. I think it addresses exactly what you're looking to solve.

commands:
  - name: app build
    description: Build and push the container image for a component
    arguments:
      - name: component
        required: true
    flags:
      - name: stack
        shorthand: s
        required: true
    steps:
      - name: build
        type: script
        interpreter: starlark
        script: |
          component = ctx.arguments["component"]
          stack = ctx.flags["stack"]

          def build():
              r = atmos.container("build", component, flags = {"stack": stack}, check = False)
              if r.exit_code != 0:
                  return r
              return atmos.container("push", component, flags = {"stack": stack}, check = False)

          # The image digest summary is already written by the container component in CI.
          result = ci.group("build", build)
          ok = result.exit_code == 0
          ci.output("built", "true" if ok else "false")
          ci.comment(
              ("✅ Built and pushed" if ok else "❌ Build failed") + " `" + component + "` @ `" + ci.context.sha[:12] + "`",
              key = "app-build:" + component + ":" + stack,
          )
          if not ok:
              fail("build failed")

  - name: app test
    description: Run tests and the OPA policy check for a component
    arguments:
      - name: component
        required: true
    flags:
      - name: stack
        shorthand: s
        required: true
    steps:
      - name: test
        type: script
        interpreter: starlark
        script: |
          component = ctx.arguments["component"]
          stack = ctx.flags["stack"]
          dependencies.tools("gotestyourself/gotestsum", "1.12.0")

          def run_tests():
              return exec.run(["gotestsum", "--junitfile", "junit.xml", "--", "./..."],
                              check = False, output = "capture")
          tests = ci.group("test", run_tests)
          if fs.exists("junit.xml"):  # A compile error produces no report.
              steps.junit(files = ["junit.xml"], title = "API tests")

          policy = atmos.validate("component", component, flags = {"stack": stack},
                                  check = False, output = "capture")
          if policy.exit_code != 0:
              ci.annotate("error", "OPA policy failed for " + component + " in " + stack, title = "policy")

          ok = tests.exit_code == 0 and policy.exit_code == 0
          ci.output("tested", "true" if ok else "false")
          body = "\n".join([
              "## Test: `" + component + "` in `" + stack + "`",
              "",
              "| Check | Result |",
              "|---|---|",
              "| Tests | " + ("✅" if tests.exit_code == 0 else "❌") + " |",
              "| OPA policy | " + ("✅" if policy.exit_code == 0 else "❌") + " |",
              "",
              "<details><summary>Policy output</summary>",
              "",
              "```",
              policy.stdout + policy.stderr,
              "```",
              "</details>",
          ])
          ci.summary(body)
          ci.comment(body, key = "app-test:" + component + ":" + stack)
          if not ok:
              fail("tests or policy failed")

  - name: app deploy
    description: Preview the deployment on PRs and locally, apply it on pushes
    arguments:
      - name: component
        required: true
    flags:
      - name: stack
        shorthand: s
        required: true
      - name: apply
        type: bool
        description: Apply even when this is not a push (for example from a laptop)
    steps:
      - name: deploy
        type: script
        interpreter: starlark
        script: |
          component = ctx.arguments["component"]
          stack = ctx.flags["stack"]
          key = "app-deploy:" + component + ":" + stack

          # Deploy must not trust an earlier job: re-check policy.
          policy = atmos.validate("component", component, flags = {"stack": stack},
                                  check = False, output = "capture")
          if policy.exit_code != 0:
              ci.comment("❌ Policy check failed, not deploying `" + component + "`", key = key)
              fail("policy check failed")

          if ctx.flags["apply"] or ci.context.event == "push":
              r = atmos.helm("deploy", component, stack, flags = {"target": "deployment-repo"},
                             check = False)
              ci.comment(("✅ Deployed" if r.exit_code == 0 else "❌ Deploy failed") + " `" + component + "`",
                         key = key)
              if r.exit_code != 0:
                  fail("deploy failed")
          else:
              # PRs and local runs: show what would change, change nothing.
              diff = atmos.helm("diff", component, stack, check = False, output = "capture")
              ci.comment("## Deploy preview: `" + component + "` in `" + stack + "`\n\n```diff\n" + diff.stdout + "\n```",
                         key = key)

  - name: app all
    description: Run build, test, and deploy in order; stops at the first failure
    arguments:
      - name: component
        required: true
    flags:
      - name: stack
        shorthand: s
        required: true
      - name: apply
        type: bool
    steps:
      - name: all
        type: script
        interpreter: starlark
        script: |
          component = ctx.arguments["component"]
          for phase in ["build", "test", "deploy"]:
              argv = ["app", phase, component, "--stack=" + ctx.flags["stack"]]
              if phase == "deploy" and ctx.flags["apply"]:
                  argv.append("--apply")
              ci.group(phase, lambda: atmos.run(argv))  # Raises on a nonzero exit, so the loop stops there.

Great minds think alike

It also supports an interpretive mode, so you can

#!/usr/bin/env atmos

and write CLI apps

Any advice how to approach solving CI now before it becomes available? in a way to avoid painting myself into a corner now. And is there anything I can do to contribute to accelerate development?

What is your timeline?

To short, a week, two at best. But I don't need to have everything polished, I can cover 1st and 2nd base and improve/extend along the way.

Here's how we use python today in our custom commands

can you confirm you're current use-case is with GHA + local exec

GHA + local exec, monorepo with 5 to 10 (micro)services/APIs backends for starters, with Nx under the hood for project discovery, graph for dependencies, identifying test targets, integration tests, e.g.

nx affected -t lint test build

with IaC via Atmos alongside 🍒 . So basically I would call Nx (or alike) in Atmos CI/custom_commands/steps, because setting and maintaining those is not my responsibility anyway, and then finally use Atmos to build artifacts, sign and push to AR.

aha, that's helpful

yeah I can see that

Yep, keep nx though for js/typescript projects

Rust, C#, Go but yes