Wondering how other people have their github/seed ...
# seed
o
Wondering how other people have their github/seed workflow set up - haven’t found something that checks all the boxes for a team setup: • The setup I tried has 3 upstream branches in Github (dev, staging, prod) and 3 stages in seed (same ones) • I tried auto-deploying each stage based on its branch: Merge via pull request from feature branches into upsteam dev, then merge from dev to staging via pull request (both using squash commits) • The issue was that because the commits in dev never end up in the staging branch, any PR from dev to staging contains every commit - so you can’t tell what’s actually being merged into staging and part of a release • Another option is to only have on upstream branch and do all promotion via the SEED UI, but I’d like to have a record in git of what’s been deployed in each release Is there a better way of managing the upstream branches in github to drive deployments?
What I used to do was to have stages auto-deploy from branches, then use local git to fast-forward staging to the commit in the dev branch I wanted to deploy, then push it upstream. That would keep all branches on one linear history just at different points I’ve also tried rebase merges in Github, but that creates a commit in the staging branch that doesn’t exist in the dev branch
s
We use this https://medium.com/serverless-transformation/serverless-flow-a-ci-cd-branching-workflow-optimized-for-speed-and-quality-6b98c5a4e489 , and also do use the promote feature of seed to release to production, earlier we used https://nvie.com/posts/a-successful-git-branching-model/ , but the problem is as we create everything by serverless (with custom cloudformation in it), wanted to see to the actual infra changes before releasing something to production, as we are still new to cloudformation, and earlier had issues with rollbacks and stack getting "stuck" , but with this serverless flow life seems much more peaceful
So in our case, master is in staging, ie commits to master (via feature branch merge), lead to auto release to staging. When we want to release to production, we just promote the staging to production, and seed shows a nice change list additionally to show what exact infra changes are coming to production
so if i am doing some very crucial infra change, i will tell other devs to stop prod release, first merge my mr to master, see staging is fine and then promote it to production
also we use the Auto PR flow of seed
s
I'm an advocate for trunk flow, one main upstream, and using promotion of staging. If the build artefacts you use in Dev/staging/prod come from different git histories, you run the risk of having something unexpected happen. A lot of people are coupling continuous integration with continuous delivery but I feel the two should be separate. Would it be possible to add a post deploy step in your seed buildspec.yml that puts commit sha and release tag into another tool, or, use the GitHub deployment API for this, and mark that Sha's currently deployed to Dev, Stag, Prod. I think something like this could be supported by seed webhooks (@Frank could confirm). This would enable you to keep GitHub as your source of truth, without dictating your git branching strategy (less overhead). This way you can keep track of releases, without needing to do all the branch merging and git history manipulation.
o
OK so seems like both of you are using one upstream branch, with all the promotion happening in SEED
I did find out how to do fast forward merges with Github - turns out you can’t, but you can use git and push upstream. The process would be to open the pr for promotion, have it pass review and checks, then fast-forward the branch getting merged into locally, and push it directly upstream. This will automatically close the PR. Sounds dangerous, but this process actually respects the protected branch settings, and only lets you push upstream is there’s a PR open that’s green to merge
f
add a post deploy step in your seed buildspec.yml that puts commit sha and release tag into another tool
Yup, I’ve seen folks tag the commit after deploy.
@Omi Chowdhury It felt a bit weird that dev commits are squashed and merge to staging. I always only squashed commits from “ephemeral” branches (ie. feature/bug-fix branches) that are removed on merge. So if we had
dev
and
staging
branches, it’s easier to see how many commits
staging
lags
dev
.
o
Yup - that’s why I wanted to use Github’s rebase and merge strategy for those PRs - so that
staging
would get fast-forwarded to where
dev
is at - but GH adds a new commit with a different hash on top of what’s in
dev
. Hence doing it locally and pushing
SEED’s branch auto-deploy doesn’t get triggered when I push to upstream with a fast forward update
Also sometimes some github build statuses get stuck in yellow, but rerunning the build in seed gets them to go green
f
SEED’s branch auto-deploy doesn’t get triggered when I push to upstream with a fast forward update
Taking a look!
github build statuses get stuck in yellow
Hmm.. sounds like the API to GitHub to update the status are timing out. How often does this happen?
o
It’s a function of us having ~20 services in seed, then that same commit gets more status checks as it goes through each stage. So only 1 in 60 needs to fail for the entire commit to get stuck. We also probably don’t need a status per service in Github, although its nice to know exactly which service failed in a deploy
f
Got it.. Seems to happen fairly frequently. Will take a look.
o
Hey Frank, just wanted to ping you on this, we’ve made the problem worse 😛 - now we have 6 envs all deploying the same commit