Soren Jensen
04/28/2023, 2:47 PMbootstrap/
├── .github/workflows
│ ├── aws_core.yml
│ ├── client.yml
│ ├── data_api.yml
│ └── ingestion_api.yml
├── aws_core/
├── client/
├── data_api/
├── ingestion_api/
├── shared_lib/
Each module has its own folder with a nice mixture between terraform, python and nodejs files. and it's own pipeline defined in .github
We got a few share libs with code used in multiple modules which triggers multiple pipelines. E.g. a chance in the shared_lib will trigger the pipelines for all our api's.
Pipeline for aws_core got the following triggers:
name: aws_core
on:
push:
paths:
- .github/workflows/aws_core.yml
- scripts/load_env.py
- aws_core/**
Pipeline for the data_api
name: data_api
on:
push:
paths:
- .github/workflows/data_api.yml
- scripts/load_env.py
- scripts/requirements.txt
- data_api/**
- api_lambda_infra/**
- shared_lib/**
Pipeline for shared_lib
name: shared_lib
on:
push:
paths:
- .github/workflows/shared_lib.yml
- scripts/requirements.txt
- shared_lib/**
Each pipeline got it's own test job, the first problem is when opening up a PR to merge a branch into main if you select Auto-Merge it merges before all the pipelines has completed, so some tests might fail after the merge is complete breaking prod. 💥
The second problem is as you can see in the pipelines above they are only triggered on certain changes as we only want the relevant pipelines to run when working on a branch. It's common we modify code in multiple modules on the same branch, e.g. a change there require a new endpoint in the api and a change in the client. Committing this step by step will trigger one pipeline and in next commit another. Before merging into main it would be nice to force all pipelines to run.
Any suggestions on how to get this working? How do others working with a mono repo solve these problems?