I finished the staging POC for scaling foreground inference. I basically proved that the solution can work on our infra for our application and its currently running side by side with the existing inference server.
This document is basically my thought process laid out from start to finish. It covers:
• Goals
• Why our current architecture does not meet our goals
• Tradeoffs
• Technology choices
• Architecture
• Cost
• etc.
My next step is to broadly run my plan by a friendly staff engineer to see if I missed anything and then focus on how to get to production