@Marcos Marx (Airbyte) I have resolved this issue. Here is a brief description should this help others in the future. TL;DR I had custom DNS setup on my vm and that made it so that the workers couldn't connect to the temporal container.
The vm I was running on was an ubuntu 22.04 running on a t3a.xlarge instance in AWS. Initially it was running Alpha-0.36.2 and it appears that the workers were not able to connect to the temporal container, I pulled the latest docker-compose and environment files from github (0.40.29) and it ran into the same issue. I then setup a new vm using Amazon Linux 2 AMI and pulled the latest version, and it ran successfully. I then setup the custom DNS on the new machine and ran into the same issue, so I scrapped the DNS and restarted and it started up again.