How do we go about upgrading postgres versions to the latest supported by supabase? Or must we create a new project, and migrate to the new project?

For now the way you described is the way to go about it.