Found another issue:
If I have an existing all-day event in Google calendar that's marked as busy, it is ignored by Cal when it checks availability. It still shows that time as available.
However, if I create a new all-day event and mark it busy on the same Google calendar, then Cal finds it and properly marks that time unavailable.