Is 4 tokens per second usable?

I think most folks consider the 'router' as a POC that demonstrates how much working memory is needed for larger models. That said, even 0.1 tok/s would let you have an overnight job on a non-gpu instance. That can be powerful enough for people who just need one 'smart task'. Also, for air-gapped systems, this is a big shift in what can be accomplished I just think it's worth mentioning as the start of something that could snowball into something big