Are there any plans to have a function to generate the perplexity of a model on a given text? E.g. perplexity of qwen1.5b on a particular string?

Hey @Kyle, would love to learn what you want to use it for?

Mainly to assess how well a model has learnt a topic (so a list of texts) and compare it with another model's avg perplexity and use it as a proxy for acc (if it's been tested to be correlated)