If this is a bug I will create a github issue, but honestly I'm not sure if it's a bug or I'm just misunderstanding something

Unexpected performance behaviour is always a bug, so please go ahead and open a GH issue and provide all the information the template asks for. Thanks.