I can relate. I've had many interests, each of which took time away from the others. At the end of the day, there are many ways to go about alleviating the guilt you feel. It just depends on what you want from your experience of life. I used to struggle with this, until I changed my frame of mind around how it affects me. I have settled on the understanding that we will never have enough time to do everything we're interested in, and much less to discover ALL OF THE THINGS you could be interested in. So, to me it just made sense to stop thinking too much about it and welcome it as a part of my life experience. When I relaxed about it, I found that many of my interests dissipated and only those that resonated on a deeper level stayed with me.