I've been thinking about this a lot this weekend actually, so I can ramble on a bit about it:
First, let's see what the Catechism says: "On Sundays and other holy days of obligation, the faithful are to refrain from engaging in work or activities that hinder the worship owed to God, the joy proper to the Lord's Day, the performance of the works of mercy, and the appropriate relaxation of mind and body...
Sunday is traditionally consecrated by Christian piety to good works and humble service of the sick, the infirm, and the elderly. Christians will also sanctify Sunday by devoting time and care to their families and relatives, often difficult to do on other days of the week. Sunday is a time for reflection, silence, cultivation of the mind, and meditation which furthers the growth of the Christian interior life." (2185-2186)
It seems like it's not saying to literally physically rest all day or even avoid any exertion, but just to take a step away from all the worldly 'rat race' type things that distract us from God. In a sense, this seems actually very freeing - there's tons of things I would like to do that I often don't have time to because of school, work, etc. In addition to giving honor to God, this seems very beneficial to health as well!
So things that you mentioned like reading, music, and exercise (if it's not stressful or anything) are perfectly valid ways to spend time. Spending extra time with family and friends and in prayer seem even better. The thing that trips me up is avoiding using that as an excuse to lay around all day and accomplish nothing. It's so easy for me to lay in bed and open Reddit or Youtube and just spend hours on it, hours wasted that I could have used for something actually meaningful. But I always have to remember that each second of time is a gift from God, so we want to make sure we don't waste those gifts.
To me it seems like a lot of it has to do with the reason behind the things you are doing. For instance, I'm not certain, but I don't think the obligation rules out charity or service-type work, because you are practicing the virtues and doing as Christ taught. Here's a (rather long) quote I found that I think explains this quite nicely:
"If you have a family, perhaps you might wish to get together as a family to plan special family activities for Sunday. This does not mean you need to spend money. Even what otherwise might be a “chore,” such as gardening or working on a home improvement project, might offer opportunities for “spiritual enrichment, greater freedom, [and] fraternal communion” when done together as a family. If you are single, you could plan to spend the day with friends or with extended family. If you have a skill you truly enjoy, such as cooking or some other creative activity, perhaps you might offer it to someone in need. For example, perhaps you have an elderly neighbor who would enjoy a home-cooked meal. Perhaps there is a local charity that would be happy to accept hand-sewn clothing or hand-crafted toys for the needy in your community. In short, Sundays and the holy days of obligation should be time each week, and several days throughout the year, to live out the joy of our Lord’s Resurrection."(https://www.catholic.com/magazine/online-edition/keeping-holy-the-lords-day)
So like I said, this obligation to me seems more freeing than anything; you get to step back from life and just give the entire day to God, doing things that will actually enrich your life. I wouldn't stress about it too much, or else you are defeating the purpose of the day!