For many people saying "no" is the most difficult thing in the world.
When they say no they feel guilty.
They don't know how to refuse or turn down an invitation, an offer, or even a lousy deal.
If you are one of them, what you need to do is when you don't feel like it, just say no first and feel guilty later. You'll find to your surprise that instead you'd feel super good about yourself rather than feeling guilty at all.
A book I read decades ago:
https://www.goodreads.com/book/show/1098624.When_I_Say_No_I_Feel_Guilty