When we say that 'We don't have time' to do something, it implies that we are overloaded. Not only does this send a signal to others that we are operating at capacity (and therefore should not be considered for that great new project) but it signals to our brains that we are overwhelmed, increasing our feelings of stress.
Instead, replace it with 'It's not my priority'. The implication with this phrase is that your time, and how you spend it, is a choice. You are currently engaged in those activities of your choosing, that will propel you, your career and life, forward. If you are not happy with the way that things are going for you, then you have complete control over your priorities, and therefore have the time to implement the changes required.