You just explained why yourself. You're too concerned about what other people might think of your actions, and as a result, you try do take whatever path will upset the fewest people instead of doing what needs to be done.

Trying to please everyone will only end in you pleasing no one. That's one of the first things a leader has to learn. If you can't do that, then you really aren't fit to lead.