You answered your question in the first sentence when you said "when I let something bother me". You are in control, and you allow things to bother you.
I get a feeling that you question your own value, and you want other people to tell you that you are a lovely person.
The solution is in asking yourself why these things bother you, until you get to the real answer. For example, here are a few possible reasons:
One is that they were too busy with their own things to think about you. Sometimes you get so busy that you forget about other people and what is important for them.
Another reason is that they do not attach much value to birthdays and always forget them. That does not make you a less special person. That is just how some people are, and you should love them for being as they are. There are things that you also not really value, but those things are important to other people.
Or maybe they truly did forget your birthday. You have also forgotten things that caused you some embarrassment later on.
No matter what the reason is, whenever you feel that people care less about you, your response should be to care more about yourself. That will make people see you in a new light, and that will tell them that you are worth appreciating because you appreciate yourself.