In fact, ask yourself, even if you meet your ex-girlfriend with another man, will you feel uncomfortable? So, it's not your brother's fault. He's innocent. It's just your heart. It's an instinct that you mind. Animals are very possessive. Understandable.
There's just no need to get too attached to it. It's not the woman you gave up, but the relationship you gave her. In the future, I believe you will meet others. When a new relationship begins, you will get rid of this state of self-torture and even laugh at how stupid you are.