Why he did it is complicated. Perhaps something happened to him or perhaps he was simply curious what a girl's body felt like as he was maturing through puberty at the time. Don't let that latter explanation seem innocent, it was very very wrong regardless and should have never happened, but if that was the reason. Then it would happen because he didn't have proper boundaries developed. Those boundaries would not have developed properly considering your family environment and the physical abuse from your father. I would definently suggest confronting him about it, learn why he did it and get an apology at the very least. If it was only this one time it dosn't concern me that much. I know that even one time can have strong negative effects on a person. Only you though know if you think it's had effects on you. If you think it has definently get a therapist and talk to him/her. I know it may seem difficult because you'd have to get your parents involved but it's just something you'd have to step up to.
I am personally more concerned with your dad's anger issues. Even if they're over now and as you put it "he never hurt us too badly" simply hitting you any physical abuse like that can be quite devastating. That and your parents fights depending how bad they got? I mean verbal only or would he loose it and start hitting her? Those can severely affect a person; how they affect everyone always changes. Personally I went for depression and suicide but I had alot more things affecting me as well. If your not having any emotional problems like that or an inability to control your emotions especially anger. Then your doing great. Infact you sound great. The only thing that really stands out is that your afraid of being intimate with people emotionally and especially physically; watching your parents fight especially if it got physical I'm not surprised at all. It's a common side effect. The physical abuse and family environment that resulted from that and your parents fighting is going to create problems for you when it comes to relationships and likely cause other problems in life as well. Parenting ability being the most obvious. I'm sorry but I'd have to know alot more about you to notice any developmental problems it's had and naming the ones that could result is just pointless. Relationship and parenting problems are the two that, will, happen.. You can see therapy and counselling for the physical abuse and family environment and I always suggest that. I personally though get the feeling that the majority of this stuff will work itself out as you grow up so long as you keep standing up to it as you are now.
You got scared when your boyfriend kissed you, most likely, because of your parents fighting and you connecting at a young age pain with intimacy because of the fighting and the physical abuse. I know that that is particularly stressing you out so let me be clear it's very very very healthy for relationships to fall apart even into your late twenties. I know you won't understand that but trust me it's very healthy. We learn from the mistakes to make better ones next time. And it can take a long time for people to learn how to have relationships and how to love. So don't worry when your relationships fall apart just learn from it and go onto the next one. The only time you should start getting worried about your relationships is if you keep having them but they never improve with each one. In that case you need to figure out why and if you can't or it's serious then you get therapy.
As for why you can't stop thinking about the molestation, I think your trying to blame it for alot of things. So figure out, what, your trying to blame it for and then confront those problems. With that and after you confront your brother I think you'll be able to get it under control and won't be thinking about it all the time.
I really hope that helps! If you have any more questions or problems or anything anytime please don't hesitate to share. Good luck.