I hope you don't mind me interjecting at this late stage of the thread. I have been following this thread with some interest as your experience appears to be a common problem in a lot of marriages - mine included.
Can I start off by saying that most divorces are initiated by the woman. There are many reasons for this, it may be DV or abuse; these issues are not gender specific. The same literature also says that divorced women generally end up "worse off".
It would appear that most couples enter into marriage with very different expectations. I think it is fair to say that most men enter a marriage with low expectations and most women have higher expectations; "and they all lived happily ever after".
Unfortunately, when life gets in the way the spouse with the higher expectations will disengage from the relationship. The high divorce rate initiated by women and this thread are perfect examples of disengagement due to unmet expectations.
You started this thread with, "I live with a perfect man .. I'm somehow always sad, lonely, rejected, neglected and depressed". A little later, you justified your feelings of rejection and neglect by saying, "he is too narcissistic and full of himself that he can’t be in the wrong". I noticed that you partially retracted the accusation.
Understandable, you then went looking for answers elsewhere. You referenced the book, "why does he do that?". I can only assume that you found the answer that justified your feelings as you went on to say, "I could see him within the lines of the book?" ... he is abusing me".
This type of self diagnosis always worries me. It is very easy to misinterpret, reinterpret the past. The so-called facts are often open to wide interpretation. For example, I could legitimately argue that goldilocks in the fairytale, "goldilocks and the three bears" was a hardened criminal - break and enter, theft of porridge, vandalism of chairs etc.
A little later, I think you hit the nail on the head when you said, you were losing your "sense of self-worth". I can only guess that this is where the real problem lies. Sometimes, it is easier to blame someone else for these negative feelings of self-worth. I know that is what my ex-wife did to me. My support was not enough, she needed constant assurance and admiration to fill a bottomless void within her.
I am not suggesting your decision to leave is right or wrong; only you can answer that question. If you do leave, make sure it is for the right reason.