Hi Lost, welcome
Relationship advice is not an easy thing to offer, it is based on personal opinion by people like me that are not professionals so as long as you know that.
Early signs appear to be that this man doesnt fit in the love glove of what you require nor desire. The "good times" do not compensate for the negatives he is dishing out and his history with his ex re: violence tells you everything but I must highlight- this in no way is something you can be blamed for. Early emotions eg " I thought I’d finally found someone that will treat me right." was based on initial feelings and that is entirely understandable. You are being too hard on yourself.
I would pack up and leave and return to your life where you lived. Your children will secretly thankyou for it if not straight away, soon.
A bit early to say this but I've had 4 long term relationships all over 7 years. I've had a narcissist and an alcoholic and like you was going to give up. In fact I had a cabin built in the hills to be on my own with my daughter. My best friend, I invited her for a dinner 10 years ago and we ended up marrying, I'm very happy. She is "mum" to my daughter. So it is a case I think to remain open minded and in future take more time deciding your future. If you move in with someone you havent had one big argument with, you get a shock as to how they handle it.
I wish you well. Your future is yours to decide.