How do you manage not even an hour apart? That may not be the best for either of you. He needs to get to know other people and you need a life outside of motherhood. Don't let your fears become reality. Don't let them control you.
It sounds like your mother was sadly affected by your fathers drinking and she tried her best.
I found myself just recently thinking of things in my family that were far from perfect. Then I dug deeper and started to remember the good times. Little things that made me feel so much better. I found there were a lot of them. Maybe try that. You must have had some good times. That might help you with your fears.
I'm sorry about your mom. That is hard for you not being able to tie up loose ends in your relationship.
You seem really fragile right now. Take care of yourself.