Relationship advice? Am I wrong to leave? Feelings of guilt. 💔

We've been together 9 years. And have two kids together (6) and (1). We are unmarried and he has financially carried our family since our eldest was born. He drinks, every night to the point of nearly stumbling. It's been an issue since my daughter was born. We both used to drink and go out a lot in our teens/20s. But once my daughter was born I essentially stopped (except special occasions like weddings, bachelorette parties and birthdays). I've asked him to cut back, and we've went to counseling about it. He didn't like that the counselor was asking him to cut back and so we stopped going. Things have been shakey and we fight frequently. Often in the evenings when he's a few beers deep. He works as a firefighter and is often gone for most of the week (stays at the station). But he has a recent back injury that has him home 24-7. Our fighting has increased, we rarely have sex, and 99% of the child and household stuff is my responsibility. I recently had to go back to work part time to help make up for him not working. And it's been great to get out of the house, but when I'm gone he stays home with our 1 year old and does the other child duties. He has become increasingly resentful towards me for working. Even though I'm working because we needed it. Given I'm working a retail position he has no respect for my job and often mocks it. On Saturday, after he'd spent his entire day hiking with his friend... While I was sick and home with the kids. He came home, began drinking and started an argument with our 6 year old. It quickly escalated to him yelling at her about this being HIS house and we just live here. My daughter cried and yelled back that it was not his house but our house as a family. I asked him to stop and he went on the porch. I followed him and told him he was being mean and it was all unprovoked. He yelled at me to go move in with my mom then. The next morning I packed up clothes, toys and stuff for the kids and headed to my mom's house (an hour away). Once I got here he sent me a message asking me to "reconsider the distance that I was taking" and that he wanted me in his life. Am I doing the right thing by leaving? I feel like I'm failing my children. This is not what I want for them. But also neither is staying. I'm so heartbroken. And he sees no wrong in what has happened and what he said on Saturday (primarily because this isn't the first time I stayed at my mom's with the kids).