You are definitely not the asshole. He is a full grown man who has both a right and a left hand to take care of his own sexual needs. You are NOT required to take care of them. His blue balls are just that, HIS problem to deal with. Also, if he feels that comfortable telling you that he would cheat if you don't give him what he wants then leave his ass and buy him a bottle of lotion to leave with the divorce papers.
You are absolutely not the arsehole. It is completely unacceptable of your husband to tell you he would have cheated on you, if circumstances were different. It might have been a comment in the heat of the moment but a disrespectful comment nonetheless. If you can, try to find a time where you’re both calm and can talk about it from both points of view. Depending on the outcome I would have a hard think about next steps, potentially counselling where you’re in a moderated environment and both of you can be heard. I also have a very low/non-existent sex drive since the birth of my second baby and if my partner behaved like that I would absolutely be assessing my options.