Would you be annoyed?

Not overreacting, I'd be blowing up his phone tbf 😅

Personally, I’d probably expect it, to be honest. If my husband was at a friend’s house watching the football, I’d assume he’d be drinking, and although I know he’d be sensible about it if our 3-year-old was going back to him, I’d still expect him to be home later than usual. To me, the odd one-off doesn’t really matter.
And unfortunately, as much as I sympathise with you feeling unwell, being a parent sometimes means having to just get on with it (again, just my personal opinion).
That being said, my husband wouldn’t leave me to deal with all three of our children while he went off to watch football with his friends. He helps me as much as he can, and if we were going trick-or-treating, he would have come along too. If he wanted to watch the football with his friends, he’d have them round to ours so he could still help with the kids.
I think for me, it’s less about him staying out late on a one-off and more about the balance of responsibility between you both.

I personally don’t agree with drinking while being in charge of a child so I’d be fuming. How does he plan to get home with the child if he’s been drinking all night?

I’m sorry to hear you are ill! So tough when looking after a baby. I probably would have expected him to come back settle your 4 year old to bed and then if he wanted to could have gone back out to friends. I personally wouldn’t be happy with other half drinking whilst on their own looking after child (unless I was there too, sober, and he could then have a few drinks)