Baby always seems sad?

I'm just so worried that he's not happy. Lots of things make him cry, he whines so easily, like if I go to the bathroom, so I now try to take him with me every single time I need to go. I just always worry he's upset or in pain, it's just a constant anxiety of mine. He never seems happy. He has bursts of energy, where he will giggle at a silly face and we'll play and he'll seem happy, but otherwise he just seems so sad