I've been feeling really low about this. Mine is 7 weeks today and we've hardly had a smile yet (he almost did one for his dad). Anytime he is near me, he just frantically tracks for my boobs or cries. I didn't think much of it until my health visitor really worried me about it. We have loads of sensory toys and we talk and sing and take him on walks but he seems to just be a very unhappy baby. I suppose you just have to think that you're the one keeping him alive and take some comfort in that. Mine will only settle/fall asleep on me which makes me think he doesn't hate me! I understand your worry though, you love them so much, you just want them to be happy x
It's still a bit young for many smiles though, my kid smiles in the morning when she wakes up and maybe once or twice in the day and that's it so most the day isn't smiles but it's still so young a time