I completely get this. I always get a sense about someone and always follow my gut if I think something is off about them! When I was younger there was a man who started at my work and from the off I said there was something about him that made me feel very unsettled and uncomfortable, I would always avoid him and declined his Facebook friend request multiple times. A few months later he was sacked following an arrest for possessing child pornography! It always concerned me how many people at my work with children accepted his request on Facebook as I just had this gut feeling about him.
Follow your gut feeling, if you feel something is off or creepy, keep your baby away from him as much as possible! I know somebody who never allowed her children near her uncle and had a strong feeling about him, years later it came out that the uncle was abusing her cousin (uncles nephew) for years and years.. our gut instincts are usually right x