Am I wrong?

Me and my boyfriend have a 5 month old daughter. He has a 3 year old daughter from a previously relationship. She stays with her mom and he gets her for 2 days every other week. Tonight he brought her home and she obviously has a cold so I told him they should sleep in the guest room or in her bedroom instead of in the room with me and baby because I don’t want our 5 month old getting sick. She’s already had rsv before and I’m the one that takes off when she’s sick and my job is already threatening my position because I’m out of PTO/sick and still have to take off when her or me are sick. He keeps saying their sisters and it shouldn’t be a problem but he’s missing my point of saying I don’t want her to get sick. It’d be different if they lived together but we literally don’t know if it’s just a cold, or if it’s flu, rsv, anything because neither of us know how long she’s been coughing and having symptoms. Now once again I’m being blamed for “treating his daughter different” when I’m literally just trying to protect my baby.