If your husband isn't going to do it then you'll have to stick up for yourself. I had to do that in the beginning with my in laws when they were out of hand, and then my husband got on board. It took him longer to see it my way, probably because it is his parents and he doesn't want to see it, but he does now and we are on the same page. There's no need for someone to walk off with an infant that isn't theirs!
Totally agree with Amber I also had to set up my boundaries and when talking isn’t enough and they don’t listen I act and I really do I just stand up and go between my child and anyone who is trying to do something I don’t agree with.. then they understand. My husband also took a bit longer to understand my points but now knowing his child better he also stands up. Good luck I know it’s not easy but don’t forget you are doing only what’s best for you and your baby!
he pushes your buttons because he knows he can because your husband doesn’t stick up for you and it’s going to be a long time for him to do so