Postnatal depression impact on baby

Hi everyone, Just looking for a bit of advice really. I struggled really bad from postnatal depression at the start of been a mummy. So me and my baby went to stay with my parents for a while and my mum helped care for my LO. Anyway I have been home a while now and my mum sometimes has my LO for the day, but every time she does he seems to be unsettled, he has been really bad today though and wouldn't settle at all. But as soon as she brought him home he was fine. Do you think he could remember me been poorly and that's why he is really unsettled?