Pointing with middle finger

My baby only points with his middle finger. we correct him when we catch him, but it also doesn't seem like a big deal to us since we know he doesn't know it's a rude finger. Well it wasn't a problem untill a lady approach us in a restaurant to say he was cute, he was clapping and playing with his hands when he lifts his han in the air along with his middle finger ( he often does this just because) well the lady took offense and said that we weren't raising him right. I could care less what some stranger says, but it did get me thinking maybe his middle finger pointing is more of a problem than I originally thought.