Am I being petty or do I have a right to feel this way?

You have a good enough reason to feel that way. In fact, I wouldn't blame you for keeping you and your child away from them. If they're saying things like that, there's no reason for you to be stressed out or for the baby (born or not) to be around that.