A little of both... my current situation is pretty good. There's always room for improvement, but I've been in some spots where I was definitely penalized for having a family, and I'm grateful my situation has changed since then.
I worry though, sometimes, since I have so many kids that any legit reason I have for needing to leave early starts to sound like an excuse to slack off... I mean, how many times can you claim "my kid is sick" before it sounds unreal? Whatever that number is, you reach it pretty quickly when there are 5 kids at home and one of them brings a virus home from school!