Oliver was raised by his mother with a lot of estrogen fueled pampering