It's amazing how something that sounds kind of goofy in description manages to be thoroughly unsettling in execution. I think it's the wavering pitch of that voice.
Fun fact: Vincent Price and Charles Herbert could reportedly not film this scene without bursting into hysterical laughter. Eventually, to get through it, they agreed not to look at each other.