I've definitely had great sex with people whose physical features I wouldn't normally be taken with -- however they all had serious skills to make up for it. And during sex my eyes are rolled back in my head most of the time anyway, so what he looks like when we're intimate? Not as important as knowing what to do with his hands/mouth/etc.
However, I've never had sex with someone I hated or despised. I really don't see that working for me. How do you make the transition from wanting to shiv someone to ripping his clothes off?