I can't help but to think in a deviously, dastardly way, "Dude, I had a swell time with your wife." or, "Geez, dude. Your wife's pussy is tight." or thinking while we're in the act, "Oh, man. I'm ON TOP of your wife, right now. Ohh, she's a great kisser.", or, "Oh, dude. Her head game is ridiculous! And you were tired of fucking THIS??"

Fuck yeah.
Of course, this is all from a personal fantasy standpoint. I would never want to taunt a guy who's having marital problems. It's just fun to do this kind of thing in my head, you know.
But other times, it's kinda cool going to the 12th floor of a high rise, fancy schmancy hotel. Stepping out to the window and looking over the city, thinking, "Yeah, I'm fixing to get some pussy."

Then again, bending over a short skirt wearing with red panties peaking under chick in a raunchy notel where you had to walk past a creepy weirdo, one hall light out, oh fuck slightly clinched up bracing for the worst but made it ok...."did I lock my car doors, yeah, I did...I think, let me hit the key one more time to be sure....ok, I'm good...I think" scenarios has its merit, too. If she ends up being a good, no FUCKING GREAT fuck, it's totally worth it.
So, there you go. It all depends.