I enjoy headphone listening... But not as much as actual speakers in an actual room :-). I see headphones more as a tool for detailed listening, and will reach for them out of necessity when I'm on the road or for private listening. Otherwise, I would much prefer to be enjoying the sound from my speakers.
Beyond typical reasons such as comfort (no matter how comfortable, I'm just not a fan of things touching the ears or around the head), a big part of the issue is that headphones have never sounded natural to me. Subjectively, the "inside the head" sound, while I can get used to, just isn't an experience that I find particularly pleasurable. A reason I bought the Sennheiser HD800 a number of years back was because it was said that these headphones were capable of projecting sound so that the experience was more outside the cranium. Alas, if one is expecting any headphone to do this well, there will be inevitable disappointment; unless you're applying some kind of playback processing like crossfeed, the physics will not permit such a thing. As a result, I am intrigued by ways of making that head-fi experience more life-like which is what the products today can potentially do.