Generally I start on Normal. If I like the game enough to play again, I'll take it to Hard mode. If I feel like it. I play mainly for story and frankly, most difficulty settings are more cheap than challenging. In my opinion. I've gotten to a point where I find that sort of thing frustrating and don't find the bragging rights or whatever to be worth the effort.

That said, the only game I've ever really had to turn down the difficulty on in recent memory was Dragon Age: Origins, and then only for certain parts. It's one of those games I put on hold until I really feel like putting in the time to plan out my build and all that.