Exactly. I've been told many times that a modification I plan to do to my Jeep, my car, my firearms will decrease the resale value. My response is, "I didn't buy it nor am I modifying it for the next guy". I liken it to being told by a car salesman that if I get the automatic transmission the car will have higher resale value. I respond, "Why would I pay an extra thousand dollars to have a transmission that I don't want so I can get an extra $50.00 6 years from now"?
My father always said that about seat covers. He'd say "Why should I look at ugly seats, if there are nice ones underneath?" or something like that.
That said, I have what is described as a "Glock Sock" on my P-95. That thing has slippery plastic grips without it. It adds a couple bumps and some rubbery grip, which helps in the hot or cold.