I wash them because you never know who else might have tried them on in the store. You'd hope things like sealed multipacks of underwear wouldn't be tried on but, you know, yick. It doesn't take much to throw them in the wash first.
I wash before I wear. A lot of people don't realize there is leftover residues from manufacture or treatments to the fabric. If nothing else, the dye can rub off on you and your other clothing. I've had that happen before.