For starters, there's the whole "size" issue. I understand that the size doesn't really matter, and more women complain of penises being too big (thank you, Loveline), but tell that to the man with a less-than-average size dick. He's not going to agree, at least not in his heart. No matter what, he's always going to see his little man as too little. And, of course, it's no help that society is chock full of humorous spite for this poor bastard. If someone spends lot of money, drives a big, expensive car, or is really cocky, well, of course he's "compensating for something." In comedy and movies, indicating that someone has a small dick only serves to support the idiocy or ridiculousness of their negative character. If a guy is in the locker room (especially in high school; those kids can be straight evil) and his dick is the smallest, he's going to get ragged on about his size, or lack thereof. It might even affect his relations with girls; I mean, after all, what is the both the quickest and most hard-hitting insult that a women can hurl at a man? Tell him he's lame in the sack, and his cock is so tiny. He's not going to be very eager to risk that humiliation, especially after the girl tells all of her friends. If you're not huge, or at least average, your penis size can be very detrimental to your confidence and happiness.
Personally, I'm pretty happy with my penis. He's not really that big, but he gets the job done. In all honesty, I'm pretty sure that he's a touch below average. But, of course, one must also realize that the penis, in all it's wonder, never really stays the same size for very long. If it's really, really cold, it can be just a tiny little nub peeking out from a bit of shrubbery, and if it armed and ready for love, it can be a huge, aching monstrosity.(Fun Fact: If a penis is huge, it won't really get that much bigger when erect, but if it's smaller, then it usually grows quite a bit.) Or it resides somewhere between these states, changing size according to mood, temperature, and arousal.
The penis obviously plays a large part in a man's self image. If he's not happy when he looks in the mirror, he's just plain not going to be happy. Men are just as vain and worried about their looks and bodies as women are. Although I'm content with my penis, I constantly question other areas of my appearance. Sometimes I worry if I'm getting to fat, or my weird hips freak me out, or I feel badly because I'm not too muscular, you know, whatever. Point is, even with my body issues, I can accept my appearance because I accept my penis. It is the cornerstone of a man's confidence in his body, and if he has issues with his penis, he will have issues with his whole self-image, no matter how built the rest of him is.
Well, that's all I got 'til I can think of something else. People, feel free to contribute. Mahalo.