This actually works two different ways:
- It advertises manliness by saying that behaving in a way perceived as feminine cannot tarnish it. Maybe his masculinity is so accepted that no one needs proof of his masculinity.
- Related to this was the use seen in Unseen Academicals: the wearer is so manly that he's not afraid of being challenged on the perceived femininity of his appearance.
- It suggests (or asserts) that the person is so manly that he causes his behaviors and attire to be masculine, rather than the other way around.