What is the defining feature of a human being? His soul. The body is just the packaging, the wrapping paper.
What do we see? The body. And that is how we assess the person. Even worse - the clothing. An even more external wrapping.
It is true that a persons face reflects what is happening in their soul - the word "ponim" face, is related to the word "pnimiyus" - inside. But still, when we just look at the ponim, we are missing a lot. Kal Va-chomer when we assess people based on their clothing. Yet that is EXACTLY what we do, in the absence of any other available methods.
Remember - the real person is who his or her soul is and we can't see souls.
People often get married based on externals. The girl is physically beautiful so he wants to marry her but he may well be blinded by her beauty. Inside - she is flat out ugly. She has the looks of a top model but is really ugly.
The girl thinks the boy is really frum because of his bushy peyos but it might not be so.
Remember the pasuk:
כִּי הָאָדָם יִרְאֶה לַעֵינַיִם וה' יִרְאֶה לַלֵּבָב
