OK, first, for the sake of argument, we must define manliness. I will give two criteria that will be difficult to dispute for my definition: 1) The ability to conquer, and 2) the ability to attract the ladies.
For the first point, I bring up three of the most conquering societies in history: the Romans, the Mongols, and the English. The Romans were one of the earliest societies to make clean-shaven men the norm, and they are commonly acknowledged as some of the greatest conquerors of the ancient world. Surpassing them in conquering ability, however, was the Mongols, who ruled land from Bavaria to the Pacific Ocean. While commonly portrayed as bearded heathen, they actually eschewed facial hair. Then surpassing them, and everyone else, in conquering ability, was the British. The sun didn’t set on the English empire as during the 19th century they ruled land clear around the globe. And everyone knows that they are a traditionally rather clean-shaven bunch. Therefore, according to our powerful examples, facial hair is typically negatively associated with the most powerful conquering peoples.
As to attracting the ladies, most women do not like the scratchiness of beards and other forms of facial hair. Ask any marketing specialist and they will tell you that all of the studies in the Americas find clean-shaven men to be far more attractive to women than their bearded counterparts.
Therefore, based upon scientific evidence, facial hair is not a sign of manliness.