It was so confusing for me as a kid - I remember being about 4 years old looking out the window at my 2 older brothers playing with their friend (who had a similar hair color/style) and trying to figure out which one was the Not-Brother - I couldn’t puzzle it out. Or once when I was with my mom at the store, and a guy with facial hair walked past with a son about the same age as my brother, and I remember staring at them trying to figure out if it was my dad and one of my brothers, or a stranger. My mom didn’t seem to notice them, and I knew we’d taken the only family car, so I was able to follow the logic train and conclude that it could not be my family members - but I absolutely could not tell just by looking.
I’ve gotten better at recognizing people in adulthood, but I still probably won’t recognize a person I’m used to seeing at work or school if I run into them in a context I don’t expect, like the grocery store in a different part of town for example. Usually I wait until someone greets me, then have a very rapid mental process that goes “Who do I know who has brown hair AND wears glasses AND lives in this part of town AND is very short - ah, it must be Jane!”
Interestingly, I found a weird loophole in my face blindness a couple of years ago. I find I can remember objects just fine… and that includes photographs, if it’s a physical printed one. So while I can’t picture a face, I can picture the framed photo that sits on my desk, and I can kind of “see” the faces of the people in it that way. Not in great detail, but certainly much more so than I can when just trying to imagine them!