To me there’s a difference between people who left a description and people who just input a seemingly nonsense ID. An observer might say “There is a bird in this tree” (bird is not visible) or “I want to know what are the little butterflies on this bush” (the butterflies in question didn’t hold still enough to make it into the picture.) I would neither agree nor disagree with their ID, since I wasn’t present and can’t say what was there. I’d probably leave a comment with the name of the plant. I have not done it before myself, but I suppose you have the right to down-vote the evidence of organism; the user is logging an observation of that bird/butterfly with no evidence; a casual observation. In these cases where it is pretty clear the user has an observation subject in mind, I don’t believe you have the right to change the subject by putting the ID of the plant.
But then the other case, the people who didn’t explain themselves at all and really have an ID that makes ZERO sense (let’s say the image is a flower and they put a fish species as the ID) I’d probably add a disagreement, by IDing the flower, and leave a note.