Since the DQA here is “can the Community Taxon still be confirmed or improved” (my emphasis), I would suggest that votes only need to be removed by the system if and when the Community Taxon changes. And then, both yes and no votes should be removed, since all of those votes were based on the Community Taxon at the time the votes were cast, and not on the newly changed Community Taxon. I would vote for something that functions this way.
EDIT: although, I did express this concern about such functionality a while back…