This seems too strong. A consensus would be better, as suggested by this previous proposal:
The rules should be probably be weaker than for IDs, though. A simple majority (like the DQA voting) would be sufficient (and probably much easier to implement). If there was a pressing need to reset an annotation, pinging a few knowledgeable users would overturn the vote quickly enough.