Can't unset an observation field

Yes, clicking the x clears the value of the field (until you save…), but doesn’t remove the field itself.