Basically, the following scenario applies: Select the attributes which you used during the manual annotation to decide which value is the right choice. That secures that the model uses the same knowledge you used before.
Example: If you are using the color Colorattribute to decide the gender, you should add this attribute to your training set.