It depends on what you are shooting at. Little tiny varmints, I would prefer more mag as well. Larger targets like deer, elk, or man sized steel or paper targets 10x is good. Targets are easier because the target is usually large and you have a center point to shoot at. Small varmints you need to be able to see them well in order to shoot them.
Downside to higher mag is increased mirage, more shake, harder to find targets. etc etc. But nice thing is you can always turn a high mag scope down but not the other way around.