I'm also not sure that we want to encourage further ambiguity and inconsistency. Either call the rules that are on the books, or change the rules to be specific about certain times, or player positioning, which requires a call or no-call.
More importantly: Something is a foul (or not), based on whether it can be used to give an unfair advantage to one side or the other, based on how the fundamental game ought to look, and what the players ought to be trying to do / prevent the other team from doing.
A standard of "let the players decide the game" is farcical, especially to hear a trained lawyer say, when the plain meaning of that statement means "call some fouls and ignore others, according to an arbitrary standard that no two people would have the same definition of". Say what you will about the reviews on things like catch / no-catch, fumble / down-by-contact, blocking downfield, hell even DPI / OPI, at least when you slow them down and get people who know the rules talking about them, the "right answer" is usually consistent.
Holding, or other blocking fouls, are pretty much the most arbitrarily-enforced rules in sports, unless you're a fan of the carry rule in basketball. People need to decide whether those things should be legal or illegal.