Well, you have to understand a few things about “wilderness” and the assumptions made about it by those folks and others.
First off, the idea of “untrammeled wilderness” is a fantasy born out of colonialism, diseases, and genocide (inadvertent or not). The modern idea of “wilderness”, the one that conservation has been working with until recently, the one that most people have in their heads, that of a place where no people have set foot and nature reigns over all, came out of European colonists moving across the Americas and entering areas on the heels of virulent diseases they’d brought with them that wiped out 85-100% of the native people it encountered as the diseases spread across two continents.
What people saw as “wilderness” with no people in it was a system in the midst of a radical ecological overturn as a keystone species had been dropped from that position. Coupled with this was the inability of the colonialists to recognize the land management practices native people used, so even when the encountered managed lands they often though they were “empty” lands.
The only places on Earth in the last 1000 years that have had “wilderness” in the manner people think of it are Antartica, some small, remote islands people never reached, and a few extremely inhospitable environments, mainly cold ones. Everywhere else has humans occupying and using it at least part of the time. As much as I like and admire Lepold and Muir, there was a lot they didn’t know or understand as they lacked some of the necessary pieces of the larger picture. Edward Abbey is a larger than life character, but he is also somewhat of a caricature as well, something I think he would not only agree with, but that he reveled in, and his take on many things is more than a little libertarian and irresponsible (eg. measuring distances by how many beers you drink along the way, and throwing the cans out the window because cleaning them up gives the Boy Scouts and others “something to do”).
Setting the history of the concept of “wilderness” aside, environments have a carrying capacity. That’s a fundamental aspect of ecology and biology (and social sciences too, for that matter). Every person, no matter how well meaning nor how careful, has an impact. At a moderate number that impact is below the threshold of permanent damage, or within the recovery capacity of the environment, if you prefer. Above that threshold the impacts are too great and permanent (at least on a human relevant timescale) damage is done; there is no way around this, it’s a basic, unavoidable fact, like gravity.
Sure, you can try to mitigate it, but many of the mitigation techniques require infrastructure (eg. a few people going into a natural area can shit where they like, but more than X amount and you need to start putting in bathrooms and sewage systems) which exacerbates the impact. Again, unavoidable fact, like gravity.
It’s a bit like your house (eco → oikos → home after all); maybe 3 or 4 people can live there comfortably if everyone pulls their weight. For a few months maybe you can add a few extra guests without posing significant stress, but they can’t live with you permanently. You can throw a weekend party with 20-30 people no problem, but they can’t all stay the night. You can have a hundred or so people for an hour or two, but beyond that there are problems as its far too many people for your 2.5 bathrooms to keep up with.
Responsible ecological management absolutely must take into account the capacity of the environment to cope with the added stresses people place on it when they visit. To do otherwise is little different from strip-mining an area for its coal or copper, we just do it for its beauty for as long as that beauty lasts, all the while chipping away at what creates that beauty (biodiversity, lack of infrastructure, quiet, cleanliness, lack of roads, etc), then move on leaving desolation in our wake.
The places that have those requirements (as long as they are set intelligently and responsibly) are exactly the sorts of places that need greater support, even though the restrictions feel stifling at times.