Source: TW
The Veda empowers them expressly. At the end of the day, the Veda & Śistas form an insider community/religious system, into which others can enter but only on the terms as set by the Śistas.
Much like any of the myriad tribal/aboriginal religions which have existed. It would be as ludicrous as getting invited to a Hopi ritual or Aboriginal Australian spiritual ceremony & then asking the Hopi or Aboriginal elders, what gives them the incontrovertible authority in matters of Hopi or Aboriginal religions?
The foundation is unbroken transmission. There is a reason why the Purāṇas associate, in their chapters on the growth of the Veda or Ṛsis, expressly associate the contents of the Vedas with particular Ṛşi-descendants; the same sub-groups/lineages we see among that “detested” group even today. It’s like entering a club founded by certain individuals & ran by them since Devas-know-when, with a particular club constitution or vision, knowing full well that they do not intend to change that constitution/motto/vision except by way of a consensus among only the governing board of that club and none else, and then hating them for not changing the rules.
What I believe is what scriptures reveal already: Adhikāra for certain (not all) things has lineage as a component.
shiShTas need not be infallible; they only need to be more perfected & blessed, compared to us. Such is their Tapas & Pakvatva that even if their rulings were not absolutely perfect, it would be capable of yielding fruit for us.
As for the unfortunate ones who think that they are superior to Śiṣṭas, they have the option to indulge in whatever transgressions they please, provided they don’t cause impediments to those who wish to follow Śiṣṭācāra. They exercise that option anyway. Who stops them? But those who believe in & uphold Śruti, Smṛti & Śiṣṭācāra, even though they don’t stop anyone from violating those three sources of Dharma, are free to critique the violators. One can (figuratively) spit in the face of Īśvara, Śruti, Smṛti & Śiṣṭas by virtue of his or her choices, sure; but you can’t expect those who respect them to approve of said choices, let alone embrace, welcome or praise them.