Who governs AI governance?

Abdelghany Fouda, Director of Legal Affairs at Al Sondos Holding in UAE, discusses governing the framework behind the algorithm – asking what happens when AI follows its governance rules perfectly, but the rules themselves are wrong.

What if an AI system follows its governance rules perfectly – but the rules are wrong?

In an earlier GC Elite article, I argued that AI is becoming a legal risk multiplier, turning some technical risks into foreseeable institutional exposures. Edimar Silva then pushed the discussion further by asking whether organisations can demonstrate governance through traceability, auditability and reconstruction.

But proving that governance operated as designed does not prove that the framework itself was legitimate or appropriate. If a system complies perfectly with rules built on unsuitable assumptions, the problem may lie not in the AI, but in the framework governing it.

The question shifts: who sets the rules, what gives that actor legitimacy, should the same rules operate across different legal and institutional environments, and how transparent should the choices behind them be?

Who Has the Right to Govern AI?

If governance determines what AI may do, what it must not do and what level of risk is acceptable, designing a governance framework cannot be a neutral technical exercise. It involves choices about values, priorities, human oversight and competing interests.

That requires a distinction between authority, expertise and legitimacy.

Developers understand technical capabilities, deploying organisations understand operational risks, regulators derive authority from law, and affected people hold rights and interests that may not be visible from within the design process. These perspectives do not automatically coincide, nor does any one of them necessarily have the legitimacy to set the balance between privacy and efficiency, automation and human control, or innovation and protection from harm.

International frameworks support this distinction from different directions. The Council of Europe Framework Convention aims to ensure AI lifecycle activities are consistent with human rights, democracy and the rule of law. UNESCO’s Recommendation emphasises multi-stakeholder governance, transparency and accountability. Neither supplies a single formula for legitimacy, but both resist reducing governance to a technical function. Comparative scholarship likewise supports broader, multidisciplinary participation in legal organisations and more distributed institutional oversight.

The more precise question is who has legitimate authority to determine the values, risks and trade-offs embedded in the rules – and to whom that actor is accountable. When a company or developer translates high-level principles into thresholds or decision boundaries affecting rights or opportunities, the exercise becomes one of institutional power as well as technical design.

A sound framework should be tested both by how it controls the system and by the legitimacy of the process that produced it.

One AI, Different Governance

If the first question is who has legitimacy to design the rules, the next is whether those rules should be identical everywhere. AI systems enter different legal systems, institutions and sectors, affecting different rights and interests; agreement on broad principles therefore does not require one method of implementation.

UNESCO states that the appropriate level of transparency and explainability depends on context. NIST’s AI RMF 1.0 likewise treats risk tolerance as highly contextual and use-case specific, while its Profiles can be tailored to sectors, applications, legal requirements and organisational priorities. A level of human oversight suitable for a low-impact internal tool cannot simply be transferred to recruitment, credit or public services, where outputs may directly affect rights or opportunities.

OECD work on interoperability makes the same broader point: the order of risk-management steps, target audience, scope and terminology may differ while frameworks still follow similar, sometimes functionally equivalent, processes.

Universal principles do not require uniform governance.

Accountability, human oversight and transparency may be shared principles while their institutional form, intensity and disclosure requirements differ with the decision, risks and rights affected.

Interoperability does not mean uniformity.

The danger is not that governance differs. Difference may show that a framework fits its environment. The greater risk lies in importing a framework designed for one sector, institution or legal system into another as a ready-made template, without testing its legal and institutional assumptions.

For general counsel operating across jurisdictions, the task is not simply to choose the “best” global framework, but to distinguish constant principles from mechanisms that must differ locally.

“A record proves that a choice was made; it does not, by itself, establish that the choice was legitimate or proportionate.”

The Hidden Black Box

Once governance is understood as a product of contextual choices, another question follows: who is entitled to understand why those choices were made?

Leading transparency frameworks largely focus on the system, its use and its outputs. OECD principles call for meaningful information about factors, processes or logic underlying an AI-enabled decision; UNESCO treats transparency and explainability as core principles; and the Council of Europe Framework Convention requires relevant information to enable affected persons to challenge AI-based decisions and even the use of the system itself.

But a system can be relatively transparent while the rules governing it remain opaque. We may know that a risk threshold was applied or human review was triggered, yet still not know why that threshold was selected, why review was required there rather than elsewhere, or what assumptions shaped the rule.

This article therefore proposes a distinction between two levels of transparency.

Transparency of execution asks what the system did and how it applied the rules.

Transparency of norm-setting asks how the rules themselves were selected, on what basis, what alternatives were considered or rejected, and what assumptions and trade-offs preceded their adoption.

This is an analytical distinction, not a term adopted by OECD, UNESCO or the Council of Europe. It matters because governance rules are not merely technical facts: they embody choices about acceptable risk, human oversight, disclosure and competing values.

The EU AI Act provides a useful but limited example. Annex IV requires technical documentation for high-risk AI systems to describe key design choices, rationales, assumptions, optimisation objectives and certain technical trade-offs. This does not create a general principle of transparency in governance design, but it shows that some pre-output choices may be important enough to document.

That leads to a second distinction: documentation does not equal justification.

Documenting that a threshold was adopted, or human review is required answers what was decided. It does not necessarily explain why that choice was appropriate for this environment, these rights and these risks. A record proves that a choice was made; it does not, by itself, establish that the choice was legitimate or proportionate.

Nor does this require absolute transparency. Trade secrets, cybersecurity, intellectual property and legitimate confidentiality matter. The question is not whether everything should be disclosed, but whether choices that materially shape rights, risks and accountability can remain entirely invisible.

If governance determines acceptable risk, human intervention and competing interests, accountability remains incomplete if we can scrutinise application but not the choices that produced the rules. Transparency therefore reaches beyond explaining the algorithm to revealing enough of the reasoning that shaped its governing framework.

The algorithm may no longer be the only black box. The governance framework itself can become one.

Governing the Governor

AI governance should not be assessed only by asking whether a system is safe, auditable or compliant with its internal rules. Each of those questions assumes that the governance framework itself deserves to be followed.

That assumption must also be tested through three connected lenses: legitimacy, context and transparency. Legitimacy asks who has authority to make the rules. Context asks whether those rules fit this legal and institutional environment. Transparency asks whether the material choices behind them can be understood and examined.

For general counsel, adopting a recognised framework or sophisticated internal policy should trigger further questions: who translated broad principles into operational rules, on what basis were acceptable risks defined, does the policy reflect the actual environment, and can its core choices withstand scrutiny?

Auditability may tell us whether AI followed the rules. Governance of governance asks the prior question: who made those rules, what gave them legitimacy, and why should they govern this system in this environment?

If AI governance is itself an exercise of power, one question should no longer remain outside the governance framework:

Who governs the governor?

Related Publications