AI Governance in Regulated Industries: A Practitioner's Framework from FDA to Fortune 100

A Career at the Intersection of Power and Constraint

I've spent my career at the intersection of enterprise integration and regulated industries — first at the U.S. Department of Veterans Affairs building HL7v2 interoperability pipelines, then at the FDA modernizing the drug review process through middleware integration, and now at Salesforce working with pharmaceutical and healthcare giants on AI-powered solutions. The common thread through all of it: making powerful technology work within strict governance constraints.

This isn't a theoretical exercise. At the VA, governance meant ensuring that clinical data reached the right clinician at the right time — every time — because a failure could affect patient safety. At the FDA, governance meant maintaining an immutable audit trail for every piece of data in the drug review pipeline, because the integrity of drug approvals depends on the integrity of the data behind them. At Salesforce, governance means helping organizations like Johnson & Johnson, Eli Lilly, Northwell Health, and AstraZeneca adopt AI in ways that accelerate their missions without violating the regulatory frameworks they operate within.

Through all of these experiences, I've developed a practitioner's framework for AI governance in regulated industries — not the kind of framework you find in a consultant's slide deck, but the kind you build through years of seeing what works, what fails, and what gets a project cancelled three months before launch.

The Governance Imperative: Why Regulated Industries Are Different

Regulated industries can't "move fast and break things." This isn't a cultural preference — it's a structural constraint enforced by law. Every system must be auditable. Every decision must be traceable. Every data flow must be documented. In pharmaceuticals, a drug that reaches the market through a flawed review process can harm millions. In healthcare, a clinical decision based on incorrect data can harm the patient in front of you. In financial services, an algorithmic decision that discriminates can violate civil rights law.

The governance overhead in these industries isn't bureaucratic waste — it's the price of operating in domains where mistakes have consequences that can't be reversed with a hotfix. An e-commerce company that ships a buggy recommendation engine loses some sales. A pharmaceutical company that ships a flawed clinical trial analysis loses lives and faces criminal liability.

This context matters for AI adoption because AI introduces a new category of risk: probabilistic decision-making in domains that historically required deterministic processes. When a rules-based system makes a decision, you can trace the exact rule that triggered. When an AI agent makes a decision, you're dealing with a probabilistic model that may not produce the same output given the same input, and whose reasoning is not fully transparent. Governance frameworks for AI must bridge this gap — providing the auditability and traceability that regulators demand while accommodating the inherent uncertainty of AI-driven processes.

Lesson 1: Governance Is Architecture, Not Policy

The most important lesson I learned at the FDA is that governance cannot be a document sitting in a SharePoint folder. If your governance strategy is a set of policies that people are expected to follow voluntarily, you don't have governance — you have aspirations.

At the FDA, the middleware platform I built had audit trails as a core data flow, not an afterthought. Every message transformation was logged. Every routing decision was captured with the context that drove it. Every delivery confirmation was recorded with a timestamp and recipient identity. The audit trail wasn't a reporting feature we could toggle on or off — it was as fundamental to the system as the message routing itself.

When auditors from the Office of Inspector General reviewed the system, they didn't need to trust that our team followed policies. They could query the system and receive a complete, immutable provenance trail for any piece of data: where it originated, how it was transformed, where it was routed, when it was delivered, and who accessed it. The architecture enforced governance in a way that policy documents never could.

This principle applies directly to AI governance. If you want AI agents to operate within compliance boundaries, those boundaries must be enforced architecturally:

• Audit trails must be a mandatory component of every agent invocation, not an optional logging level.
• Permission scopes must be enforced at the platform layer (the gateway, the MCP server, the orchestration framework), not left to the agent's system prompt to self-police.
• Data access controls must be implemented in the integration layer, so that an agent literally cannot access data it's not authorized for — regardless of what its prompt says.

If your governance can be bypassed by changing a prompt, it's not governance — it's a suggestion.

Lesson 2: The Gateway Is the Governance Layer

At Salesforce, working with MuleSoft Flex Gateway and the broader MuleSoft integration platform, I've seen how AI governance follows the same architectural pattern as API governance — but with higher stakes and more complex requirements.

The API gateway has been the enforcement point for API policies for over a decade: rate limiting, authentication, payload validation, routing. As AI traffic grows, the gateway is evolving into the AI Gateway — the natural enforcement point for AI-specific policies. Token metering replaces request counting. Prompt sanitization replaces payload validation. Agent identity verification replaces user authentication. Output moderation replaces response schema validation.

For regulated industries, the gateway-as-governance-layer pattern is especially powerful because it provides a centralized, auditable enforcement point that exists independently of any individual agent or model. When a regulator asks "how do you ensure that sensitive patient data isn't included in AI prompts?" the answer isn't "we told the agent not to" — it's "the gateway scans every prompt and redacts sensitive data before it reaches the model, and every redaction is logged."

This centralization also simplifies governance updates. When a new regulation takes effect or a policy changes, you update the gateway rules — and the change applies to every agent, every model, and every tool invocation that flows through the gateway. You don't need to update dozens of individual agent configurations and hope that none of them were missed.

Lesson 3: Identity Is the Foundation

At the VA, every HL7v2 message carried identity metadata: the sending facility, the sending application, the security credentials of the user who triggered the event. This wasn't optional — it was a required component of every message, because clinical data without provenance is useless and potentially dangerous.

In the agentic AI world, we need the same rigor for agent identity. When an AI agent accesses a patient record, generates a clinical recommendation, or updates a regulatory submission, the system must know: which agent acted, on whose authority, with what permissions, as part of what task, triggered by which human user.

This is what I call Dynamic Agent Authorization — contextual, time-bounded, scope-limited permissions that travel with the agent's task context. It's the identity framework that regulated industries need for autonomous systems, because static API keys and broad service accounts don't provide the granularity or auditability that regulators require.

The identity challenge becomes even more complex in multi-agent systems. When a supervisor agent delegates to a worker agent, which then invokes a tool via MCP, the identity chain must be preserved end-to-end. The tool's audit log should record not just "agent X accessed this data" but "agent X accessed this data as a delegate of supervisor Y, operating under task Z, which was initiated by human user W." This complete identity chain is what makes agent actions auditable in the way that regulated industries demand.

Working with Pharma and Healthcare at Salesforce

At Salesforce, I've had the privilege of working with some of the most important pharmaceutical and healthcare organizations in the world on their AI adoption journeys. Johnson & Johnson, Eli Lilly, Northwell Health, AstraZeneca — these organizations are early movers in adopting MCP, agentic architectures, and AI-powered automation. But they all share a common priority: governance first.

These organizations want AI agents that can accelerate drug development timelines, streamline clinical trial operations, improve patient engagement, and optimize supply chain management. The potential value is enormous — AI-assisted drug discovery alone could reduce development costs by billions and bring treatments to patients years earlier. But every one of these organizations has learned, through decades of operating in regulated environments, that capability without governance is liability.

The conversations I have with these teams follow a consistent pattern. They start with the use case: "We want an agent that can analyze clinical trial data and flag anomalies for review." Then they immediately move to governance: "How do we ensure the agent's analysis is auditable? How do we prevent the agent from accessing trial data it's not authorized for? How do we maintain the integrity of the audit trail if the agent's underlying model is updated? How do we demonstrate to the FDA that the agent's outputs meet the same rigor standards as human analysis?"

These aren't hypothetical concerns — they're the concerns that determine whether a project gets funded, whether it passes compliance review, and whether it can be deployed in production. And they're the concerns that my framework is designed to address.

A Practitioner's Framework for AI Governance in Regulated Industries

Based on my experience across the VA, the FDA, and Salesforce, here is the five-point framework I use for every AI governance engagement:

1. Classify First, Build Second

Before writing a single line of code, understand the risk classification of the system you're building. Under the EU AI Act, this means determining whether your system is minimal, limited, high, or unacceptable risk. Under FDA regulations, this means understanding the software's intended use and whether it qualifies as a medical device. Under financial services regulations, this means assessing whether the system makes or influences decisions covered by fair lending or anti-discrimination law.

Risk classification determines every downstream architectural decision: the level of audit trail detail, the human oversight requirements, the testing rigor, and the documentation obligations. Building first and classifying later is how projects get cancelled — you build a system that's fundamentally incompatible with its regulatory classification, and the remediation cost exceeds the remaining budget.

2. Audit Trails Are Infrastructure

Build audit trails into the architecture from day one. Every agent invocation, every tool call, every data access, every decision point, every human override — captured in an immutable, queryable log with full context. This isn't logging in the traditional sense (debugging information for developers). This is regulatory evidence that demonstrates how the system behaved, why it behaved that way, and whether that behavior was within authorized boundaries.

The cost of adding audit trails from the start is 10-15% of development effort. The cost of retrofitting them after the system is built is 50-100%. And the cost of failing a compliance review because audit trails are incomplete is the project itself.

3. Gateway-Level Enforcement

Don't rely on agent self-governance. Agents are probabilistic systems that can behave unexpectedly. Governance must be enforced at the traffic layer — the AI Gateway, the MCP Server Gateway, the orchestration framework — where policies are deterministic and auditable. The gateway scans every prompt for sensitive data. The gateway enforces token budgets. The gateway verifies agent identity and permissions. The gateway logs every transaction. The agent doesn't need to be trusted to follow policies, because the gateway enforces them regardless.

4. Human Oversight Is Non-Negotiable

For high-stakes decisions in regulated industries, humans must remain in the loop. This doesn't mean humans approve every action — that would negate the value of AI automation. It means the system is architected with an autonomy gradient: routine, low-risk decisions are fully automated; medium-risk decisions are automated with human review; high-risk decisions require human approval before execution. The gradient should be configurable, so it can be tuned as the organization gains confidence in the system's reliability.

5. Compliance Is Continuous

Governance isn't a gate you pass through once at launch. It's a continuous process. Models drift. Data distributions change. Regulations evolve. Personnel turn over. Build evaluation agents — dedicated AI systems whose sole purpose is monitoring production agents for compliance drift, performance degradation, and policy violations. These evaluation agents run automated assessments on a scheduled and event-triggered basis, flagging issues before they become incidents.

Connection to StackAhead.ai

This framework is what drives StackAhead.ai's content and architecture. Every page on this site — from the Agentic Stack to the Maturity Model to the Governance section — reflects lessons learned from building governance-first systems in regulated environments. The architectural patterns we document, the frameworks we analyze, and the recommendations we make are all grounded in the practical reality of making AI work where the stakes are highest.

When we describe Dynamic Agent Authorization, it's because I built the precursors of that pattern at the VA and the FDA. When we detail AI Gateway architectures, it's because I've implemented those patterns at Salesforce for organizations that can't afford to get them wrong. When we emphasize audit trails and human oversight, it's because I've seen what happens when organizations treat them as optional.

The EU AI Act Opportunity

I want to close with a perspective that might seem counterintuitive: for organizations that have already built governance-first architectures, the EU AI Act isn't a burden — it's a competitive advantage.

Organizations that have embedded audit trails, risk classification, human oversight, and continuous monitoring into their AI architectures from the start are already doing most of what the Act requires. Compliance for these organizations is a documentation and certification exercise, not an architectural overhaul. They can move to market faster, with lower compliance costs, and with greater confidence that their systems will pass regulatory scrutiny.

The organizations that will struggle are those that built fast and are now trying to retrofit governance. They're discovering that adding audit trails to a system that wasn't designed for them is extraordinarily expensive. They're finding that bolting human oversight onto a fully autonomous system requires fundamental architectural changes. They're learning that risk classification after the fact often reveals that the system they built doesn't meet the requirements for its risk category.

My advice, whether you're building for the EU AI Act, FDA compliance, HIPAA, or any other regulatory framework: don't wait for regulation to force you. Build governance in from day one. It's always cheaper to build it right than to fix it later. And the discipline of governance-first architecture doesn't just satisfy regulators — it produces systems that are more reliable, more trustworthy, and more valuable to the organizations that depend on them.