This is episode 6 of our 'AI in the Public Sector' series. In the previous episode, we discussed human oversight in AI systems. This week we dive into the crucial role of procurement and contracts in AI compliance.
Before signing or renewing an AI contract, define the AI system and versions in scope, the provider and deployer roles, required documentation, logs, human oversight, incident and change notices, audit access, data terms, suspension and exit support. A contract cannot transfer statutory AI Act duties, but it can make the vendor evidence and operational support you need enforceable.14
Need to check a live vendor or tender?
Discuss your privacy or AI governance question with Zahed Ashkara. We agree the scope, deliverables and planning after intake.
The blind spot in AI procurement
The practical blind spot is often contract scope. A SaaS agreement may name a product, but not the scoring, matching, summarisation or generation features added later. A useful contract test is simple: if the supplier activates a new AI feature or changes the underlying model, must it notify you, provide updated documentation and allow a fresh risk assessment before use?
If the answer is no, the buyer may lose control over classification, instructions, monitoring and evidence. The updated EU model contractual AI clauses address this problem by defining the system, documentation, changes, audit access and cooperation that public buyers can tailor to their procurement.4
The AI Act and chain responsibility
The EU AI Act assigns duties according to the role each party actually performs. When high-risk obligations apply, deployers must use the system according to its instructions, assign effective human oversight, monitor operation and retain logs that are under their control. Providers have separate duties around system design, documentation, conformity and post-market monitoring.15
A contract cannot rewrite those statutory roles. It can require the supplier to provide the instructions, logs, version notices, incident assistance and technical access that the deployer needs to perform its own duties. Procurement is therefore where legal obligations become testable delivery requirements.
Which requirements belong in your contract by default?
Effective AI contracts go beyond standard delivery conditions. The exact clauses depend on role, classification and use context, but the following controls are a defensible starting point:
Explainability and transparency
Require clear instructions, intended purpose, limitations, input requirements, expected performance and the information needed for human oversight. Do not demand a universal explanation format for every AI system. Specify the explanation or traceability evidence that is necessary for the actual decision and affected user.
Bias and performance monitoring
Define the metrics, logs and review frequency that fit the use case. This may include accuracy, error rates, subgroup performance, drift and serious-incident indicators. Require evidence in an agreed format and enough access to verify it, while respecting privacy, security and intellectual-property constraints.
Mitigation options and correction possibilities
Specify who can review, override, pause or escalate an output and what information that person receives. Human oversight must work in the real workflow. Include training, permissions, response times and a route for correcting outcomes or input data where that is technically and legally appropriate.
Kill switch and shadow mode
Agree when the buyer may suspend the AI function, what the supplier must do during an incident and how service continues safely. For material model or feature changes, require advance notice, release notes, testing evidence and, where proportionate, a sandbox or shadow test before production use.
Data governance and quality requirements
Describe which data categories, sources, quality controls, retention periods, subprocessors and update procedures the supplier must document. Distinguish AI Act evidence from GDPR, security and confidentiality terms. The contract should make those layers work together without pretending they are the same obligation.
Audit rights and reporting
Define what may be verified, by whom, how often and which evidence the supplier must provide. Include assistance with your AI register, regulatory questions and contract exit. Exit support should cover data return or deletion, export of relevant logs and records, and continuity when the AI feature is disabled.24
How do you incorporate this into your tender?
A successful AI tender starts with thorough preparation and clear requirements. The traditional approach of functional specifications is insufficient for AI systems that are inherently more complex and less predictable.
AI questionnaire and market exploration
Start with an AI questionnaire during market exploration: which AI functionalities are included, how are they secured, what is the chain of (sub)suppliers? This phase is crucial to understand what the market can offer and where the risks lie.
Important questions are: Which AI technologies are used? How is bias prevented and monitored? What data is used for training? How is explainability ensured? What certifications does the supplier have? Who are the sub-suppliers in the AI chain?
Program of requirements and award criteria
Anchor the relevant AI requirements in your specification, acceptance tests and award criteria. Distinguish mandatory evidence from features that add value. Avoid a generic requirement that a vendor must be "AI Act compliant". Ask for the documents, controls and test results that let you verify the claim for the role and system in scope.
Model documents and standardization
Add model documents such as an AI compliance annex where these conditions are standardized. This prevents having to reinvent the wheel with each tender and ensures consistency within your organization.
Develop templates for AI contract clauses, checklists for AI assessments, and standard reporting formats. This makes the process more efficient and increases the quality of your contracts.
Expertise in the assessment committee
Include legal, procurement, security, technical and end-user expertise in the assessment team when the system's impact warrants it. The team should verify vendor claims against documents, demonstrations and acceptance tests, not only presentation slides.
Practical acceptance test for an AI contract
Use an acceptance scenario before award or renewal. For a system that produces a score or recommendation, ask the supplier to demonstrate three things with the exact production configuration:
- which model and version produced the output;
- which instructions, limits and human-review controls apply;
- which logs, performance evidence and incident route the buyer receives.
Then test a material update. Ask what happens if the supplier changes the model, adds a new data source or alters the scoring logic. The contract should state whether notice, renewed testing, documentation and buyer approval are required before the change reaches production.
The pass condition is not a polished demo. It is reproducible evidence that matches the clauses, acceptance criteria and operating process.
Who owns the contract controls?
- Legal and privacy confirm the actual provider and deployer roles, data terms and required notices.
- Procurement turns those requirements into deliverables, acceptance criteria, change control and remedies.
- IT and security verify integration, access, logging, incident response and continuity.
- The process owner defines intended use, human oversight, performance thresholds and the decision to accept or suspend the system.
Record one owner for every contract control. A clause without an internal owner, evidence source and review moment is difficult to operate.
A 30-minute procurement action list
- List the AI-enabled products and features in the contract scope.
- Ask the supplier for current system documentation, model or feature change history and available logs.
- Compare the draft contract with the EU model contractual AI clauses.4
- Mark which controls are mandatory for the system's likely role, classification and use context.
- Put the three largest evidence gaps into the negotiation or renewal plan.
The strategic value of proactive AI contracting
Good AI contracting turns legal and operational requirements into evidence that can be tested before go-live. Clear scope, change control, acceptance tests and exit support also make renewals and incident response easier to manage.
The goal is not a contract that merely repeats the AI Act. It is a contract that tells both parties what must be delivered, how it will be verified and what happens when the system or its risk profile changes.
Frequently asked questions
Which clauses should an AI vendor contract include?
Define the AI system and versions in scope, provider and deployer roles, required instructions and documentation, logging and performance evidence, human oversight, incident and change notices, audit access, data and subprocessor terms, suspension and exit support. Tailor each clause to the system's classification and use context.
Can a contract transfer EU AI Act duties to the vendor?
No. Statutory duties follow the role each party actually performs. A contract can allocate tasks, evidence delivery, assistance and remedies, but it cannot turn a deployer into a compliant organization merely by assigning every duty to the supplier.
Do all AI systems need the same contractual clauses?
No. Requirements should be proportionate to the system, role, risk classification and use context. The EU model contractual AI clauses therefore include a fuller high-risk version and a lighter version for non-high-risk AI.4
What does an AI vendor review by Embed AI cost?
The required support depends on your role, systems, suppliers and open decisions. We agree scope and commercial terms for consultancy after intake.
Where should a procurement team start?
Start with the free 7-question AI Act quickscan. If a live tender, renewal or supplier claim needs review, continue to the AI vendor contract check with the system, draft contract and available vendor evidence.
In episode 7 of our series, we dive into the registration and transparency process: how and where do you record which models you use and what they do? From EU database to the Dutch algorithm register.
If you are already reviewing AI vendors, start with the free AI Act quickscan. If a live contract needs review, continue to Embed AI's AI Act gap intake or read about the AI vendor contract check.
Sources
What an intake sheet looks like
Short fictional example. Supplier X ranks applicants for an initial selection. The supplier is new; documentation and a contract are missing.
Known: ordinary personal data, EU processing according to the requester, influence on a selection decision. A recruiter checks the output.
Missing evidence: documentation on operation and limitations, plus contract terms. The stated facts have not been checked.
- Procurement: Request documentation on intended use and limitations.
- Privacy officer or DPO: Assess the influence on selection and request the supplier’s reasoning.
- Privacy officer or DPO: Assess whether a DPIA is needed before use.
Provisional direction: closer attention to selection; assess classification and whether a DPIA is needed. This is not a final legal conclusion.
Open decision: What evidence is needed before we allow a trial with applicant data? Who decides: HR manager.
