What does it take to build a technical AI assessment sandbox?

The missing infrastructure for technical AI assessment

Technical assessment asks concrete questions about an AI system: How well does it perform under realistic conditions? Is it robust to attacks or distribution shifts? Does it expose sensitive information? Can its behaviour be explained? A technical AI assessment sandbox provides the controlled environment, tools and evidence needed to investigate such questions.

This is the necessary complement to AI regulatory sandboxes. A regulatory sandbox is a broader supervised process in which providers interact with competent authorities and other experts to explore regulatory requirements and compliance before market deployment. A technical assessment sandbox can supply part of the infrastructure and evidence used within that process, but it can equally support internal self-assessment, independent technical evaluation or conformity-assessment activities.

Building this technical infrastructure creates two connected problems. First, AI assessment tools are fragmented. A robustness benchmark, a privacy check and an explainability tool may each use different inputs, produce different outputs and document results differently. Repeating the same assessment in another organisation—or even six months later—can become difficult.

Second, there can be no universal assessment recipe. A large language model processing legal documents does not expose the same risks as a computer-vision system in a vehicle or a tabular model supporting a financial decision. They operate under different requirements, process different data modalities and rely on different model families. Their evaluation procedures must therefore differ too: from the controls and datasets selected to the metrics, attack scenarios and expertise needed to interpret the evidence.

The challenge is not to create one more test or one fixed checklist. It is to provide technical infrastructure that can assemble a unique assessment procedure for each AI system, execute it in a controlled environment and preserve enough common structure to make the resulting evidence usable by engineers, domain experts, auditors and decision-makers.

Each AI system needs its own technical assessment procedure. The sandbox should make that tailored evaluation reproducible, traceable and reusable.

Configure then assess

Our answer is the AI Assessment Sandbox Configurator, an open-source framework developed within the Luxembourg AI Factory. Rather than delivering a pre-defined battery of tests, it helps experts compose a technical assessment around the system at hand. Its central design choice is to separate that configuration work from the execution of the resulting technical sandbox.

This “sandbox à la carte” approach preserves an essential distinction: the procedure is unique to each system, while the infrastructure is shared. A credit-risk model, a multilingual assistant and an autonomous-driving component do not require the same evidence, but the way their assessments are configured, documented and made reproducible can follow common principles.

Integrate your own test in the plugin architecture

The framework is deliberately tool-agnostic. Individual testing tools can evolve without forcing the whole platform to be rebuilt. A plug-in interface connects them to a shared catalogue, while a common data model harmonises their inputs, results and metadata.

How third-party tests integrate with AISC

The plug-in architecture is designed so that third-party developers can implement and integrate their own evaluation methods without modifying the core platform. In practice, each plug-in is an independent Python package containing one or more evaluation classes. The package depends on the AISC plug-in interface, while the platform’s plug-in manager discovers, loads and executes it.

Each class inherits from BaseEvaluationPlugin and provides five elements:

1. ConfigurationA Pydantic model defines the parameters, defaults and validation rules. The platform turns this model into the configuration form shown to the user.
2. InputsThe plug-in declares the datasets or models it expects with @evaluation_input. Built-in or custom input providers translate uploaded files into the objects used by the test.
3. EvaluationThe evaluate method validates the configuration, retrieves the parsed inputs and runs the actual assessment logic.
4. MeasurementsMethods marked with @metric convert the evaluation output into standard Measure objects that the platform can store, compare and visualise.
5. FeedbackLong-running tests can report progress and upload artefacts such as predictions, detailed logs or supplementary result files.

What the platform cannot automate

The Configurator organises an assessment; it does not decide what “safe enough” means. Experts must still determine which risks matter, whether a dataset is representative, how findings interact with a specific legal or operational context and which residual risks are acceptable.

The framework is also still evolving. The initial pilot exercised its harmonisation and reporting capabilities, not the complete automated workflow. Catalogue governance, full audit coverage, portability across infrastructures and alignment with future harmonised standards remain active research and engineering questions.

That is precisely why an open architecture matters. Technical assessment sandboxes will encounter new models, sectors and risks continuously. Their infrastructure must support learning across assessments without freezing innovation into a single checklist.

The AI Assessment Sandbox Configurator is available under the Apache 2.0 licence.

Contribute to the project

Resources