Integrate a classification-accuracy test into AISC

Step by step: add a classification-accuracy test

Suppose we have a CSV file containing the expected class and the class predicted by a model. We want to add a test that computes classification accuracy: the proportion of rows for which both labels match.

A plug-in is made of three blocks

AISC discovers a plug-in by importing its package and registering every class that inherits from BaseEvaluationPlugin and is exported from __init__.py. Each such class combines three blocks:

• Configuration: a Pydantic model (AccuracyConfig) declaring the parameters shown in the evaluation form, here the names of the expected-label and predicted-label columns. It is passed as the type parameter of the plug-in class.
• Input(s): one @evaluation_input decorator per input slot, each backed by an input provider that parses the uploaded bytes into usable data. This test needs a single CSV dataset, parsed by the ready-made CsvInputProvider; a plug-in can just as well declare several inputs.
• Plug-in class: the BaseEvaluationPlugin subclass containing plugin_name (its display name), evaluate() (validates the config, reads the inputs, computes the result and returns a dict) and one @metric method per measurement (a read-only accessor that converts that dict returned by evaluate() into standard Measure objects).

The steps below implement each of these blocks in turn, starting with the package that will hold them.

1. 1. Create the packageCreate a separate directory for local plug-ins, enforce Python 3.12, initialise the library with that version and add the AISC plug-in interface as a dependency.

mkdir -p /absolute/path/to/aisc-plugins
cd /absolute/path/to/aisc-plugins
uv python install 3.12
uv init --lib --python 3.12 accuracy-plugin
cd accuracy-plugin
uv add aisc-plugin-interface

The plug-in interface requires Python 3.12 or later. uv python install 3.12 guarantees an interpreter is available, and passing --python 3.12 to uv init pins the version (requires-python = ">=3.12"), preventing uv from selecting an older system interpreter. The command creates a package named accuracy-plugin with a Python module named accuracy_plugin; AISC uses its name and version in pyproject.toml for discovery. The interface is published on PyPI as aisc-plugin-interface, so uv add pulls it from there.

2. 2. Implement the testIn src/accuracy_plugin/plugin.py, define the column names as typed configuration, declare the CSV input, calculate the accuracy and export it as a standard AISC measurement. Keep heavy imports out of the module top-level.

# src/accuracy_plugin/plugin.py
# Module-level imports: only the interface and pydantic. Heavy dependencies
# (pandas, the target eval library, ...) are imported lazily, inside method
# bodies - the module stays importable without them installed.
from pydantic import BaseModel, Field

from aisc_plugin_interface import (
    BaseEvaluationPlugin,
    InputType,
    Measure,
    evaluation_input,
    metric,
)
from aisc_plugin_interface.input_providers.csv_input_provider import (
    CsvInputProvider,
)


# Block 1 - Configuration: the fields the evaluation form asks the user for.
class AccuracyConfig(BaseModel):
    expected_label: str = Field(
        description="Column containing the expected class"
    )
    predicted_label: str = Field(
        description="Column containing the predicted class"
    )


# Block 2 - Input: one decorator per input slot; CsvInputProvider parses the
# uploaded file into rows.
@evaluation_input(
    name="predictions",
    label="Prediction dataset",
    input_provider_class=CsvInputProvider,
    input_type=InputType.DATASET,
)
# Block 3 - Plugin class: evaluation and metrics live here.
class ClassificationAccuracyPlugin(
    BaseEvaluationPlugin[AccuracyConfig]
):
    plugin_name = "Classification accuracy"  # display name in AISC

    # Evaluate: validate the config, read the input, compute, return a dict.
    def evaluate(self, config_data: dict):
        # Heavy dependencies are imported inside the method body so the
        # module stays importable without them installed.
        config = self.validate_config_form_data(config_data)
        rows = self.get_input_data("predictions") or []
        if not rows:
            raise ValueError("The prediction dataset is empty")

        correct = sum(
            row[config.expected_label]
            == row[config.predicted_label]
            for row in rows
        )
        return {"accuracy": correct / len(rows)}

    # Metric: read-only accessor that translates the dict returned by
    # evaluate() into standard Measure objects.
    @metric("Classification accuracy")
    def accuracy_metric(self, result: dict) -> list[Measure]:
        if not result:
            return []
        return [
            Measure(
                name="Classification accuracy",
                score=result["accuracy"],
                unit="ratio",
            )
        ]

The configuration model tells AISC which parameters to request from the user. The input decorator tells it that the test needs a CSV dataset. The evaluate method contains the test itself, while @metric translates its result into the shared representation used by the platform. “Heavy imports” matter because AISC discovers a plug-in by importing its module: anything imported at the top of plugin.py - numpy, pandas, the target evaluation library - is imported in an environment where the plug-in’s own dependencies are not installed, and one missing import silently prevents registration. This minimal example has no such dependency, but a real one imports them inside method bodies (or behind if TYPE_CHECKING:), never at module level.

3. 3. Export the classExpose the plug-in from src/accuracy_plugin/__init__.py. The loader imports this module and registers each exported subclass of BaseEvaluationPlugin.

# src/accuracy_plugin/__init__.py
from .plugin import ClassificationAccuracyPlugin

__all__ = ["ClassificationAccuracyPlugin"]

4. 4. Test itAdd pytest as a dev dependency and write a small unit-test suite that feeds a CSV through the plug-in and checks the exported measurement.

uv add --dev pytest

# tests/test_plugin.py
import sys

sys.path.insert(0, "src")

from accuracy_plugin import ClassificationAccuracyPlugin


def test_accuracy_computed_from_csv():
    plugin = ClassificationAccuracyPlugin()
    plugin.set_input_content(
        "predictions",
        b"expected,predicted\n0,0\n1,1\n1,0\n",
    )
    result = plugin.evaluate(
        {"expected_label": "expected", "predicted_label": "predicted"}
    )
    measures = plugin.export_metrics(result)
    assert [m.score for m in measures] == [2 / 3]
    assert measures[0].unit == "ratio"

Run the suite before connecting the package to AISC:

uv run python -m pytest -q tests

5. 5. Get AISCClone the main project with its submodules. This provides the web application, backend, evaluation worker, plug-in manager and local development environment.

git clone --recursive https://github.com/lux-ai-factory/aisc.git
cd aisc

6. 6. Register the folderIn AISC’s env.development, set PLUGIN_PATH to the absolute directory that contains accuracy-plugin.

PLUGIN_PATH=/absolute/path/to/aisc-plugins

7. 7. Start AISCLaunch the infrastructure and applications with the development configuration. The plug-in directory is mounted into both the backend and evaluation worker, and the package is discovered automatically.

docker compose --env-file env.development \
  -f docker-compose-infra.development.yml \
  -f docker-compose.development.yml up

8. 8. Verify discovery and lazy importsFrom the plug-in folder (PLUGIN_PATH), load the package through the AISC plug-in manager in an isolated environment. This proves the plug-in is discoverable and imports cleanly without its dependencies installed - the guarantee that lazy imports exist for.

cd /absolute/path/to/aisc-plugins
uv run --isolated \
  --with git+https://github.com/lux-ai-factory/aisc-plugin-manager.git \
  python -c "
from aisc_plugin_manager import Loader
loader = Loader('/absolute/path/to/aisc-plugins', '', '', '', '')
packages = loader.list_packages(refresh=True)
assert 'accuracy_plugin' in packages, f'plugin not discovered: {packages}'
print('Discovered:', packages)
"

Loader takes the parent directory of the plug-in project - here, PLUGIN_PATH itself. If the assertion fails, the most likely culprit is a top-level import of a dependency that is not installed in the fresh environment; move it inside a method body.

9. 9. Run the testOpen AISC, select “Classification accuracy”, upload a CSV, enter the expected-label and predicted-label columns, and launch the evaluation. The resulting accuracy is stored and displayed like the measurements produced by any other plug-in.

Alternative: two evaluation inputs

This minimal example expects both labels in a single CSV. It could equally have been written to take two separate evaluation inputs - one CSV (or npy/npz) of predictions and one CSV (or npy/npz) of targets - by stacking two @evaluation_input decorators on the class:

@evaluation_input(
    name="predictions",
    label="Prediction dataset",
    input_provider_class=CsvInputProvider,
    input_type=InputType.DATASET,
)
@evaluation_input(
    name="targets",
    label="Target dataset",
    input_provider_class=CsvInputProvider,
    input_type=InputType.DATASET,
)
class ClassificationAccuracyPlugin(BaseEvaluationPlugin[AccuracyConfig]):
    def evaluate(self, config_data: dict):
        config = self.validate_config_form_data(config_data)
        predictions = self.get_input_data("predictions") or []
        targets = self.get_input_data("targets") or []
        # ... pair rows by index and compare prediction against target

Each decorator declares one input slot; inside evaluate you read them by name with get_input_data(...). A production plug-in can also accept a model, perform inference, expose several metrics, create visualisations, report progress and upload result artefacts. The performance-evaluation plug-in provides a complete example covering classification, regression, ONNX models and multiple input formats.

Resources