L1: Self-Description and Standardized I/O Formats#

Design#

The goal of ABV3-L1 (hereafter L1) is to provide the most basic layer of standardization for directories related to agent assessment. An L1-compatible repository can be managed by a common index, and — as long as its needs are not too complex — supports fully automated end-to-end execution that produces assessment results.

L1 covers four areas: self-description, dependency declaration, run-config standardization, and result-format standardization.

Self-describing repository. A single, uniform format describes the agents the repository contains, how the assessment is invoked, and so on. This lowers the mental cost of learning a new repository, and makes it possible to build a shared index for search, filtering, and management.
Standardized dependencies. A single, uniform format describes what an assessment run needs in order to execute: environment setup commands, LLM access, Docker access, and so on. Given the environment they can provide, developers can decide up front which assessments they are able to run, and prepare accordingly.
Standardized run configs. A complete agent assessment involves choosing data, choosing the target being assessed, fixing hyperparameters, and more. One full execution usually corresponds to exactly one run config, and an experiment that yields a comparable conclusion usually involves several runs. L1 provides a baseline scheme for managing run configs, which makes it convenient to:
1. ship a sample run config for smoke-testing the environment, or a representative run config for a headline experiment; and
2. ship the run configs behind each reported data point, so results can be reproduced.
Standardized result format. An experiment result is usually "some agent scored some quantitative metric on some set of data". L1 imposes a uniform format for that, which cuts design work for developers and makes leaderboard construction and cross-benchmark analysis much easier.

Spec overview#

Repository self-description#

Self-description has two parts: the agentbeats.toml file in the repository root, and information from Git itself. (For example, when agentbeats.toml does not declare a version, the Git tag is used instead.)

A basic agentbeats.toml looks like this:

toml

[abv3_repo]
name = "example_repo_name"
role = "judge"  # or "subject"
author = "example_author"
version = "0.1.0"  # semver
logo_file = "./logo.svg"  # URL, or relative to the repo root
tags = ["tag1", "tag2"]  # optional, for search and filtering

[run]
command = "uv run python main.py $RUN_CONFIG_FILE"

[run_configs]
folder = "./run_configs"  # relative to the repo root; where run configs are stored
default_exp = "default_experiment_folder_name"  # "default" if not specified

[results]
folder = "./results"

Here, run declares the command that executes the assessment.

The environment variables provided by default, and the full list of supported fields, are documented in fields.md.

Standardized dependencies#

An ABV3 repository has two broad kinds of dependency. The first kind needs a specific file, credential, or resource — supplying the right value is enough. The second kind needs a configuration step to be carried out. ABV3 lets a repository declare both in agentbeats.toml, and abv3-cli handles the hand-off in a standardized way:

   user ──(setup)──▶ abv3-cli ──(match requirements and prepare)──▶ repo
                        │                                            │
                        │  credentials, env vars, docker socket …    │
                        └────────────────────────────────────────────┘

For example:

toml

[install]
command = "uv sync"

[dependency.openai_api_key]
required = true

[dependency.docker_sock]
required = false  # optional, but if provided, enables Docker-based assessment

The dependencies most commonly declared are:

LLM access — for instance an OpenAI API key. The dependency you need varies with the kind of access required: through LiteLLM, through the Anthropic message format, through an account login, and so on. See llm_access.md.
Docker access — if the assessment needs a Docker environment, declare the docker_sock dependency. abv3-cli checks whether the current environment has Docker and provides an execution environment where the corresponding Docker commands work.

Note. If agentbeats.toml declares no dependencies, the repository is assumed to need no extra resources or configuration, and the assessment can be executed directly.

Standardized run configs#

Compared with traditional ML experiments, agent assessment tends to have longer single runs, higher cost, and a need to retain the raw process data.

By default, ABV3 provides a simple experiment-management scheme: one folder per experiment (the folder name is the experiment name), and each experiment folder holds a set of run config files (one config file per run). At assessment time you select an experiment folder, and abv3-cli executes every run config in that folder in turn, saving results to the corresponding result folder.

- run_configs/                   - results/
    - exp0/                          - exp0/
        - config1.json                   - config1/
        - config2.json        ═══▶           - result.json
        - ...                            - config2/
    - exp1/                                  - result.json
        - configx.json                   - ...
        - ...                            - results.json
                                     - exp1/
                                         - ...

During execution, the selected config file is passed to run.command through the $RUN_CONFIG_FILE environment variable, which drives the corresponding assessment task. For more on the interaction with run.command, and on the supported config formats, see run_config.md.

Note. If agentbeats.toml does not provide a run-configs folder, the repository is assumed to have exactly one default way of running an assessment, invoked through run.command.

Standardized result format#

The mapping from run config to result described above has two tiers: the result of a single run, and the result of a complete experiment. L1 requires the experiment result to follow the format; it does not require per-run results to be recorded.

For example, a typical agent assessment involves a series of tasks (data points — typically one run per data point), and each data point may carry several metrics such as accuracy, latency, or cost. L1 requires the result to record how many data points were covered plus the aggregated metrics, and to designate one main metric (the first one by default). JSON is the default encoding:

json

{
    "count": 100,
    "main_metric": "success_rate",
    "success_rate": 0.85
}

The results.json above is expected to contain at least the three fields count, main_metric, and the main metric itself. Requirements for the other fields are documented in result_format.md.

Note. If agentbeats.toml does not specify a results folder, it defaults to ./results. If no run-configs folder is specified either, the experiment name defaults to default and results are saved under ./results/default.

Checking compliance with abv3-cli#

You can use abv3-cli to check whether a repository meets the L1 standard. Clone the target repository, then from its root directory run:

shell

abv3 check -l1

abv3-cli first reads agentbeats.toml, then checks that:

every required self-description field is present and well-formed;
no dependency is declared in an unrecognized format; and
when a run-configs folder is specified, the structure inside that folder is correct.

The tool prints the check results to the terminal.

Note. abv3-cli only checks formats. It does not verify that a command is valid or that it runs successfully.

You can also use abv3-cli to configure the environment (needed only the first time), or to execute a chosen experiment:

shell

# Set up dependencies. This is interactive: it asks for any required
# credentials or resources that have not already been provided.
abv3 setup

# Run an experiment. With no experiment name, the default experiment runs.
abv3 run_exp <experiment_name>

The tool saves the results to the appropriate location and reports completion in the terminal.

Extension: providing transcripts#

For data analysis or offline training, agent assessment often needs the full record of the agent's interactions during each run — the transcript. As an extension of L1, L1+t requires a result for every run, and requires that run to ship a transcript meeting a specified set of requirements. See transcript.md.