mellea.stdlib.components.unit_test_eval

LLM Evaluation with Unit Tests in Mellea.

Classes

CLASS Message

Schema for a message in the test data.

CLASS Example

Schema for an example in the test data.

CLASS TestData

Schema for test data loaded from json.

Methods:

FUNC validate_examples

validate_examples(cls, v)

Ensure examples list is not empty.

CLASS TestBasedEval

Each TestBasedEval represents a single unit test.

Methods:

FUNC parts

parts(self) -> list[Component | CBlock]

The set of constituent parts of the Component.

FUNC format_for_llm

format_for_llm(self) -> TemplateRepresentation

Formats the test for judge evaluation.

FUNC set_judge_context

set_judge_context(self, input_text: str, prediction: str, targets_for_input: list[str])

Set context for judge evaluation.

FUNC from_json_file

from_json_file(cls, filepath: str) -> list['TestBasedEval']

Load test evaluations from json/jsonl file, return list of TestBasedEval instances, one per ‘unit test’.