armlet.eval.evaluators module#

class armlet.eval.evaluators.MultiCriteriaBinaryClassEval(eval_every: int, n_classes: int, sensitive_attributes: list[str], metrics: dict[str, dict[str, Metric]] = {})#

Bases: ClassificationEval

evaluate(round: int, model: Module, eval_data_loader: FastDataLoader | Iterable[FastDataLoader], loss_fn: Module | None = None, additional_metrics: dict[str, Metric] | None = None, device: device = device(type='cpu')) → dict#

Evaluate the model. The metrics computed are accuracy, precision, recall, f1 and the loss according to the provided loss function loss_fn. Metrics are computed both in a micro and macro fashion.

Warning

The loss function loss_fn should be defined on the same device as the model. Moreover, it is assumed that the only arguments of the loss function are the predicted values and the true values.

Parameters:

  • round (int) – The current round.

  • model (torch.nn.Module) – The model to evaluate. If None, the method returns an empty dictionary.

  • eval_data_loader (Union[FastDataLoader, Collection[FastDataLoader]]) – The data loader(s) to use for evaluation. If None, the method returns an empty dictionary.

  • loss_fn (torch.nn.Module, optional) – The loss function to use for evaluation.

  • additional_metrics (dict[str, Metric], optional) – Additional metrics to use for evaluation. If provided, they are added to the default metrics.

  • device (torch.device, optional) – The device to use for evaluation. Defaults to “cpu”.

Returns:

A dictionary containing the computed metrics.

Return type:

dict