% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/benchmark.R
\name{benchmark}
\alias{benchmark}
\title{Benchmark Multiple Learners on Multiple Tasks}
\usage{
benchmark(
  design,
  store_models = FALSE,
  store_backends = TRUE,
  encapsulate = NA_character_,
  allow_hotstart = FALSE,
  clone = c("task", "learner", "resampling"),
  unmarshal = TRUE
)
}
\arguments{
\item{design}{(\code{\link[=data.frame]{data.frame()}})\cr
Data frame (or \code{\link[data.table:data.table]{data.table::data.table()}}) with three columns: "task", "learner", and "resampling".
Each row defines a resampling by providing a \link{Task}, \link{Learner} and an instantiated \link{Resampling} strategy.
The helper function \code{\link[=benchmark_grid]{benchmark_grid()}} can assist in generating an exhaustive design (see examples) and
instantiate the \link{Resampling}s per \link{Task}.
Additionally, you can set the additional column 'param_values', see \code{\link[=benchmark_grid]{benchmark_grid()}}.}

\item{store_models}{(\code{logical(1)})\cr
Store the fitted model in the resulting object=
Set to \code{TRUE} if you want to further analyse the models or want to
extract information like variable importance.}

\item{store_backends}{(\code{logical(1)})\cr
Keep the \link{DataBackend} of the \link{Task} in the \link{ResampleResult}?
Set to \code{TRUE} if your performance measures require a \link{Task},
or to analyse results more conveniently.
Set to \code{FALSE} to reduce the file size and memory footprint
after serialization.
The current default is \code{TRUE}, but this eventually will be changed
in a future release.}

\item{encapsulate}{(\code{character(1)})\cr
If not \code{NA}, enables encapsulation by setting the field
\code{Learner$encapsulate} to one of the supported values:
\code{"none"} (disable encapsulation),
\code{"try"} (captures errors but output is printed to the console and not logged),
\code{"evaluate"} (execute via \CRANpkg{evaluate}) and
\code{"callr"} (start in external session via \CRANpkg{callr}).
If \code{NA}, encapsulation is not changed, i.e. the settings of the
individual learner are active.
Additionally, if encapsulation is set to \code{"evaluate"} or \code{"callr"},
the fallback learner is set to the featureless learner if the learner
does not already have a fallback configured.}

\item{allow_hotstart}{(\code{logical(1)})\cr
Determines if learner(s) are hot started with trained models in
\verb{$hotstart_stack}. See also \link{HotstartStack}.}

\item{clone}{(\code{character()})\cr
Select the input objects to be cloned before proceeding by
providing a set with possible values \code{"task"}, \code{"learner"} and
\code{"resampling"} for \link{Task}, \link{Learner} and \link{Resampling}, respectively.
Per default, all input objects are cloned.}

\item{unmarshal}{\code{\link{Learner}}\cr
Whether to unmarshal learners that were marshaled during the execution.
If \code{TRUE} all models are stored in unmarshaled form.
If \code{FALSE}, all learners (that need marshaling) are stored in marshaled form.}
}
\value{
\link{BenchmarkResult}.
}
\description{
Runs a benchmark on arbitrary combinations of tasks (\link{Task}), learners (\link{Learner}), and resampling strategies (\link{Resampling}), possibly in parallel.

For large-scale benchmarking we recommend to use the \CRANpkg{mlr3batchmark} package.
This package runs benchmark experiments on high-performance computing clusters and handles failed experiments.
}
\note{
The fitted models are discarded after the predictions have been scored in order to reduce memory consumption.
If you need access to the models for later analysis, set \code{store_models} to \code{TRUE}.
}
\section{Predict Sets}{

If you want to compare the performance of a learner on the training with the performance
on the test set, you have to configure the \link{Learner} to predict on multiple sets by
setting the field \code{predict_sets} to \code{c("train", "test")} (default is \code{"test"}).
Each set yields a separate \link{Prediction} object during resampling.
In the next step, you have to configure the measures to operate on the respective Prediction object:

\if{html}{\out{<div class="sourceCode">}}\preformatted{m1 = msr("classif.ce", id = "ce.train", predict_sets = "train")
m2 = msr("classif.ce", id = "ce.test", predict_sets = "test")
}\if{html}{\out{</div>}}

The (list of) created measures can finally be passed to \verb{$aggregate()} or \verb{$score()}.
}

\section{Parallelization}{


This function can be parallelized with the \CRANpkg{future} package.
One job is one resampling iteration, and all jobs are send to an apply function
from \CRANpkg{future.apply} in a single batch.
To select a parallel backend, use \code{\link[future:plan]{future::plan()}}.
More on parallelization can be found in the book:
\url{https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html}
}

\section{Progress Bars}{

This function supports progress bars via the package \CRANpkg{progressr}.
Simply wrap the function call in \code{\link[progressr:with_progress]{progressr::with_progress()}} to enable them.
Alternatively, call \code{\link[progressr:handlers]{progressr::handlers()}} with \code{global = TRUE} to enable progress bars
globally.
We recommend the \CRANpkg{progress} package as backend which can be enabled with
\code{progressr::handlers("progress")}.
}

\section{Logging}{


The \CRANpkg{mlr3} uses the \CRANpkg{lgr} package for logging.
\CRANpkg{lgr} supports multiple log levels which can be queried with
\code{getOption("lgr.log_levels")}.

To suppress output and reduce verbosity, you can lower the log from the
default level \code{"info"} to \code{"warn"}:

\if{html}{\out{<div class="sourceCode">}}\preformatted{lgr::get_logger("mlr3")$set_threshold("warn")
}\if{html}{\out{</div>}}

To get additional log output for debugging, increase the log level to \code{"debug"}
or \code{"trace"}:

\if{html}{\out{<div class="sourceCode">}}\preformatted{lgr::get_logger("mlr3")$set_threshold("debug")
}\if{html}{\out{</div>}}

To log to a file or a data base, see the documentation of \link[lgr:lgr-package]{lgr::lgr-package}.
}

\examples{
# benchmarking with benchmark_grid()
tasks = lapply(c("penguins", "sonar"), tsk)
learners = lapply(c("classif.featureless", "classif.rpart"), lrn)
resamplings = rsmp("cv", folds = 3)

design = benchmark_grid(tasks, learners, resamplings)
print(design)

set.seed(123)
bmr = benchmark(design)

## Data of all resamplings
head(as.data.table(bmr))

## Aggregated performance values
aggr = bmr$aggregate()
print(aggr)

## Extract predictions of first resampling result
rr = aggr$resample_result[[1]]
as.data.table(rr$prediction())

# Benchmarking with a custom design:
# - fit classif.featureless on penguins with a 3-fold CV
# - fit classif.rpart on sonar using a holdout
tasks = list(tsk("penguins"), tsk("sonar"))
learners = list(lrn("classif.featureless"), lrn("classif.rpart"))
resamplings = list(rsmp("cv", folds = 3), rsmp("holdout"))

design = data.table::data.table(
  task = tasks,
  learner = learners,
  resampling = resamplings
)

## Instantiate resamplings
design$resampling = Map(
  function(task, resampling) resampling$clone()$instantiate(task),
  task = design$task, resampling = design$resampling
)

## Run benchmark
bmr = benchmark(design)
print(bmr)

## Get the training set of the 2nd iteration of the featureless learner on penguins
rr = bmr$aggregate()[learner_id == "classif.featureless"]$resample_result[[1]]
rr$resampling$train_set(2)
}
\seealso{
\itemize{
\item Chapter in the \href{https://mlr3book.mlr-org.com/}{mlr3book}:
\url{https://mlr3book.mlr-org.com/chapters/chapter3/evaluation_and_benchmarking.html#sec-benchmarking}
\item Package \CRANpkg{mlr3viz} for some generic visualizations.
\item \CRANpkg{mlr3benchmark} for post-hoc analysis of benchmark results.
}

Other benchmark: 
\code{\link{BenchmarkResult}},
\code{\link{benchmark_grid}()}
}
\concept{benchmark}
