% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/loco.roo.R
\name{loco.roo}
\alias{loco.roo}
\title{Variable importance via excess test error}
\usage{
loco.roo(
  x,
  y,
  train.fun,
  predict.fun,
  vars = 0,
  alpha = 0.1,
  mad.train.fun = NULL,
  mad.predict.fun = NULL,
  split = NULL,
  seed = NULL,
  out.roo = NULL,
  verbose = FALSE
)
}
\arguments{
\item{x}{Matrix of variables, of dimension (say) n x p.}

\item{y}{Vector of responses, of length (say) n.}

\item{train.fun}{A function to perform model training, i.e., to produce an
estimator of E(Y|X), the conditional expectation of the response variable
Y given features X. Its input arguments should be x: matrix of features,
and y: vector of responses.}

\item{predict.fun}{A function to perform prediction for the (mean of the)
responses at new feature values. Its input arguments should be out: output
produced by train.fun, and newx: feature values at which we want to make
predictions.}

\item{vars}{A list specifying the variables (indices between 1 and p) to
for which variable importance should be investigated. Alternatively, if
set equal to 0, the default, then all variables are investigated.}

\item{alpha}{Miscoverage level for the prediction intervals, i.e., intervals
with coverage 1-alpha are formed. Default for alpha is 0.1.}

\item{mad.train.fun}{A function to perform training on the absolute residuals
i.e., to produce an estimator of E(R|X) where R is the absolute residual
R = |Y - m(X)|, and m denotes the estimator produced by train.fun.
This is used to scale the conformal score, to produce a prediction interval
with varying local width. The input arguments to mad.train.fun should be
x: matrix of features, and y: vector of absolute residuals. The default for
mad.train.fun is NULL, which means that no training is done on the absolute
residuals, and the usual (unscaled) conformal score is used. Note that if
mad.train.fun is non-NULL, then so must be mad.predict.fun (see next).}

\item{mad.predict.fun}{A function to perform prediction for the (mean of the)
absolute residuals at new feature values. Its input arguments should be
out: output produced by mad.train.fun, and newx: feature values at which we
want to make predictions. The default for mad.predict.fun is NULL, which
means that no local scaling is done for the conformal score, i.e., the
usual (unscaled) conformal score is used.}

\item{split}{Indices that define the data-split to be used (i.e., the indices
define the first half of the data-split). Note that this split is common
both to conformal.pred.roo (whether already run, or to be run internally,
see below) and the excess error intervals computed by this function.
Default is NULL, in which case the split is chosen randomly.}

\item{seed}{Integer to be passed to set.seed before defining the random 
data-split to be used. Default is NULL, which effectively sets no seed.
If both split and seed are passed, the former takes priority and the latter
is ignored.}

\item{out.roo}{Output from running conformal.pred.roo on the given data set;
default is NULL, which means that conformal.pred.roo will be run internally
in order to compute in-sample prediction intervals. If out.roo is NULL,
then the data-split common to conformal.pred.roo and this function will be
determined by the split or seed arguments. If non-NULL, then the common
data-split will be determined by that recorded in out.roo, and the split
and seed arguments will be (silently) ignored.}

\item{verbose}{Should intermediate progress be printed out? Default is FALSE.}
}
\value{
A list with the following components: lo, up, vars, split,
  out.roo. The third component is a list of variables that were tested (i.e.,
  dropped one at a time, to compute excess error). The first two are arrays
  The indices used for the half of the data-split are
  returned in split. Finally, for convenience, the output from running
  conformal.pred.roo is stored in out.roo.
}
\description{
Compute prediction intervals for the excess test error due to dropping a
  variable, measured in-sample and having valid in-sample coverage
}
\details{
For concreteness, suppose that we want to use the predictions from
  forward stepwise regression at steps 1 through 5 in the path. In this case,
  there are m = 5 internal tuning parameter values to predict.fun, in the
  notation used above, and each of the returned matrices pred, lo, up, fit
  will have 5 columns (one for each step of the forward stepwise path).
  The code is structured in this way so that we may defined a single pair of
  functions train.fun and predict.fun, over a set of m = 5 tuning parameter
  values, instead of calling the conformal function separately m = 5 times.
}
\examples{
## Use lasso + CV to choose a model, then test variable importance in the
## selected model

# Generate some example training data
set.seed(11)
n = 200; p = 500; s = 10
x = matrix(rnorm(n*p),n,p)
beta = 8*c(rnorm(s),rep(0,p-s)) 
y = x \%*\% beta + rnorm(n)

# Lasso training and prediction functions, using cross-validation over 100
# values of lambda, with the 1se rule
funs = lasso.funs(nlambda=100,cv=TRUE,cv.rule="1se")

# In-sample rank-one-out conformal inference 
out.roo = conformal.pred.roo(x, y, alpha=0.1,
  train.fun=funs$train, predict.fun=funs$predict)

# Look at which variables were nonzero in the lasso + CV model fit to the whole
# data set
out.all = out.roo$out.all
vars = which(coef(out.all$glmnet.fit, s=out.all$lambda.1se) != 0)

# Look at error inflation due to variable dropping, among vars
out.loco = loco.roo(x, y, alpha=0.1, vars=vars,
  train.fun=funs$train, predict.fun=funs$predict, verbose=TRUE)

# Compute "c-values", the proportion of intervals that have a left endpoint
# less than or equal to 0
cvals = colMeans(out.loco$lo[,,1] <= 0)
names(cvals) = vars
cvals

xlab = "Sample number"
ylab = "Interval"

# For each dropped variable in consideration, plot the intervals
for (j in 1:length(vars)) {
  plot(c(),c(),xlim=c(1,n),ylim=range(c(out.loco$lo[,j,1],out.loco$up[,j,1])),
       xlab=xlab,ylab=ylab,main=sprintf("Variable \%i, c-value = \%0.3f",
                             vars[j], cvals[j]))
  cols = ifelse(out.loco$lo[,j,1] <= 0, 1, 3)
  segments(1:n,out.loco$lo[,j,1],1:n,out.loco$up[,j,1],col=cols)
  abline(h=0, lty=2, lwd=2, col=2)
}
}
\references{
"Distribution-Free Predictive Inference for Regression" by Lei,
  G'Sell, Rinaldo, Tibshirani, Wasserman (2018).
}
