{"word_count": 1074, "character_count": 10299}
Tuesday, at the end of my 5-hour crash course on machine learning for actuaries, Pierre asked me an interesting question about computational time of different techniques. I've been presenting the philosophy of various algorithm, but I forgot to mention computational time. I wanted to try several classification algorithms on the dataset used to illustrate the techniques&#62; rm(list=ls())&#62; myocarde=read.table("http://freakonometrics.free.fr/myocarde.csv",head=TRUE,sep=";")&#62; levels(myocarde$PRONO)=c("Death","Survival")But the dataset is rather small, with 71 observations and 7 explanatory variables. So I decided to replicate the observations, and to add some covariates,&#62; levels(myocarde$PRONO)=c("Death","Survival")&#62; idx=rep(1:nrow(myocarde),each=100)&#62; TPS=matrix(NA,30,10)&#62; myocarde_large=myocarde[idx,]&#62; k=23&#62; M=data.frame(matrix(rnorm(k*+ nrow(myocarde_large)),nrow(myocarde_large),k))&#62; names(M)=paste("X",1:k,sep="")&#62; myocarde_large=cbind(myocarde_large,M)&#62; dim(myocarde_large)[1] 7100   31&#62; object.size(myocarde_large)2049.064 kbytesThe dataset is not big... but at least, it does not take 0.0001 sec. to run a regression.  Actually, to run a logistic regression, it takes 0.1 second&#62; system.time(fit&#60; glm(PRONO~.,+ data=myocarde_large, family="binomial"))       user      system     elapsed       0.114       0.016       0.134 &#62; object.size(fit)9,313.600 kbytesAnd I was surprised that the regression object was 9Mo, which is more than four times the size of the dataset. With a large dataset, 100 times larger,&#62; dim(myocarde_large_2)[1] 710000     31it takes 20 sec.&#62; system.time(fit&#60;-glm(PRONO~.,+ data=myocarde_large_2, family="binomial"))utilisateur     système      écoulé      16.394       2.576      19.819 &#62; object.size(fit)90,9025.600 kbytesand the object is 'only' ten times bigger.Note that with a spline, computational time is rather similar&#62; library(splines)&#62; system.time(fit&#60;-glm(PRONO~bs(INSYS)+.,+ data=myocarde_large, family="binomial"))       user      system     elapsed       0.142       0.000       0.143 &#62; object.size(fit)9663.856 kbytesIf we use another function, more specifically the one I use for multinomial regressions, it is two times longer&#62; library(VGAM)&#62; system.time(fit1&#60;-vglm(PRONO~.,+ data=myocarde_large, family="multinomial"))       user      system     elapsed       0.200       0.020       0.226 &#62; object.size(fit1)6569.464 kbyteswhile the object is smaller. Now, if we use a stepwise procedure, backward, it is a bit long : almost one minute,  500 times longer than a single logistic regression&#62; system.time(fit&#60;-step(glm(PRONO~.,data=myocarde_large,family="binomial")))             ...Step:  AIC=4118.15PRONO ~ FRCAR + INCAR + INSYS + PRDIA + PVENT + REPUL + X16         Df Deviance    AIC&#60;none&#62;       4102.2 4118.2- X16    1   4104.6 4118.6- PRDIA  1   4113.4 4127.4- INCAR  1   4188.4 4202.4- REPUL  1   4203.9 4217.9- PVENT  1   4215.5 4229.5- FRCAR  1   4254.1 4268.1- INSYS  1   4286.8 4300.8       user      system     elapsed      50.327       0.050      50.368 &#62; object.size(fit)6,652.160 kbytesI also wanted to try caret. This package is nice to compare models. In a review of the bookComputational Actuarial Science with R in JRSS-A, Andrey Kosteko noticed that this package was not even mentioned, and it was missing. So I tried a logistic regression&#62; library(caret)&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large,method="glm"))       user      system     elapsed       5.908       0.032       5.954 &#62; object.size(fit)12,676.944 kbytesIt took 6 seconds (50 times more than a standard call of the glm function), and the object is rather big. It is even worst if we try to run a stepwise procedure&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large,method="glmStepAIC"))                ...Step:  AIC=4118.15.outcome ~ FRCAR + INCAR + INSYS + PRDIA + PVENT + REPUL + X16         Df Deviance    AIC&#60;none&#62;       4102.2 4118.2- X16    1   4104.6 4118.6- PRDIA  1   4113.4 4127.4- INCAR  1   4188.4 4202.4- REPUL  1   4203.9 4217.9- PVENT  1   4215.5 4229.5- FRCAR  1   4254.1 4268.1- INSYS  1   4286.8 4300.8       user      system     elapsed    1063.399       2.926    1068.060 &#62; object.size(fit)9,978.808 kbyteswhich took 15 minutes, with only 30 covariates... Here is the plot (I used microbenchmark to plot it)Let us consider some trees.&#62; library(rpart)&#62; system.time(fit&#60;-rpart(PRONO~.,+ data=myocarde_large))       user      system     elapsed       0.341       0.000       0.345 &#62; object.size(fit4)544.664 kbytesHere it is fast, and the object is rather small. And if we change the complexity parameter, to get a deeper tree, it is almost the same&#62; system.time(fit&#60;-rpart(PRONO~.,+ data=myocarde_large,cp=.001))       user      system     elapsed       0.346       0.000       0.346 &#62; object.size(fit)544.824 kbytesBut again, if we run the same function through caret, it is more than ten times slower,&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large,method="rpart"))       user      system     elapsed       4.076       0.005       4.077 &#62; object.size(fit)5,587.288 kbytesand the object is ten times bigger. Now consider some random forest.&#62; library(randomForest)&#62; system.time(fit&#60;-randomForest(PRONO~.,+ data=myocarde_large,ntree=50))       user      system     elapsed       0.672       0.000       0.671 &#62; object.size(fit)1,751.528 kbytesWith 'only' 50 trees, it is only two times longer to get the output. But with 500 trees (the default value) it takes twenty times more (with a reasonable proportional time, growing 500 trees instead of 50)&#62; system.time(fit&#60;-randomForest(PRONO~.,+ data=myocarde_large,ntree=500))       user      system     elapsed       6.644       0.180       6.821 &#62; object.size(fit)5,133.928 kbytesIf we change the number of covariates to use, at each node, we can see that there is almost no impact. With 5 covariates (which is the square root of the total number of covariates, i.e. it is the default value), it takes 6 seconds,&#62; system.time(fit&#60;-randomForest(PRONO~.,+ data=myocarde_large,mtry=5))       user      system     elapsed       6.266       0.076       6.338 &#62; object.size(fit)5,161.928 kbytesbut if we use 10, it is almost the same (even less)&#62; system.time(fit&#60;-randomForest(PRONO~.,+ data=myocarde_large,mtry=10))       user      system     elapsed       5.666       0.076       5.737 &#62; object.size(fit)2,501.928 bytesIf we use the random forest algorithm within caret, it takes 10 minutes,&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large,method="rf"))       user      system     elapsed     609.790       2.111     613.515and the visualisation isIf we consider a k-nearest neighbor technique, with caret again, it takes some time, with again 10 minutes&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large,method="knn"))       user      system     elapsed      66.994       0.088      67.327 &#62; object.size(fit)5,660.696 kbyteswhich is almost the same time as a bagging algorithm, on trees&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large,method="treebag"))Le chargement a nécessité le package : plyr       user      system     elapsed      60.526       0.567      61.641 &#62; object.size(fit)72,048.480 kbytesbut this time, the object is quite big !We can also consider SVM techniques, with standard Euclidean distance&#62; library(kernlab)&#62; system.time(fit&#60;-ksvm(PRONO~.,+ data=myocarde_large,+ prob.model=TRUE, kernel="vanilladot")) Setting default kernel parameters         user      system     elapsed      14.471       0.076      14.698 &#62; object.size(fit)801.120 kbytesor using some kernel&#62; system.time(fit&#60;-ksvm(PRONO~.,+ data=myocarde_large,+ prob.model=TRUE, kernel="rbfdot"))       user      system     elapsed       9.469       0.052       9.701 &#62; object.size(fit)846.824 kbytesBoth techniques take around 10 seconds, much more than our basic logistic regression (one hundred times more). And again, if we try to use caret to do the same, it takes a while....&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large, method="svmRadial"))       user      system     elapsed     360.421       2.007     364.669 &#62; object.size(fit)4,027.880 kbytesThe output is the following&#160;I also wanted to try some functions, like ridge and LASSO.&#62; library(glmnet)&#62; idx=which(names(myocarde_large)=="PRONO")&#62; y=myocarde_large[,idx]&#62; x=as.matrix(myocarde_large[,-idx])&#62; system.time(fit&#60;-glmnet(x,y,alpha=0,lambda=.05,+ family="binomial"))       user      system     elapsed       0.013       0.000       0.052 &#62; system.time(fit&#60;-glmnet(x,y,alpha=1,lambda=.05,+ family="binomial"))       user      system     elapsed       0.014       0.000       0.013I was surprised to see how fast it. And if we use cross validation to quantify the penalty&#62; system.time(fit10&#60;-cv.glmnet(x,y,alpha=1,+ type="auc",nlambda=100,+ family="binomial"))       user      system     elapsed      11.831       0.000      11.831It takes some time... but it is reasonnable, compared with other techniques. And finally, consider some boosting packages.&#62; system.time(fit&#60;-gbm.step(data=myocarde_large,+ gbm.x = (1:(ncol(myocarde_large)-1))[-idx], + gbm.y = ncol(myocarde_large),+ family = "bernoulli", tree.complexity = 5,+ learning.rate = 0.01, bag.fraction = 0.5))       user      system     elapsed     364.784       0.428     365.755 &#62; object.size(fit)8,607.048 kbytesThat one was long. More than 6 minutes. Using the glmboost package via caret was much faster, this time&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large,method="glmboost"))       user      system     elapsed      13.573       0.024      13.592 &#62; object.size(fit)6,717.400 bytesWhile using gbm via caret was ten times longer,&#62; system.time(fit&#60;-train(PRONO~.,+ data=myocarde_large,method="gbm"))       user      system     elapsed      121.739       0.360     122.466 &#62; object.size(fit)7,115.512 kbytes&#160;All that was done one a laptop. I now have to run the same codes on a faster machine, to try much larger datasets....