{"word_count": 538, "character_count": 4745}
We will start, in our Data Science course,  to discuss classification techniques (in the context of supervised models). Consider the following case, with 10 points, and two classes (red and blue)&#62; clr1 &#60;- c(rgb(1,0,0,1),rgb(0,0,1,1))&#62; clr2 &#60;- c(rgb(1,0,0,.2),rgb(0,0,1,.2))&#62; x &#60;- c(.4,.55,.65,.9,.1,.35,.5,.15,.2,.85)&#62; y &#60;- c(.85,.95,.8,.87,.5,.55,.5,.2,.1,.3)&#62; z &#60;- c(1,1,1,1,1,0,0,1,0,0)&#62; df &#60;- data.frame(x,y,z)&#62; plot(x,y,pch=19,cex=2,col=clr1[z+1])To get a prediction, i.e. a partition of the space in two parts, consider some logistic regression&#62; reg=glm(z~x+y,data=df,family=binomial)&#62; summary(reg) Call:glm(formula = z ~ x + y, family = binomial, data = df) Deviance Residuals:     Min       1Q   Median       3Q      Max  -1.6593  -0.4400   0.2564   0.5830   1.5374   Coefficients:            Estimate Std. Error z value Pr(&#62;|z|)(Intercept)   -1.706      1.999  -0.854    0.393x             -5.489      5.360  -1.024    0.306y              8.568      5.515   1.554    0.120 (Dispersion parameter for binomial family taken to be 1)     Null deviance: 13.4602  on 9  degrees of freedomResidual deviance:  8.1445  on 7  degrees of freedomAIC: 14.144 Number of Fisher Scoring iterations: 5Given some point, the predicted class is obtained using&#62; pred_1 &#60;- function(x,y){+ predict(reg,newdata=data.frame(x=x,+ y=y),type="response")&#62;.5+ }(here, the predicted class is simply the one that is the most likely). To visualize it use&#62; x_grid&#60;-seq(0,1,length=101)&#62; y_grid&#60;-seq(0,1,length=101)&#62; z_grid &#60;- outer(x_grid,y_grid,pred_1)&#62; image(x_grid,y_grid,z_grid,col=clr2)&#62; points(x,y,pch=19,cex=2,col=clr1[z+1])Since the logistic regression is a (generalized) linear model, the line that separate the two regions is a straight line.What if we had three classes, and not two (red, yellow and blue)&#62; clr1=c(rgb(1,0,0,1),rgb(1,1,0,1),rgb(0,0,1,1))&#62; clr2=c(rgb(1,0,0,.2),rgb(1,1,0,.2),+ rgb(0,0,1,.2))&#62; x=c(.4,.55,.65,.9,.1,.35,.5,.15,.2,.85)&#62; y=c(.85,.95,.8,.87,.5,.55,.5,.2,.1,.3)&#62; z=c(1,2,2,2,1,0,0,1,0,0)&#62; df=data.frame(x,y,z)&#62; plot(x,y,pch=19,cex=2,col=clr1[z+1])We should use here a multinomial model. Recall that for the logistic regression model, if , theni.e.whileTo derive a multivariate extension, writeandHere, maximum likelihood techniques can be used, sincewhere variable   – which take three levels – is splitted in three indicators (like any categorical explanatory variables in standard regression model). The code to estimate that model is simply&#62; library(nnet)&#62; reg=multinom(z~x+y,data=df)# weights:  12 (6 variable)initial  value 10.986123 iter  10 value 0.794930iter  20 value 0.065712iter  30 value 0.064409iter  40 value 0.061612iter  50 value 0.058756iter  60 value 0.056225iter  70 value 0.055332iter  80 value 0.052887iter  90 value 0.050644iter 100 value 0.048117final  value 0.048117 stopped after 100 iterationsand the output is&#62; summary(reg)Call:multinom(formula = z ~ x + y, data = df) Coefficients:  (Intercept)          x         y1    14.44215 -126.90076  47.684882  -114.49676   65.23419 100.31341 Std. Errors:  (Intercept)        x        y1    23.03561 144.8639 52.990492    50.54085 267.1013 98.60386 Residual Deviance: 0.09623369 AIC: 12.09623In the binomial case, there was only one interesting case (once we have the probability to be equal to 1, we have the probability to be null). Here, there are two. So we have two lines, for the three parameters. The prediction function, for the class, is here&#62; pred_2 &#60;- function(x,y){+ which.max(predict(reg,+ newdata=data.frame(x=x,y=y),type="probs"))+ }Now, to visualize predictions, use&#62; x_grid&#60;-seq(0,1,length=101)&#62; y_grid&#60;-seq(0,1,length=101)&#62; z_grid &#60;- outer(x_grid,y_grid,pred_2)Error in outer(x_grid, y_grid, pred_2) :   dims [produit 10201] do not match the length of object [1]Unfortunately, it is not working. Neither is&#62; z_grid &#60;- outer(x_grid,y_grid,function(x,y) pred_2(x,y))Error in outer(x_grid, y_grid, function(x, y) pred_2(x, y)) :   dims [produit 10201] do not match the length of object [1]If any one has a suggestion, I'd be glad to know. Just because we really want to get our graph, use&#62; Outer &#60;- function(x,y,fun) {+ mat &#60;- matrix(NA, length(x), length(y))+ for (i in seq_along(x)) {+ for (j in seq_along(y)) mat[i,j]=fun(x[i],y[j])+ }+ return(mat)}&#62; z_grid &#60;- Outer(x_grid,y_grid,pred_2)(which is slow, but at least, it's running). The prediction is&#62; image(x_grid,y_grid,z_grid,col=clr2)&#62; points(x,y,pch=19,cex=2,col=clr1[z+1])We now have three regions, the frontier being linear, and the intersection being the equiprobable case.