{"word_count": 890, "character_count": 6236}
This afternoon, André did send me an interesting graph about the use of Lorenz curve in the context of insurance pricing (and modeling)It is some sort of Lorenz curve, upside-down, with on the x-axis the proportion of the population, and on the y-axis the proportion of the losses. The important point is that the population is sorted according the their risk, i.e. their premium. The code to generate such a curve is actually quite simple,L &#60;- function(u,varx="premium",vary="losses"){  base=base[order(base[,varx],decreasing=TRUE),]  base$cum=(1:nrow(base))/nrow(base)return(sum(base[base$cum&#60;=u,vary])/sum(base[,vary]))} vu=seq(0,1,by=.01)vv=Vectorize(function(u) L(u))(vu)My concern was more on two labels on the figure, with on the top-left "perfect pricing" and on the first diagonal "average pricing". What could that possibly mean? Is there even such a thing as a "perfect pricing"? In order to understand what we plot here, let us generate some dataset, and fit some model. Including things that might be seen as the "perfect model": the price base on the parameters used to generate the data, and the model used to generate the data, fitted on the data.But first of all, let us generate some data,&#62; n  = 1e5&#62; X1 = runif(n)&#62; X2 = 1+rexp(n)&#62; s  = -3+X1-X2&#62; p  = exp(s)/(1+exp(s))&#62; m  = exp(X2)&#62; v  = 2&#62; Y  = (runif(n)&#60;p)*  rgamma(n,shape=m^2/v,scale=v/m)&#62; base &#60;- data.frame(Y,X1,X2)There are here 10,000 observations, two covariates, a logistic model for the occurence of claim, and when there is a claim, a Gamma distribution with some log link function.My first "perfect pricing" model is the one obtained using the probability used to generate the occurence of the claims, and the expected value of the loss size,&#62; base$p0 = p*mThe second "perfect model" is the one we can get but fitting the model we used to generate the dataset,&#62; sbase = subset(base,Y&#62;0)&#62; reg_freq=glm((Y&#62;0)~.,data=base,family=binomial)&#62; reg_cost=glm(Y~X2,data=sbase,  family=Gamma(link="log"))&#62; Freq = predict(reg_freq,newdata=base,  type="response")&#62; Cost = predict(reg_cost,newdata=base,  type="response")&#62; Premium = Freq*Cost&#62; base$p1 = PremiumHere we have&#62; head(base)  Y          X1       X2         p1         p01 0 0.007109038 1.739082 0.04802496 0.049704422 0 0.014693911 1.548622 0.04849476 0.049987253 0 0.683403423 1.965448 0.09353131 0.097264974 0 0.929720222 2.273194 0.11897077 0.124532645 0 0.275401199 1.699431 0.06266035 0.064795876 0 0.811859695 2.026768 0.10611473 0.11049269If we try to visualize the two Lorenz curve, use&#62; vu = seq(0,1,by=.01)&#62; vv = Vectorize(function(u) L(u,"p1","Y"))(vu)&#62; plot(vu,vv,type="l",col="red")&#62; vv=Vectorize(function(u) L(u,"p0","Y"))(vu)&#62; lines(vu,vv,type="l",col="blue")It looks like our "perfect" curves are much closed to the diagonal than the one we had on the figure above. If we compute some sort of Gini index, we get less than 20%,&#62; sum(vv)/100*2-1[1] 0.1821662Before trying to see if we can improve it, maybe we can try to see what an "average pricing" could mean. If the connexion with Lorenz curve is valid, then it should be the "equalitarian" version, i.e. no segmentation.&#62; base$p3=mean(base$Y)&#62; base=base[sample(1:nrow(base)),]Here we sample because there will be some ties in the premium, when we will try to sort it, and Lorenz curve is&#62; vv=Vectorize(function(u) L(u,"p3"))(vu)&#62; sum(vv)/100*2-1[1] 0.04438427We are not far away from the diagonal and observe that we have almost 5% for Gini index. Let us get back to the possible "perfect pricing". Actually, from a modelling point of view (and not an actuarial one, I believe), the "perfect pricing" might be an a posteriori pricing : every one pays a premium equal to the actual loss.&#62; base$p4 = base$Y&#62; vv = Vectorize(function(u) L(u,"p4","Y"))(vu)&#62; lines(vu,vv,type="l",lwd=2)Indeed, using that technique, we can reach the upper corner of the figure. But from an economic perspective, it cannot be seen as "perfect" since there is no longer risk pooling here. And from a statistician perspective, it might be perfect, but clearly with zero predictive power ! It can be seen as an heavily over-fitted model, that works perfectly on the training sample, but will have no predictive power at all on the validation sample.Let us try something a bit more realistic. Like some machine learning algorithm, like a boosting one (since many people doing data science claim that boosting is usually a great technique, with great predictive performence).&#62; library(dismo)&#62; reg=gbm.step(data=df, gbm.x = 2:3, gbm.y = 1,+ family = "gaussian", tree.complexity = 5,+ learning.rate = 0.001, bag.fraction = 0.5)  GBM STEP - version 2.9  Performing cross-validation optimisation of a boosted regression tree model for Y and using a family of gaussian Using 400 observations and 2 predictors creating 10 initial models of 50 trees   folds are unstratified total mean deviance =  0.2194 tolerance is fixed at  2e-04 ntrees resid. dev. 50    0.2145 now adding trees... 100   0.2094 150   0.2048 200   0.2006 250   0.1969(etc)2050   0.1693 2100   0.1695 fitting final gbm model with a fixed number of  1700  trees for  Y  mean total deviance = 0.219 mean residual deviance = 0.139  estimated cv deviance = 0.169 ; se = 0.014  training data correlation = 0.628 cv correlation =  0.495 ; se = 0.057  elapsed time -  0.1 minutes &#62; base$p5 = predict(reg,newdata=df,n.trees=1700)If we visualize the Lorenz curve, we get&#62; vv = Vectorize(function(u) L(u,"p5","Y"))(vu)&#62; lines(vu,vv,type="l",lwd=2)Here it looks like the "performence" of the boosting algorithm is the same as using an "average pricing".I still have some trouble analyzing those curves. I know that some insurance companies use that intensively, but the only article I could find online is a paper by Jed Frees with Glenn Meyers and Dave Cummins, Insurance Ratemaking and a Gini Index (recently published in the Journal of Risk and Insurance). If someone has more experience about those curves, and above the possible upper bound with a "realistic" model, that could be interpreted as "perfect", I would be glad to hear about it...