<!DOCTYPE html>
<!-- Generated by pkgdown: do not edit by hand --><html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>tutorial • fairmodels</title>
<!-- jquery --><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script><!-- Bootstrap --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/css/bootstrap.min.css" integrity="sha256-bZLfwXAP04zRMK2BjiO8iu9pf4FbLqX6zitd+tIvLhE=" crossorigin="anonymous">
<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/js/bootstrap.min.js" integrity="sha256-nuL8/2cJ5NDSSwnKD8VqreErSWHtnEP9E7AySL+1ev4=" crossorigin="anonymous"></script><!-- bootstrap-toc --><link rel="stylesheet" href="../bootstrap-toc.css">
<script src="../bootstrap-toc.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous">
<!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- dalexverse --><link href="../dalexverse.css" rel="stylesheet">
<link href="../dalexverse-2.css" rel="stylesheet">
<!-- pkgdown --><link href="../pkgdown.css" rel="stylesheet">
<script src="../pkgdown.js"></script><meta property="og:title" content="tutorial">
<meta property="og:description" content="fairmodels">
<!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]--><!-- google analytics --><script async src="https://www.googletagmanager.com/gtag/js?id=UA-5650686-14"></script><script>
 window.dataLayer = window.dataLayer || [];
 function gtag(){dataLayer.push(arguments);}
 gtag('js', new Date());

 gtag('config', 'UA-5650686-14');
</script>
</head>
<body data-spy="scroll" data-target="#toc">
    <div class="container template-article">
      <header><div class="navbar navbar-default navbar-fixed-top" role="navigation">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false">
        <span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>

      <div class="navbar-brand-container">
        <a class="navbar-brand" href="../index.html">fairmodels</a>
        <div class="info">
          <span class="partof">part of the <a href="https://github.com/ModelOriented/DrWhy">DrWhy.AI</a>
           developed by the <a href="https://mi2.mini.pw.edu.pl/">MI^2 DataLab</a> </span>
          <span class="version version-default">0.1.0</span>
        </div>
      </div>
    </div>
    <div id="navbar" class="navbar-collapse collapse">
      <ul class="nav navbar-nav navbar-right">
<li>
  <a href="../index.html">
    <span class="fas fa fas fa-home fa-lg"></span>
     
  </a>
</li>
<li>
  <a href="../reference/index.html">Reference</a>
</li>
<li class="dropdown">
  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
    Articles
     
    <span class="caret"></span>
  </a>
  <ul class="dropdown-menu" role="menu">
<li>
      <a href="../articles/tutorial.html">tutorial</a>
    </li>
  </ul>
</li>
        
      </ul>
</div>
<!--/.nav-collapse -->
  </div>
<!--/.container -->
</div>
<!--/.navbar -->

      

      </header><script src="tutorial_files/header-attrs-2.1/header-attrs.js"></script><div class="row">
  <div class="col-md-9 contents">
    <div class="page-header toc-ignore">
      <h1 data-toc-skip>tutorial</h1>
                        <h4 class="author">Jakub Wiśniewski</h4>
            
      
      
      <div class="hidden name"><code>tutorial.Rmd</code></div>

    </div>

    
    
<div id="objects-and-structure" class="section level1">
<h1 class="hasAnchor">
<a href="#objects-and-structure" class="anchor"></a>Objects and Structure</h1>
<div id="why" class="section level2">
<h2 class="hasAnchor">
<a href="#why" class="anchor"></a>Why?</h2>
<p>Let’s say you are building court system that predicts if someone will become recidivist in the futre. First you gather information then you build a model and predict outcomes. You get accuracy score of 90%. It is preety good, but it appears that the model is more likely to say that African Americans will become recidivists. Model was trained on data that was discriminating certain ethnic groups. So now we have some options. First one is to change the data, and the second one is to tune model, and check if it behaves as we would like it to be. We will choose the second option.</p>
</div>
<div id="data" class="section level2">
<h2 class="hasAnchor">
<a href="#data" class="anchor"></a>Data</h2>
<p>We will use modified ProPublica’s compas data to represent our problem.</p>
<div class="sourceCode" id="cb1"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/utils/data.html">data</a></span>(<span class="st">"compas"</span>)

<span class="fu"><a href="https://rdrr.io/r/utils/head.html">head</a></span>(<span class="no">compas</span>)</pre></body></html></div>
<pre><code>#&gt;    Two_yr_Recidivism Number_of_Priors Age_Above_FourtyFive Age_Below_TwentyFive
#&gt; 4                  0       -0.6843578                   no                   no
#&gt; 5                  1        2.2668817                   no                   no
#&gt; 7                  0       -0.6843578                   no                   no
#&gt; 11                 0       -0.6843578                   no                   no
#&gt; 14                 0       -0.6843578                   no                   no
#&gt; 24                 0       -0.6843578                   no                   no
#&gt;       Sex Misdemeanor        Ethnicity
#&gt; 4    Male         yes            Other
#&gt; 5    Male          no        Caucasian
#&gt; 7  Female         yes        Caucasian
#&gt; 11   Male          no African_American
#&gt; 14   Male         yes         Hispanic
#&gt; 24   Male         yes            Other</code></pre>
</div>
<div id="demographic-parity" class="section level2">
<h2 class="hasAnchor">
<a href="#demographic-parity" class="anchor"></a>Demographic parity</h2>
<p>As our protected group is Ethnicity, we can plot amount of observations in particular class.</p>
<div class="sourceCode" id="cb3"><html><body><pre class="r"><span class="fu"><a href="../reference/demographic_parity.html">demographic_parity</a></span>(<span class="no">compas</span>, <span class="kw">outcome</span> <span class="kw">=</span> <span class="st">"Two_yr_Recidivism"</span>, <span class="kw">group</span>  <span class="kw">=</span> <span class="st">"Ethnicity"</span> )</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-3-1.png" width="700"></p>
<p>We train and explain our model with <code>DALEX</code></p>
<div class="sourceCode" id="cb4"><html><body><pre class="r"><span class="co"># Train</span>
<span class="no">rf_compas</span> <span class="kw">&lt;-</span> <span class="fu">ranger</span>(<span class="no">Two_yr_Recidivism</span> ~<span class="no">.</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>, <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>) <span class="co"># Wszystko</span>

<span class="co"># numeric target values</span>
<span class="no">y_numeric</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span>(<span class="no">compas</span>$<span class="no">Two_yr_Recidivism</span>)-<span class="fl">1</span>

<span class="co"># explainer</span>
<span class="no">rf_explainer</span> <span class="kw">&lt;-</span> <span class="fu">explain</span>(<span class="no">rf_compas</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>)</pre></body></html></div>
<pre><code>#&gt; Preparation of a new explainer is initiated
#&gt;   -&gt; model label       :  ranger  ( [33m default [39m )
#&gt;   -&gt; data              :  6172  rows  6  cols 
#&gt;   -&gt; target variable   :  6172  values 
#&gt;   -&gt; model_info        :  package ranger , ver. 0.12.1 , task classification ( [33m default [39m ) 
#&gt;   -&gt; predict function  :  yhat.ranger  will be used ( [33m default [39m )
#&gt;   -&gt; predicted values  :  numerical, min =  0.1337492 , mean =  0.454866 , max =  0.8627133  
#&gt;   -&gt; residual function :  difference between y and yhat ( [33m default [39m )
#&gt;   -&gt; residuals         :  numerical, min =  -0.7929314 , mean =  0.0002539071 , max =  0.8396044  
#&gt;  [32m A new explainer has been created! [39m</code></pre>
<p>Than we create <code>fairness_object</code> with <code><a href="../reference/create_fairness_object.html">create_fairness_object()</a></code></p>
<div class="sourceCode" id="cb6"><html><body><pre class="r"><span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/create_fairness_object.html">create_fairness_object</a></span>(<span class="no">rf_explainer</span>,
                                  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>, <span class="co"># it is important to specify all data if explainer doesn't have it</span>
                                  <span class="kw">outcome</span> <span class="kw">=</span> <span class="st">"Two_yr_Recidivism"</span>, <span class="co"># target variable</span>
                                  <span class="kw">group</span>  <span class="kw">=</span> <span class="st">"Ethnicity"</span>,          <span class="co"># protected group</span>
                                  <span class="kw">base</span>   <span class="kw">=</span> <span class="st">"Caucasian"</span>,          <span class="co"># to what subgroup we want to compare</span>
                                  <span class="kw">cutoff</span> <span class="kw">=</span> <span class="fl">0.5</span>)                  <span class="co"># cutoff - optional, deafult = 0.5</span></pre></body></html></div>
<p>And we plot!</p>
<div class="sourceCode" id="cb7"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-6-1.png" width="700"></p>
<p>As we can see it is more likely that model will categorise African Americans as future recidivists than for example Asians. But mabey some groups are statisticaly more likely to go do crimes in the future. It is possible, but there are more ways that the model can discriminate people.</p>
</div>
<div id="fairness-object" class="section level2">
<h2 class="hasAnchor">
<a href="#fairness-object" class="anchor"></a>Fairness Object</h2>
<p>To really see what <code>fairness_object</code> is about, we need to make some more models and explainers.</p>
<div class="sourceCode" id="cb8"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span>(<span class="fl">123</span>)

<span class="no">rf_compas_1</span> <span class="kw">&lt;-</span> <span class="fu">ranger</span>(<span class="no">Two_yr_Recidivism</span> ~<span class="no">Number_of_Priors</span>+<span class="no">Age_Below_TwentyFive</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>, <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)
<span class="no">lr_compas_1</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span>(<span class="no">Two_yr_Recidivism</span>~<span class="no">.</span>, <span class="kw">data</span><span class="kw">=</span><span class="no">compas</span>, <span class="kw">family</span><span class="kw">=</span><span class="fu"><a href="https://rdrr.io/r/stats/family.html">binomial</a></span>(<span class="kw">link</span><span class="kw">=</span><span class="st">"logit"</span>))
<span class="no">rf_compas_2</span> <span class="kw">&lt;-</span> <span class="fu">ranger</span>(<span class="no">Two_yr_Recidivism</span> ~<span class="no">.</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>, <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)
<span class="no">rf_compas_3</span> <span class="kw">&lt;-</span> <span class="fu">ranger</span>(<span class="no">Two_yr_Recidivism</span> ~ <span class="no">Age_Above_FourtyFive</span>+<span class="no">Misdemeanor</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>, <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)
<span class="no">rf_compas_4</span> <span class="kw">&lt;-</span> <span class="fu">ranger</span>(<span class="no">Two_yr_Recidivism</span> ~<span class="no">.</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>, <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)
<span class="no">df</span> <span class="kw">&lt;-</span> <span class="no">compas</span>
<span class="no">df</span>$<span class="no">Two_yr_Recidivism</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span>(<span class="no">compas</span>$<span class="no">Two_yr_Recidivism</span>)-<span class="fl">1</span>
<span class="no">gbm_compas_1</span><span class="kw">&lt;-</span> <span class="fu">gbm</span>(<span class="no">Two_yr_Recidivism</span>~<span class="no">.</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">df</span>)

<span class="no">explainer_1</span> <span class="kw">&lt;-</span> <span class="fu">explain</span>(<span class="no">rf_compas_1</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>)
<span class="no">explainer_2</span> <span class="kw">&lt;-</span> <span class="fu">explain</span>(<span class="no">lr_compas_1</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>)
<span class="no">explainer_3</span> <span class="kw">&lt;-</span> <span class="fu">explain</span>(<span class="no">rf_compas_2</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>, <span class="kw">label</span> <span class="kw">=</span> <span class="st">"ranger_2"</span>)
<span class="no">explainer_4</span> <span class="kw">&lt;-</span> <span class="fu">explain</span>(<span class="no">rf_compas_3</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>, <span class="kw">label</span> <span class="kw">=</span> <span class="st">"ranger_3"</span>)
<span class="no">explainer_5</span> <span class="kw">&lt;-</span> <span class="fu">explain</span>(<span class="no">gbm_compas_1</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>)
<span class="no">explainer_6</span> <span class="kw">&lt;-</span> <span class="fu">explain</span>(<span class="no">rf_compas_4</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>, <span class="kw">label</span> <span class="kw">=</span> <span class="st">"ranger_4"</span>)</pre></body></html></div>
<p>Now we create one object with all explainers</p>
<div class="sourceCode" id="cb9"><html><body><pre class="r"><span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/create_fairness_object.html">create_fairness_object</a></span>(<span class="no">explainer_1</span>,<span class="no">explainer_2</span>,<span class="no">explainer_3</span>,<span class="no">explainer_4</span>,<span class="no">explainer_5</span>,<span class="no">explainer_6</span>,
                                  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>,
                                  <span class="kw">outcome</span> <span class="kw">=</span> <span class="st">"Two_yr_Recidivism"</span>,
                                  <span class="kw">group</span>  <span class="kw">=</span> <span class="st">"Ethnicity"</span>,
                                  <span class="kw">base</span>   <span class="kw">=</span> <span class="st">"Caucasian"</span>)</pre></body></html></div>
<p>As we can see there is some parameters in fobject such as:<br>
1. DALEX Explainer or list of explainers<br>
2. data - if first explainer was not provided with whole data frame, we should do it here<br>
3. outcome - target variable, what we want to predict<br>
4. group - protected group/ sensitive variable which can be unfairly treated<br>
5. base - to what subgroup we want to compare metric scores with. If not provided gets first subgroup in data.</p>
<p>What consists of fairness object?</p>
<div class="sourceCode" id="cb10"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/print.html">print</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<pre><code>#&gt; Fairness Matrics: 
#&gt;   TPR_parity_loss TNR_parity_loss PPV_parity_loss NPV_parity_loss
#&gt; 1       0.5782997       0.4876338       0.3001712       0.3219684
#&gt; 2       1.1219620       0.4459198       0.8675808       0.1131800
#&gt; 3       0.7173355       0.3649473       0.4319889       0.3521222
#&gt; 4       0.3902308       0.3734750       0.3647346       0.4881695
#&gt; 5       0.9992566       0.3577169       0.6136059       0.2666404
#&gt; 6       0.7821809       0.3632692       0.3669139       0.3343032
#&gt;   FNR_parity_loss FPR_parity_loss FDR_parity_loss FOR_parity_loss
#&gt; 1       0.5782997       0.4876338       0.3001712       0.3219684
#&gt; 2       1.1219620       0.4459198       0.8675808       0.1131800
#&gt; 3       0.7173355       0.3649473       0.4319889       0.3521222
#&gt; 4       0.3902308       0.3734750       0.3647346       0.4881695
#&gt; 5       0.9992566       0.3577169       0.6136059       0.2666404
#&gt; 6       0.7821809       0.3632692       0.3669139       0.3343032
#&gt;   TS_parity_loss ACC_parity_loss F1_parity_loss MCC_parity_loss    label
#&gt; 1      0.3816694       0.2868008      0.3613032       0.5158728   ranger
#&gt; 2      0.8554819       0.1353720             NA       0.6709033       lm
#&gt; 3      0.5574042       0.3213170      0.5215862       0.6084055 ranger_2
#&gt; 4      0.2905197       0.1960918      0.3303955       0.3505547 ranger_3
#&gt; 5      0.7824425       0.2502543      0.8321117       0.7168035      gbm
#&gt; 6      0.5776680       0.3164251      0.5368256       0.5809875 ranger_4
#&gt; Models explained:
#&gt; [1] "ranger"
#&gt; [1] "lm"
#&gt; [1] "ranger_2"
#&gt; [1] "ranger_3"
#&gt; [1] "gbm"
#&gt; [1] "ranger_4"
#&gt; 
#&gt; Data: 
#&gt;   Two_yr_Recidivism Number_of_Priors Age_Above_FourtyFive Age_Below_TwentyFive
#&gt; 4                 0       -0.6843578                   no                   no
#&gt; 5                 1        2.2668817                   no                   no
#&gt;    Sex Misdemeanor Ethnicity probabilities
#&gt; 4 Male         yes     Other     0.2209553
#&gt; 5 Male          no Caucasian     0.6800501</code></pre>
<p>Fairness object gets metrics based on confussion matrix and checks them over the groups.</p>
<div class="sourceCode" id="cb12"><html><body><pre class="r"><span class="co"># for the first model</span>
<span class="no">fobject</span>$<span class="no">groups_data</span>$<span class="no">ranger</span>$<span class="no">TPR</span></pre></body></html></div>
<pre><code>#&gt;        Caucasian African_American            Asian         Hispanic 
#&gt;        0.5012165        0.7182420        0.5000000        0.4814815 
#&gt;  Native_American            Other 
#&gt;        0.8000000        0.4596774</code></pre>
<p>If we were going only to take score from certain metric (Let’s say fpr and 0.3) we wouldn’t know if it is good or bad. But we are aiming for <strong>equal treatment over all groups</strong> so if this metric score would be the same in all groups it would be very good. But the metrics wouldn’t be comparable between each others (fpr - 0.3 in all groups and acc_parity - 0.9 in all groups, both are good in terms of parity). That is wy we use <strong>base</strong> - to set benchmark. And for example Caucasian in fpr had score of 0.3 and African American 0.6. After setting <code>base = Caucasian</code> Caucasian would have score 1, and African American 2, because is two times higher score in fpr.</p>
<p><em>Note: When dealing with aggregating plots we use formula sum(abs(1-score)) to represent aggregated score in metrics. In short is how much it differs from ideal scores.</em></p>
</div>
</div>
<div id="choosing-best-model" class="section level1">
<h1 class="hasAnchor">
<a href="#choosing-best-model" class="anchor"></a>Choosing best model</h1>
<p>We now have a few models in our <code>fairness_object</code></p>
<p>Let’s see how they perform in diffrent metrics.</p>
<div id="stacked-barplot" class="section level2">
<h2 class="hasAnchor">
<a href="#stacked-barplot" class="anchor"></a>Stacked Barplot</h2>
<div class="sourceCode" id="cb14"><html><body><pre class="r"><span class="no">sm</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/plot_stacked_barplot.html">stack_metrics</a></span>(<span class="no">fobject</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">sm</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-11-1.png" width="700"></p>
<p>At first we see that lm is the worst overall model. It does not give us information if plots are similar to each other. For now only plots basic metrics.</p>
</div>
<div id="plot-metric" class="section level2">
<h2 class="hasAnchor">
<a href="#plot-metric" class="anchor"></a>Plot metric</h2>
<div class="sourceCode" id="cb15"><html><body><pre class="r"><span class="no">cm</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/choose_metric.html">choose_metric</a></span>(<span class="no">fobject</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">cm</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-12-1.png" width="700"></p>
</div>
<div id="plot-fairness-pca" class="section level2">
<h2 class="hasAnchor">
<a href="#plot-fairness-pca" class="anchor"></a>Plot fairness PCA</h2>
<p>With this task we should use <code>PCA</code>. We call <code>create_fairness_pca()</code> to create fairness pca object.</p>
<div class="sourceCode" id="cb16"><html><body><pre class="r"><span class="no">fobject</span>$<span class="no">metric_data</span></pre></body></html></div>
<pre><code>#&gt;   TPR_parity_loss TNR_parity_loss PPV_parity_loss NPV_parity_loss
#&gt; 1       0.5782997       0.4876338       0.3001712       0.3219684
#&gt; 2       1.1219620       0.4459198       0.8675808       0.1131800
#&gt; 3       0.7173355       0.3649473       0.4319889       0.3521222
#&gt; 4       0.3902308       0.3734750       0.3647346       0.4881695
#&gt; 5       0.9992566       0.3577169       0.6136059       0.2666404
#&gt; 6       0.7821809       0.3632692       0.3669139       0.3343032
#&gt;   FNR_parity_loss FPR_parity_loss FDR_parity_loss FOR_parity_loss
#&gt; 1       0.5782997       0.4876338       0.3001712       0.3219684
#&gt; 2       1.1219620       0.4459198       0.8675808       0.1131800
#&gt; 3       0.7173355       0.3649473       0.4319889       0.3521222
#&gt; 4       0.3902308       0.3734750       0.3647346       0.4881695
#&gt; 5       0.9992566       0.3577169       0.6136059       0.2666404
#&gt; 6       0.7821809       0.3632692       0.3669139       0.3343032
#&gt;   TS_parity_loss ACC_parity_loss F1_parity_loss MCC_parity_loss    label
#&gt; 1      0.3816694       0.2868008      0.3613032       0.5158728   ranger
#&gt; 2      0.8554819       0.1353720             NA       0.6709033       lm
#&gt; 3      0.5574042       0.3213170      0.5215862       0.6084055 ranger_2
#&gt; 4      0.2905197       0.1960918      0.3303955       0.3505547 ranger_3
#&gt; 5      0.7824425       0.2502543      0.8321117       0.7168035      gbm
#&gt; 6      0.5776680       0.3164251      0.5368256       0.5809875 ranger_4</code></pre>
<div class="sourceCode" id="cb18"><html><body><pre class="r"><span class="no">fair_pca</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_pca.html">fairness_pca</a></span>(<span class="no">fobject</span>)
<span class="fu"><a href="https://rdrr.io/r/base/print.html">print</a></span>(<span class="no">fair_pca</span>)</pre></body></html></div>
<pre><code>#&gt; Fairness PCA : 
#&gt;             PC1       PC2         PC3          PC4         PC5           PC6
#&gt; [1,] -1.4811263 -2.337868 -1.19295477  0.051644130 -0.07992238 -4.163336e-16
#&gt; [2,]  4.3872842 -1.186862  0.77505772 -0.057414646  0.09610786 -4.440892e-16
#&gt; [3,] -0.7374329  1.106176 -0.56633423  0.232600688  0.21730935 -5.967449e-16
#&gt; [4,] -3.3356161 -0.179074  1.78200016 -0.003749543 -0.02479929 -3.330669e-16
#&gt; [5,]  1.8628962  1.473600 -0.07664671  0.129798283 -0.23750454 -4.440892e-16
#&gt; [6,] -0.6960050  1.124028 -0.72112216 -0.352878912  0.02880900 -5.689893e-16
#&gt; 
#&gt; Created with: 
#&gt; [1] "ranger"   "lm"       "ranger_2" "ranger_3" "gbm"      "ranger_4"
#&gt; 
#&gt; First two components explained 88 % of variance.</code></pre>
<p>Let’s plot!</p>
<div class="sourceCode" id="cb20"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fair_pca</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-14-1.png" width="700"></p>
<p>It is done with loadings plot, which can be customised.</p>
<div class="sourceCode" id="cb21"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fair_pca</span>, <span class="kw">scale</span> <span class="kw">=</span> <span class="fl">0</span>) <span class="co"># deafult = 0.5</span></pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-15-1.png" width="700"></p>
</div>
<div id="plot-heatmap" class="section level2">
<h2 class="hasAnchor">
<a href="#plot-heatmap" class="anchor"></a>Plot Heatmap</h2>
<p>Another way to deal with grouped data is using heatmap.</p>
<div class="sourceCode" id="cb22"><html><body><pre class="r"><span class="no">fheatmap</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_heatmap.html">fairness_heatmap</a></span>(<span class="no">fobject</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fheatmap</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-16-1.png" width="700"></p>
<p>For both models and metrics dendograms are created. This way through hierarchical clustering we can look on similarities between models/metrics. It should give similar but more detailed information than PCA</p>
<p>It can be normalised among metrics.</p>
<div class="sourceCode" id="cb23"><html><body><pre class="r"><span class="no">fheatmap</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_heatmap.html">fairness_heatmap</a></span>(<span class="no">fobject</span>, <span class="kw">scale</span> <span class="kw">=</span> <span class="fl">TRUE</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fheatmap</span>, , <span class="kw">title</span> <span class="kw">=</span> <span class="st">"Title can be changed"</span>, <span class="kw">subtitle</span> <span class="kw">=</span> <span class="st">"subtitle too!"</span>) <span class="co"># we can turn values off via text= FALSE</span></pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-17-1.png" width="700"></p>
<p>Now we know what those scores are and how “similar” models are to each other</p>
</div>
<div id="metric-and-performance-plot" class="section level2">
<h2 class="hasAnchor">
<a href="#metric-and-performance-plot" class="anchor"></a>Metric and Performance Plot</h2>
<p>Sometimes we would like to know how good are models in performance metrics and in fairness metrics at the same time, to see the tradeoff between them.</p>
<div class="sourceCode" id="cb24"><html><body><pre class="r"><span class="no">fap</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/performance_with_fairness.html">performance_and_fairness</a></span>(<span class="no">fobject</span>, <span class="kw">fairness_metric</span> <span class="kw">=</span> <span class="st">"FPR_parity_loss"</span>)</pre></body></html></div>
<pre><code>#&gt; Performace metric is NULL, setting deafult ( auc )  
#&gt; 
#&gt; Creating object with: 
#&gt; Fairness metric FPR_parity_loss 
#&gt; Performance metric  auc</code></pre>
<div class="sourceCode" id="cb26"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fap</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-18-1.png" width="700"></p>
<p>We can add plots with help of <code>patchwork</code></p>
<div class="sourceCode" id="cb27"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">patchwork</span>)
<span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">ggplot2</span>)

<span class="no">p1</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="fu"><a href="../reference/performance_with_fairness.html">performance_and_fairness</a></span>(<span class="no">fobject</span>,  <span class="st">"TPR_parity_loss"</span>, <span class="st">"accuracy"</span>))
<span class="no">p2</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="fu"><a href="../reference/performance_with_fairness.html">performance_and_fairness</a></span>(<span class="no">fobject</span>,  <span class="st">"FPR_parity_loss"</span>, <span class="st">"auc"</span>))     + <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/labs.html">ggtitle</a></span>(<span class="st">" "</span>) + <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/theme.html">theme</a></span>(<span class="kw">legend.position</span> <span class="kw">=</span> <span class="st">"none"</span>)
<span class="no">p3</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="fu"><a href="../reference/performance_with_fairness.html">performance_and_fairness</a></span>(<span class="no">fobject</span>,  <span class="st">"NPV_parity_loss"</span>, <span class="st">"recall"</span>))  + <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/labs.html">ggtitle</a></span>(<span class="st">" "</span>) + <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/theme.html">theme</a></span>(<span class="kw">legend.position</span> <span class="kw">=</span> <span class="st">"none"</span>)
<span class="no">p4</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="fu"><a href="../reference/performance_with_fairness.html">performance_and_fairness</a></span>(<span class="no">fobject</span>,  <span class="st">"TS_parity_loss"</span>, <span class="st">"f1"</span>))+ <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/labs.html">ggtitle</a></span>(<span class="st">" "</span>) + <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/theme.html">theme</a></span>(<span class="kw">legend.position</span> <span class="kw">=</span> <span class="st">"none"</span>)</pre></body></html></div>
<div class="sourceCode" id="cb28"><html><body><pre class="r">(<span class="no">p1</span> + <span class="no">p2</span>)/(<span class="no">p3</span>+<span class="no">p4</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-20-1.png" width="700"></p>
</div>
<div id="back-to-detail" class="section level2">
<h2 class="hasAnchor">
<a href="#back-to-detail" class="anchor"></a>Back to detail</h2>
<p>When we have narrowed down our search for the best model we can use <code>two_models_plot</code> to check once again metrics within groups and decide which model to use.</p>
<div class="sourceCode" id="cb29"><html><body><pre class="r"><span class="no">fobject2</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/create_fairness_object.html">create_fairness_object</a></span>(<span class="no">explainer_1</span>,<span class="no">explainer_3</span>,
                                   <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>,
                                  <span class="kw">outcome</span> <span class="kw">=</span> <span class="st">"Two_yr_Recidivism"</span>,
                                  <span class="kw">group</span>  <span class="kw">=</span> <span class="st">"Ethnicity"</span>,
                                  <span class="kw">base</span>   <span class="kw">=</span> <span class="st">"Caucasian"</span>)


<span class="no">gm</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/group_metric.html">group_metric</a></span>(<span class="no">fobject2</span>, <span class="kw">fairness_metric</span> <span class="kw">=</span> <span class="st">"FPR"</span>)</pre></body></html></div>
<pre><code>#&gt; Performace metric not given, setting deafult ( auc )  
#&gt; 
#&gt; Creating object with: 
#&gt; Fairness metric FPR 
#&gt; Performance metric  auc</code></pre>
<div class="sourceCode" id="cb31"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">gm</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-21-1.png" width="700"></p>
</div>
<div id="radar-plot" class="section level2">
<h2 class="hasAnchor">
<a href="#radar-plot" class="anchor"></a>Radar plot</h2>
<div class="sourceCode" id="cb32"><html><body><pre class="r"><span class="no">fradar</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_radar.html">fairness_radar</a></span>(<span class="no">fobject2</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fradar</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-22-1.png" width="700"></p>
<p>We can also use simple <code>plot</code> when we have only few explainers.</p>
<div class="sourceCode" id="cb33"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject2</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-23-1.png" width="700"></p>
<p>Of course the protected group can be changed.</p>
<div class="sourceCode" id="cb34"><html><body><pre class="r"><span class="no">fobject3</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/create_fairness_object.html">create_fairness_object</a></span>(<span class="no">explainer_2</span>,<span class="no">explainer_5</span>,
                                   <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>,
                                  <span class="kw">outcome</span> <span class="kw">=</span> <span class="st">"Two_yr_Recidivism"</span>,
                                  <span class="kw">group</span>  <span class="kw">=</span> <span class="st">"Sex"</span>,
                                  <span class="kw">base</span> <span class="kw">=</span> <span class="st">"Female"</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject3</span>)</pre></body></html></div>
<p><img src="tutorial_files/figure-html/unnamed-chunk-24-1.png" width="700"></p>
</div>
</div>
  </div>

  <div class="col-md-3 hidden-xs hidden-sm" id="pkgdown-sidebar">

        <nav id="toc" data-toggle="toc"><h2 data-toc-skip>Contents</h2>
    </nav>
</div>

</div>



      <footer><div class="copyright">
  <p>Developed by Jakub Wiśniewski, Przemysław Biecek.</p>
</div>

<div class="pkgdown">
  <p>Site built with <a href="https://pkgdown.r-lib.org/">pkgdown</a>.</p>
</div>

      </footer>
</div>

  


  </body>
</html>
