<!DOCTYPE html>
<!-- Generated by pkgdown: do not edit by hand --><html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Basic Tutorial • fairmodels</title>
<!-- favicons --><link rel="icon" type="image/png" sizes="16x16" href="../favicon-16x16.png">
<link rel="icon" type="image/png" sizes="32x32" href="../favicon-32x32.png">
<link rel="apple-touch-icon" type="image/png" sizes="180x180" href="../apple-touch-icon.png">
<link rel="apple-touch-icon" type="image/png" sizes="120x120" href="../apple-touch-icon-120x120.png">
<link rel="apple-touch-icon" type="image/png" sizes="76x76" href="../apple-touch-icon-76x76.png">
<link rel="apple-touch-icon" type="image/png" sizes="60x60" href="../apple-touch-icon-60x60.png">
<!-- jquery --><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script><!-- Bootstrap --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/css/bootstrap.min.css" integrity="sha256-bZLfwXAP04zRMK2BjiO8iu9pf4FbLqX6zitd+tIvLhE=" crossorigin="anonymous">
<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/js/bootstrap.min.js" integrity="sha256-nuL8/2cJ5NDSSwnKD8VqreErSWHtnEP9E7AySL+1ev4=" crossorigin="anonymous"></script><!-- bootstrap-toc --><link rel="stylesheet" href="../bootstrap-toc.css">
<script src="../bootstrap-toc.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous">
<!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- dalexverse --><link href="../dalexverse.css" rel="stylesheet">
<link href="../dalexverse-2.css" rel="stylesheet">
<!-- pkgdown --><link href="../pkgdown.css" rel="stylesheet">
<script src="../pkgdown.js"></script><meta property="og:title" content="Basic Tutorial">
<meta property="og:description" content="fairmodels">
<meta property="og:image" content="/logo.png">
<meta name="robots" content="noindex">
<!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]--><!-- google analytics --><script async src="https://www.googletagmanager.com/gtag/js?id=UA-5650686-14"></script><script>
 window.dataLayer = window.dataLayer || [];
 function gtag(){dataLayer.push(arguments);}
 gtag('js', new Date());

 gtag('config', 'UA-5650686-14');
</script>
</head>
<body data-spy="scroll" data-target="#toc">
    <div class="container template-article">
      <header><div class="navbar navbar-default navbar-fixed-top" role="navigation">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false">
        <span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>

      <div class="navbar-brand-container">
        <a class="navbar-brand" href="../index.html">fairmodels</a>
        <div class="info">
          <span class="partof">part of the <a href="https://github.com/ModelOriented/DrWhy">DrWhy.AI</a>
           developed by the <a href="https://mi2.mini.pw.edu.pl/">MI^2 DataLab</a> </span>
          <span class="version version-default">0.1.1</span>
        </div>
      </div>
    </div>
    <div id="navbar" class="navbar-collapse collapse">
      <ul class="nav navbar-nav navbar-right">
<li>
  <a href="../index.html">
    <span class="fas fa fas fa-home fa-lg"></span>
     
  </a>
</li>
<li>
  <a href="../reference/index.html">Reference</a>
</li>
<li class="dropdown">
  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
    Articles
     
    <span class="caret"></span>
  </a>
  <ul class="dropdown-menu" role="menu">
<li>
      <a href="../articles/Advanced_tutorial.html">Advanced Tutorial</a>
    </li>
    <li>
      <a href="../articles/Basic_tutorial.html">Basic Tutorial</a>
    </li>
  </ul>
</li>
<li>
  <a href="../news/index.html">Changelog</a>
</li>
        <li>
  <a href="https://github.com/ModelOriented/fairmodels/">
    <span class="fab fa fab fa-github fa-lg"></span>
     
  </a>
</li>
      </ul>
</div>
<!--/.nav-collapse -->
  </div>
<!--/.container -->
</div>
<!--/.navbar -->

      

      </header><script src="Basic_tutorial_files/header-attrs-2.3/header-attrs.js"></script><script src="Basic_tutorial_files/accessible-code-block-0.0.1/empty-anchor.js"></script><div class="row">
  <div class="col-md-9 contents">
    <div class="page-header toc-ignore">
      <h1 data-toc-skip>Basic Tutorial</h1>
                        <h4 class="author">Jakub Wiśniewski</h4>
            
      
      <small class="dont-index">Source: <a href="https://github.com/ModelOriented/fairmodels/blob/master/vignettes/Basic_tutorial.Rmd"><code>vignettes/Basic_tutorial.Rmd</code></a></small>
      <div class="hidden name"><code>Basic_tutorial.Rmd</code></div>

    </div>

    
    
<div id="fairmodels" class="section level1">
<h1 class="hasAnchor">
<a href="#fairmodels" class="anchor"></a>fairmodels</h1>
<p>In this tutorial you will get to know when, why and how to use <code>fairmodels</code>. <code>fairmodels</code> is a tool for bias testing and fairness metrics visualization. It is compatible with <code>DALEX</code> and <code>DALEXtra</code> which are model agnostic explainers. Some knowledge of how to use those explainers will be needed but in this tutorial you should grasp the idea.</p>
<p>For this tutorial we will use <code>compas</code> data to see if someone will become recidivist in next 2 years.</p>
<div id="why" class="section level2">
<h2 class="hasAnchor">
<a href="#why" class="anchor"></a>Why?</h2>
<p>Let’s say you are building court system that predicts if someone will become recidivist in the future. First you gather information then you build a model and predict outcomes. You get accuracy score of 90%. It is pretty good, but it appears that the model is more likely to say that African Americans will become recidivists. Model was trained on data that was discriminating certain ethnic groups. So now we have some options. First one is to change the data, and the second one is to tune model, and check if it behaves as we would like it to be. We will choose the second option.</p>
</div>
<div id="data" class="section level2">
<h2 class="hasAnchor">
<a href="#data" class="anchor"></a>Data</h2>
<p>We will use modified ProPublica’s compas data to represent our problem.</p>
<div class="sourceCode" id="cb1"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">fairmodels</span>)
<span class="fu"><a href="https://rdrr.io/r/utils/data.html">data</a></span>(<span class="st">"compas"</span>)

<span class="fu"><a href="https://rdrr.io/r/utils/head.html">head</a></span>(<span class="no">compas</span>)</pre></body></html></div>
<pre><code>#&gt;   Two_yr_Recidivism Number_of_Priors Age_Above_FourtyFive Age_Below_TwentyFive
#&gt; 1                 0                0                    1                    0
#&gt; 2                 1                0                    0                    0
#&gt; 3                 1                4                    0                    1
#&gt; 4                 0                0                    0                    0
#&gt; 5                 1               14                    0                    0
#&gt; 6                 0                3                    0                    0
#&gt;   Misdemeanor        Ethnicity  Sex
#&gt; 1           0            Other Male
#&gt; 2           0 African_American Male
#&gt; 3           0 African_American Male
#&gt; 4           1            Other Male
#&gt; 5           0        Caucasian Male
#&gt; 6           0            Other Male</code></pre>
<p>For <code>fairmodels</code> package to work properly we want to flip factor levels in target variable, so positive outcome (not being a recidivist) is being predicted by models. It is only needed for one specific function but more on it later.</p>
<div class="sourceCode" id="cb3"><html><body><pre class="r"><span class="no">compas</span>$<span class="no">Two_yr_Recidivism</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/factor.html">as.factor</a></span>(<span class="fu"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span>(<span class="no">compas</span>$<span class="no">Two_yr_Recidivism</span> <span class="kw">==</span> <span class="st">'1'</span>, <span class="st">'0'</span>, <span class="st">'1'</span>))</pre></body></html></div>
</div>
<div id="basic-features" class="section level2">
<h2 class="hasAnchor">
<a href="#basic-features" class="anchor"></a>Basic features</h2>
<p>We train a <code>ranger</code> model and create an explainer with <code>DALEX</code>.</p>
<div class="sourceCode" id="cb4"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">DALEX</span>)
<span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">ranger</span>)

<span class="co"># train</span>
<span class="no">rf_compas</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/ranger/man/ranger.html">ranger</a></span>(<span class="no">Two_yr_Recidivism</span> ~<span class="no">.</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>, <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)

<span class="co"># numeric target values</span>
<span class="no">y_numeric</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span>(<span class="no">compas</span>$<span class="no">Two_yr_Recidivism</span>)-<span class="fl">1</span>

<span class="co"># explainer</span>
<span class="no">rf_explainer</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">rf_compas</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>, <span class="kw">colorize</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<pre><code>#&gt; Preparation of a new explainer is initiated
#&gt;   -&gt; model label       :  ranger  (  default  )
#&gt;   -&gt; data              :  6172  rows  6  cols 
#&gt;   -&gt; target variable   :  6172  values 
#&gt;   -&gt; predict function  :  yhat.ranger  will be used (  default  )
#&gt;   -&gt; predicted values  :  numerical, min =  0.167963 , mean =  0.5446367 , max =  0.8660183  
#&gt;   -&gt; model_info        :  package ranger , ver. 0.12.1 , task classification (  default  ) 
#&gt;   -&gt; residual function :  difference between y and yhat (  default  )
#&gt;   -&gt; residuals         :  numerical, min =  -0.8454124 , mean =  0.0002433993 , max =  0.7735712  
#&gt;   A new explainer has been created!</code></pre>
<div id="fairness-check" class="section level3">
<h3 class="hasAnchor">
<a href="#fairness-check" class="anchor"></a>fairness check</h3>
<p>Than we create call function <code><a href="../reference/fairness_check.html">fairness_check()</a></code> This function aggregates many explainers so you may compare many models. We assign object to name <code>fobject</code> which is short version of <code>fairness_object</code> - object returned by <code><a href="../reference/fairness_check.html">fairness_check()</a></code>,</p>
<div class="sourceCode" id="cb6"><html><body><pre class="r"><span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">rf_explainer</span>,                         <span class="co"># explainer</span>
                          <span class="kw">protected</span> <span class="kw">=</span> <span class="no">compas</span>$<span class="no">Ethnicity</span>,         <span class="co"># protected variable as factor</span>
                          <span class="kw">privileged</span> <span class="kw">=</span> <span class="st">"Caucasian"</span>,             <span class="co"># level in protected variable, potentially more privileged</span>
                          <span class="kw">cutoff</span> <span class="kw">=</span> <span class="fl">0.5</span>,                         <span class="co"># cutoff - optional, default = 0.5</span>
                          <span class="kw">colorize</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<pre><code>#&gt; Creating fairness object
#&gt; -&gt; Privileged subgroup       : character ( Ok  )
#&gt; -&gt; Protected variable        : factor ( Ok  ) 
#&gt; -&gt; Cutoff values for explainers  : 0.5 ( for all subgroups )
#&gt; -&gt; Fairness objects      : 0 objects 
#&gt; -&gt; Checking explainers       : 1 in total (  compatible  )
#&gt; -&gt; Metric calculation        : 13/13 metrics calculated for all models
#&gt;  Fairness object created succesfully</code></pre>
<p>Let’s see if our ranger model has bias.</p>
<div class="sourceCode" id="cb8"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/print.html">print</a></span>(<span class="no">fobject</span>, <span class="kw">colorize</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<pre><code>#&gt; 
#&gt; Fairness check for models: ranger 
#&gt; 
#&gt; ranger passes 0/5 metrics
#&gt; Total loss:  2.658761</code></pre>
<div class="sourceCode" id="cb10"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-7-1.png" width="720" style="display: block; margin: auto;"></p>
<p>Of course the <code>protected</code> parameter can be changed, for example to binary <code>compas$Sex</code>.</p>
<p>In many metrics <code>ranger</code> exceeds fairness threshold (which can be changed by <code>epsilon</code> parameter). If bars reach red field on the left it means that there is bias towards certain unprivileged subgroup. If they reach one on the right it means bias towards privileged (Caucasian - in all metrics here Caucasian subgroup is referenced as base - 0) subgroup. Someone can argue that some groups are statistically more likely commit a crime but <code><a href="../reference/fairness_check.html">fairness_check()</a></code> takes it into account. Statistical parity loss checks if proportion of assigned positive class is equal among all subgroups. In this example Native Americans and African Americans are more likely to be classified as recidivists. Other metrics measure how equal treatment and mistreatment among subgroups is. More on those metrics: <a href="https://en.wikipedia.org/wiki/Fairness_(machine_learning)">wikipedia</a></p>
<p>Why do we have this bias? Model did learn from biased data. We can see it on plot below</p>
</div>
<div id="plot-density" class="section level3">
<h3 class="hasAnchor">
<a href="#plot-density" class="anchor"></a>plot density</h3>
<div class="sourceCode" id="cb11"><html><body><pre class="r"><span class="fu"><a href="../reference/plot_density.html">plot_density</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-8-1.png" width="720" style="display: block; margin: auto;"></p>
<p>As we can see it is more likely that model will categorize African Americans as not being recidivists than for example Asians. But maybe some groups are statistically more likely to go do crimes in the future. It is possible but that is why we used <code><a href="../reference/fairness_check.html">fairness_check()</a></code> earlier. It does not only catch if subgroups are .</p>
</div>
</div>
<div id="fairness-object---idea" class="section level2">
<h2 class="hasAnchor">
<a href="#fairness-object---idea" class="anchor"></a>fairness object - idea</h2>
<p>To really see what <code>fairness_object</code> is about, we need to make some more models and explainers.</p>
<div class="sourceCode" id="cb12"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">gbm</span>)

<span class="no">rf_compas_1</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/ranger/man/ranger.html">ranger</a></span>(<span class="no">Two_yr_Recidivism</span> ~<span class="no">Number_of_Priors</span>+<span class="no">Age_Below_TwentyFive</span>,
                      <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>,
                      <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)

<span class="no">lr_compas_1</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span>(<span class="no">Two_yr_Recidivism</span>~<span class="no">.</span>,
                   <span class="kw">data</span><span class="kw">=</span><span class="no">compas</span>,
                   <span class="kw">family</span><span class="kw">=</span><span class="fu"><a href="https://rdrr.io/r/stats/family.html">binomial</a></span>(<span class="kw">link</span><span class="kw">=</span><span class="st">"logit"</span>))

<span class="no">rf_compas_2</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/ranger/man/ranger.html">ranger</a></span>(<span class="no">Two_yr_Recidivism</span> ~<span class="no">.</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>, <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)
<span class="no">rf_compas_3</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/ranger/man/ranger.html">ranger</a></span>(<span class="no">Two_yr_Recidivism</span> ~ <span class="no">Age_Above_FourtyFive</span>+<span class="no">Misdemeanor</span>,
                      <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>,
                      <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)

<span class="no">rf_compas_4</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/ranger/man/ranger.html">ranger</a></span>(<span class="no">Two_yr_Recidivism</span> ~<span class="no">.</span>,
                      <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>,
                      <span class="kw">probability</span> <span class="kw">=</span> <span class="fl">TRUE</span>)
<span class="no">df</span> <span class="kw">&lt;-</span> <span class="no">compas</span>
<span class="no">df</span>$<span class="no">Two_yr_Recidivism</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span>(<span class="no">compas</span>$<span class="no">Two_yr_Recidivism</span>)-<span class="fl">1</span>
<span class="no">gbm_compas_1</span><span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/gbm/man/gbm.html">gbm</a></span>(<span class="no">Two_yr_Recidivism</span>~<span class="no">.</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">df</span>)

<span class="no">explainer_1</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">rf_compas_1</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>)
<span class="no">explainer_2</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">lr_compas_1</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>)
<span class="no">explainer_3</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">rf_compas_2</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>, <span class="kw">label</span> <span class="kw">=</span> <span class="st">"ranger_2"</span>)
<span class="no">explainer_4</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">rf_compas_3</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>, <span class="kw">label</span> <span class="kw">=</span> <span class="st">"ranger_3"</span>)
<span class="no">explainer_5</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">gbm_compas_1</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>)
<span class="no">explainer_6</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">rf_compas_4</span>,  <span class="kw">data</span> <span class="kw">=</span> <span class="no">compas</span>[,-<span class="fl">1</span>], <span class="kw">y</span> <span class="kw">=</span> <span class="no">y_numeric</span>, <span class="kw">label</span> <span class="kw">=</span> <span class="st">"ranger_4"</span>)</pre></body></html></div>
<p>Now we create one object with all explainers</p>
<div class="sourceCode" id="cb13"><html><body><pre class="r"><span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">explainer_1</span>, <span class="no">explainer_2</span>,
                            <span class="no">explainer_3</span>, <span class="no">explainer_4</span>,
                            <span class="no">explainer_5</span>, <span class="no">explainer_6</span>,
                            <span class="kw">protected</span> <span class="kw">=</span> <span class="no">compas</span>$<span class="no">Ethnicity</span>,
                            <span class="kw">privileged</span> <span class="kw">=</span> <span class="st">"Caucasian"</span>,
                            <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<p>As we can see there is some parameters in fairness_check such as:<br>
1. x, … - list of <code>DALEX</code> explainers, and other <code>fairness_object</code> objects<br>
2. protected - factor, containing subgroups as levels. Protected stands for protected variable (or sensitive attribute)<br>
3. privileged - character, level in protected, it is subgroup suspected of having better results<br>
4. cutoff - numeric, vector of cutoffs values matching the order of levels in protected variable. It affects only explainers so if <code>fairness_object</code> is passed it’s cutoff vector won’t be changed.<br>
5. label - character, vector of labels for explainers only. Very convenient for <code><a href="../reference/fairness_check.html">fairness_check()</a></code> iterative approach - having explainer, checking for bias, mitigating bias, passing both explainer and fairness object and comparing fairness.<br>
6. epsilon - numeric, boundary position in <code><a href="../reference/fairness_check.html">fairness_check()</a></code>. Fairness metrics are satisfied if parity loss values are between (-epsilon, epsilon)</p>
<div id="what-consists-of-fairness-object" class="section level3">
<h3 class="hasAnchor">
<a href="#what-consists-of-fairness-object" class="anchor"></a>What consists of fairness object?</h3>
<p><code>fairness_object</code> is output value of <code><a href="../reference/fairness_check.html">fairness_check()</a></code> It is S3 object consisting of: * Parity loss metrics Popular confusion matrix metrics with parity loss - sum of distances of metric values between unprivileged subgroups and privileged one. If model would have 0 in certain parity loss metric it would mean that it treats all subgroups equally.</p>
<div class="sourceCode" id="cb14"><html><body><pre class="r"><span class="no">fobject</span>$<span class="no">parity_loss_metric_data</span></pre></body></html></div>
<pre><code>#&gt;         TPR       TNR       PPV       NPV       FNR       FPR       FDR
#&gt; 1 0.4876338 0.5782997 0.3219684 0.3001712 0.4876338 0.5782997 0.3219684
#&gt; 2 0.4459198 1.1219620 0.1131800 0.8675808 0.4459198 1.1219620 0.1131800
#&gt; 3 0.3668100 0.7006021 0.3716002 0.4628820 0.3668100 0.7006021 0.3716002
#&gt; 4 0.3734750 0.3902308 0.4881695 0.3647346 0.3734750 0.3902308 0.4881695
#&gt; 5 0.3590379 0.7924087 0.1493433 0.5660787 0.3590379 0.7924087 0.1493433
#&gt; 6 0.4400061 0.7241637 0.3396654 0.4492735 0.4400061 0.7241637 0.3396654
#&gt;         FOR        TS       STP       ACC        F1       MCC
#&gt; 1 0.3001712 0.3960287 0.6602397 0.2868008 0.3094422 0.5158728
#&gt; 2 0.8675808 0.3203605 0.7852645 0.1353720 0.2502984 0.6709033
#&gt; 3 0.4628820 0.4898821 0.6068959 0.3528121 0.3575490 0.7024079
#&gt; 4 0.3647346 0.3814321 0.2951150 0.1960918 0.3733294 0.3505547
#&gt; 5 0.5660787 0.3126692 0.6052424 0.1592532 0.2426811 0.5349499
#&gt; 6 0.4492735 0.5204270 0.6776752 0.3510521 0.3845279 0.6715986</code></pre>
<ul>
<li>groups_data</li>
</ul>
<p>Fairness object gets metrics based on confusion matrix and checks them over the groups.</p>
<div class="sourceCode" id="cb16"><html><body><pre class="r"><span class="co"># for the first model</span>
<span class="no">fobject</span>$<span class="no">groups_data</span>$<span class="no">ranger</span>$<span class="no">TPR</span></pre></body></html></div>
<pre><code>#&gt; African_American            Asian        Caucasian         Hispanic 
#&gt;        0.6129458        0.8695652        0.7533177        0.8062500 
#&gt;  Native_American            Other 
#&gt;        0.6666667        0.8447489</code></pre>
<p>It is simply metrics for certain subgroup.</p>
<p>What is relation between <code>$groups_data</code> and <code>$parity_loss_metric_data</code> ?</p>
<p>If we were going only to take score from certain metric (Let’s say fpr and 0.3) we wouldn’t know if it is good or bad. But we are aiming for <strong>equal treatment over all groups</strong> so if this metric score would be the same in all groups it would be very good. But the metrics wouldn’t be comparable between each others (fpr - 0.3 in all groups and accuracy - 0.9 in all groups, both are good in terms of parity). That is why we use <strong>privileged</strong> - to set benchmark. And for example Caucasian in fpr had score of 0.3 and African American 0.6. After setting <code>privilieged = Caucasian</code> Caucasian would have score 0, and African American 0.3, because is the distance between those metrics.</p>
<p><em>Note: When dealing with aggregating plots we use formula sum(abs(1-score)) to represent aggregated score in metrics. In short is how much it differs from ideal scores.</em></p>
<ul>
<li>explainers</li>
</ul>
<p>list of <code>DALEX</code> explainers</p>
<ul>
<li>cutoff</li>
</ul>
<div class="sourceCode" id="cb18"><html><body><pre class="r"><span class="co"># for first model</span>
<span class="no">fobject</span>$<span class="no">cutoff</span>$<span class="no">ranger</span></pre></body></html></div>
<pre><code>#&gt; $African_American
#&gt; [1] 0.5
#&gt; 
#&gt; $Asian
#&gt; [1] 0.5
#&gt; 
#&gt; $Caucasian
#&gt; [1] 0.5
#&gt; 
#&gt; $Hispanic
#&gt; [1] 0.5
#&gt; 
#&gt; $Native_American
#&gt; [1] 0.5
#&gt; 
#&gt; $Other
#&gt; [1] 0.5</code></pre>
<p>list of cutoff values for each model</p>
<ul>
<li><p>fairness_check_data data used in print and plot of <code>fairness_object</code>. It is already processed data and ready to plot. If someone were to use <code><a href="https://rdrr.io/r/base/MathFun.html">abs()</a></code> metrics there would be equal to particular metrics in <code>$parity_loss_metric_data</code>. It means that it allows negative values. So when value is negative it means that score of privileged group in this metric was better.</p></li>
<li><p>… - other parameters passed to <code><a href="../reference/fairness_check.html">fairness_check()</a></code></p></li>
</ul>
</div>
</div>
</div>
<div id="choosing-best-model" class="section level1">
<h1 class="hasAnchor">
<a href="#choosing-best-model" class="anchor"></a>Choosing best model</h1>
<p>We now have a few models in our <code>fairness_object</code></p>
<p>Let’s see how they perform in different metrics.</p>
<div id="stacked-barplot" class="section level2">
<h2 class="hasAnchor">
<a href="#stacked-barplot" class="anchor"></a>Stacked Barplot</h2>
<div class="sourceCode" id="cb20"><html><body><pre class="r"><span class="no">sm</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/plot_stacked_barplot.html">stack_metrics</a></span>(<span class="no">fobject</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">sm</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-14-1.png" width="720" style="display: block; margin: auto;"></p>
<p>It displays accumulated (Stacked) metric scores for each model. The least metric score the better.</p>
</div>
<div id="plot-metric" class="section level2">
<h2 class="hasAnchor">
<a href="#plot-metric" class="anchor"></a>Plot metric</h2>
<div class="sourceCode" id="cb21"><html><body><pre class="r"><span class="no">cm</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/choose_metric.html">choose_metric</a></span>(<span class="no">fobject</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">cm</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-15-1.png" width="720" style="display: block; margin: auto;"></p>
</div>
<div id="plot-fairness-pca" class="section level2">
<h2 class="hasAnchor">
<a href="#plot-fairness-pca" class="anchor"></a>Plot fairness PCA</h2>
<p>With this task we should use <code>PCA</code>. We call <code>create_fairness_pca()</code> to create fairness pca object.</p>
<div class="sourceCode" id="cb22"><html><body><pre class="r"><span class="no">fair_pca</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_pca.html">fairness_pca</a></span>(<span class="no">fobject</span>)
<span class="fu"><a href="https://rdrr.io/r/base/print.html">print</a></span>(<span class="no">fair_pca</span>)</pre></body></html></div>
<pre><code>#&gt; Fairness PCA : 
#&gt;             PC1         PC2         PC3        PC4          PC5           PC6
#&gt; [1,]  0.6034799 -0.29144784  1.80696285  0.4312988  0.106808580  5.551115e-17
#&gt; [2,] -3.5924642 -0.07210154  0.09615046 -0.6863582  0.084391975  5.551115e-16
#&gt; [3,]  1.0390695 -1.35656039 -1.22389274  0.2900697  0.231943137  4.996004e-16
#&gt; [4,]  2.5776811  2.23956785 -0.13490121 -0.5736394  0.008726504 -8.881784e-16
#&gt; [5,] -1.6094959  1.51916647 -0.54366847  0.8017552 -0.154057950 -1.637579e-15
#&gt; [6,]  0.9817296 -2.03862455 -0.00065089 -0.2631262 -0.277812247 -1.831868e-15
#&gt; 
#&gt; Created with: 
#&gt; [1] "ranger"   "lm"       "ranger_2" "ranger_3" "gbm"      "ranger_4"
#&gt; 
#&gt; First two components explained 85 % of variance.</code></pre>
<p>Let’s plot!</p>
<div class="sourceCode" id="cb24"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fair_pca</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-17-1.png" width="720" style="display: block; margin: auto;"></p>
</div>
<div id="plot-heatmap" class="section level2">
<h2 class="hasAnchor">
<a href="#plot-heatmap" class="anchor"></a>Plot Heatmap</h2>
<p>Another way to deal with grouped data is using heatmap.</p>
<div class="sourceCode" id="cb25"><html><body><pre class="r"><span class="no">fheatmap</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_heatmap.html">fairness_heatmap</a></span>(<span class="no">fobject</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fheatmap</span>, <span class="kw">text_size</span> <span class="kw">=</span> <span class="fl">3</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-18-1.png" width="720" style="display: block; margin: auto;"></p>
<p>For both models and metrics dendograms are created. This way through hierarchical clustering we can look on similarities between models/metrics. It should give similar but more detailed information than PCA Now we know what those scores are and how “similar” models are to each other</p>
</div>
<div id="metric-and-performance-plot" class="section level2">
<h2 class="hasAnchor">
<a href="#metric-and-performance-plot" class="anchor"></a>Metric and Performance Plot</h2>
<p>Sometimes we would like to know how good are models in performance metrics and in fairness metrics at the same time, to see the tradeoff between them.</p>
<div class="sourceCode" id="cb26"><html><body><pre class="r"><span class="no">fap</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/performance_and_fairness.html">performance_and_fairness</a></span>(<span class="no">fobject</span>, <span class="kw">fairness_metric</span> <span class="kw">=</span> <span class="st">"FPR"</span>)</pre></body></html></div>
<pre><code>#&gt; Performace metric is NULL, setting deafult ( accuracy )  
#&gt; 
#&gt; Creating object with: 
#&gt; Fairness metric: FPR 
#&gt; Performance metric: accuracy</code></pre>
<div class="sourceCode" id="cb28"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fap</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-19-1.png" width="720" style="display: block; margin: auto;"></p>
</div>
<div id="group-metric" class="section level2">
<h2 class="hasAnchor">
<a href="#group-metric" class="anchor"></a>Group Metric</h2>
<p>When we have narrowed down our search for the best model we can use <code>group_metric</code> to check once again metrics within groups and decide which model to use.</p>
<div class="sourceCode" id="cb29"><html><body><pre class="r"><span class="no">fobject2</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">explainer_1</span>,<span class="no">explainer_2</span>,
                                   <span class="kw">protected</span> <span class="kw">=</span> <span class="no">compas</span>$<span class="no">Ethnicity</span>,
                                   <span class="kw">privileged</span> <span class="kw">=</span> <span class="st">"Caucasian"</span>,
                                    <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)


<span class="no">gm</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/group_metric.html">group_metric</a></span>(<span class="no">fobject2</span>, <span class="kw">fairness_metric</span> <span class="kw">=</span> <span class="st">"FPR"</span>)</pre></body></html></div>
<pre><code>#&gt; Performace metric not given, setting deafult ( accuracy )  
#&gt; 
#&gt; Creating object with: 
#&gt; Fairness metric:  FPR 
#&gt; Performance metric:  accuracy</code></pre>
<div class="sourceCode" id="cb31"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">gm</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-20-1.png" width="720" style="display: block; margin: auto;"></p>
</div>
<div id="radar-plot" class="section level2">
<h2 class="hasAnchor">
<a href="#radar-plot" class="anchor"></a>Radar plot</h2>
<div class="sourceCode" id="cb32"><html><body><pre class="r"><span class="no">fradar</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_radar.html">fairness_radar</a></span>(<span class="no">fobject2</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fradar</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-21-1.png" width="720" style="display: block; margin: auto;"></p>
</div>
<div id="custom-cutoff" class="section level2">
<h2 class="hasAnchor">
<a href="#custom-cutoff" class="anchor"></a>Custom cutoff</h2>
<p>We may see how cutoff affects parity loss of metrics</p>
<div id="all-cutoffs" class="section level3">
<h3 class="hasAnchor">
<a href="#all-cutoffs" class="anchor"></a>All cutoffs</h3>
<p>All cutoffs measures where metrics exist (are not NA) and how they change if we modify cutoffs in all subgroups.</p>
<p>In this plot <code>NA</code> values are natural, so warnings are to be expected.</p>
<div class="sourceCode" id="cb33"><html><body><pre class="r"><span class="no">ac</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/all_cutoffs.html">all_cutoffs</a></span>(<span class="no">fobject2</span>)

<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">ac</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-22-1.png" width="720" style="display: block; margin: auto;"></p>
</div>
<div id="ceteris-paribus-cutoff" class="section level3">
<h3 class="hasAnchor">
<a href="#ceteris-paribus-cutoff" class="anchor"></a>Ceteris paribus cutoff</h3>
<p>This function shows how parity loss metrics would change if we modified cutoff only for one subgroup (here African American) with other cutoffs fixed.</p>
<div class="sourceCode" id="cb34"><html><body><pre class="r"><span class="no">cpc</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/ceteris_paribus_cutoff.html">ceteris_paribus_cutoff</a></span>(<span class="no">fobject2</span>, <span class="kw">subgroup</span> <span class="kw">=</span> <span class="st">"African_American"</span>)

<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">cpc</span>)</pre></body></html></div>
<p><img src="Basic_tutorial_files/figure-html/unnamed-chunk-23-1.png" width="720" style="display: block; margin: auto;"></p>
</div>
</div>
</div>
<div id="summary" class="section level1">
<h1 class="hasAnchor">
<a href="#summary" class="anchor"></a>Summary</h1>
<p><code>fairmodels</code> is powerful and flexible tool for detecting bias and asserting fairness in models. Many metrics are available (there are tools to create own metrics based on confusion matrix). Besides bias detection <code>fairmodels</code> offers also bias mitigation. It is set of pre-processing and post-processing algorithms working either on data or on explainer which function is to lower parity loss in certain metrics. If you are interested in this and adult data case study, please check Advanced Tutorial. If you encountered a bug or you have cool idea for a new feature please write and issue <a href="https://github.com/ModelOriented/fairmodels/issues">here</a>.</p>
</div>
  </div>

  <div class="col-md-3 hidden-xs hidden-sm" id="pkgdown-sidebar">

        <nav id="toc" data-toggle="toc"><h2 data-toc-skip>Contents</h2>
    </nav>
</div>

</div>



      <footer><div class="copyright">
  <p>Developed by Jakub Wiśniewski, Przemysław Biecek.</p>
</div>

<div class="pkgdown">
  <p>Site built with <a href="https://pkgdown.r-lib.org/">pkgdown</a>.</p>
</div>

      </footer>
</div>

  


  </body>
</html>
