<!DOCTYPE html>
<!-- Generated by pkgdown: do not edit by hand --><html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Advanced Tutorial • fairmodels</title>
<!-- favicons --><link rel="icon" type="image/png" sizes="16x16" href="../favicon-16x16.png">
<link rel="icon" type="image/png" sizes="32x32" href="../favicon-32x32.png">
<link rel="apple-touch-icon" type="image/png" sizes="180x180" href="../apple-touch-icon.png">
<link rel="apple-touch-icon" type="image/png" sizes="120x120" href="../apple-touch-icon-120x120.png">
<link rel="apple-touch-icon" type="image/png" sizes="76x76" href="../apple-touch-icon-76x76.png">
<link rel="apple-touch-icon" type="image/png" sizes="60x60" href="../apple-touch-icon-60x60.png">
<!-- jquery --><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script><!-- Bootstrap --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/css/bootstrap.min.css" integrity="sha256-bZLfwXAP04zRMK2BjiO8iu9pf4FbLqX6zitd+tIvLhE=" crossorigin="anonymous">
<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/js/bootstrap.min.js" integrity="sha256-nuL8/2cJ5NDSSwnKD8VqreErSWHtnEP9E7AySL+1ev4=" crossorigin="anonymous"></script><!-- bootstrap-toc --><link rel="stylesheet" href="../bootstrap-toc.css">
<script src="../bootstrap-toc.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous">
<!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- dalexverse --><link href="../dalexverse.css" rel="stylesheet">
<link href="../dalexverse-2.css" rel="stylesheet">
<!-- pkgdown --><link href="../pkgdown.css" rel="stylesheet">
<script src="../pkgdown.js"></script><meta property="og:title" content="Advanced Tutorial">
<meta property="og:description" content="fairmodels">
<meta property="og:image" content="/logo.png">
<meta name="robots" content="noindex">
<!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]--><!-- google analytics --><script async src="https://www.googletagmanager.com/gtag/js?id=UA-5650686-14"></script><script>
 window.dataLayer = window.dataLayer || [];
 function gtag(){dataLayer.push(arguments);}
 gtag('js', new Date());

 gtag('config', 'UA-5650686-14');
</script>
</head>
<body data-spy="scroll" data-target="#toc">
    <div class="container template-article">
      <header><div class="navbar navbar-default navbar-fixed-top" role="navigation">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false">
        <span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>

      <div class="navbar-brand-container">
        <a class="navbar-brand" href="../index.html">fairmodels</a>
        <div class="info">
          <span class="partof">part of the <a href="https://github.com/ModelOriented/DrWhy">DrWhy.AI</a>
           developed by the <a href="https://mi2.mini.pw.edu.pl/">MI^2 DataLab</a> </span>
          <span class="version version-default">0.1.1</span>
        </div>
      </div>
    </div>
    <div id="navbar" class="navbar-collapse collapse">
      <ul class="nav navbar-nav navbar-right">
<li>
  <a href="../index.html">
    <span class="fas fa fas fa-home fa-lg"></span>
     
  </a>
</li>
<li>
  <a href="../reference/index.html">Reference</a>
</li>
<li class="dropdown">
  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
    Articles
     
    <span class="caret"></span>
  </a>
  <ul class="dropdown-menu" role="menu">
<li>
      <a href="../articles/Advanced_tutorial.html">Advanced Tutorial</a>
    </li>
    <li>
      <a href="../articles/Basic_tutorial.html">Basic Tutorial</a>
    </li>
  </ul>
</li>
<li>
  <a href="../news/index.html">Changelog</a>
</li>
        <li>
  <a href="https://github.com/ModelOriented/fairmodels/">
    <span class="fab fa fab fa-github fa-lg"></span>
     
  </a>
</li>
      </ul>
</div>
<!--/.nav-collapse -->
  </div>
<!--/.container -->
</div>
<!--/.navbar -->

      

      </header><script src="Advanced_tutorial_files/header-attrs-2.3/header-attrs.js"></script><script src="Advanced_tutorial_files/accessible-code-block-0.0.1/empty-anchor.js"></script><div class="row">
  <div class="col-md-9 contents">
    <div class="page-header toc-ignore">
      <h1 data-toc-skip>Advanced Tutorial</h1>
                        <h4 class="author">Jakub Wiśniewski</h4>
            
      
      <small class="dont-index">Source: <a href="https://github.com/ModelOriented/fairmodels/blob/master/vignettes/Advanced_tutorial.Rmd"><code>vignettes/Advanced_tutorial.Rmd</code></a></small>
      <div class="hidden name"><code>Advanced_tutorial.Rmd</code></div>

    </div>

    
    
<div id="model-explanation-bias" class="section level1">
<h1 class="hasAnchor">
<a href="#model-explanation-bias" class="anchor"></a>Model, explanation &amp; bias</h1>
<p>In this tutorial you will learn how to tackle bias using the bias mitigation techniques supported by <code>fairmodels</code>. As always we will start from the data.</p>
<div class="sourceCode" id="cb1"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">fairmodels</span>)

<span class="fu"><a href="https://rdrr.io/r/utils/data.html">data</a></span>(<span class="st">"adult"</span>)
<span class="fu"><a href="https://rdrr.io/r/utils/head.html">head</a></span>(<span class="no">adult</span>)</pre></body></html></div>
<pre><code>#&gt;   salary age        workclass fnlwgt education education_num     marital_status
#&gt; 1  &lt;=50K  39        State-gov  77516 Bachelors            13      Never-married
#&gt; 2  &lt;=50K  50 Self-emp-not-inc  83311 Bachelors            13 Married-civ-spouse
#&gt; 3  &lt;=50K  38          Private 215646   HS-grad             9           Divorced
#&gt; 4  &lt;=50K  53          Private 234721      11th             7 Married-civ-spouse
#&gt; 5  &lt;=50K  28          Private 338409 Bachelors            13 Married-civ-spouse
#&gt; 6  &lt;=50K  37          Private 284582   Masters            14 Married-civ-spouse
#&gt;          occupation  relationship  race    sex capital_gain capital_loss
#&gt; 1      Adm-clerical Not-in-family White   Male         2174            0
#&gt; 2   Exec-managerial       Husband White   Male            0            0
#&gt; 3 Handlers-cleaners Not-in-family White   Male            0            0
#&gt; 4 Handlers-cleaners       Husband Black   Male            0            0
#&gt; 5    Prof-specialty          Wife Black Female            0            0
#&gt; 6   Exec-managerial          Wife White Female            0            0
#&gt;   hours_per_week native_country
#&gt; 1             40  United-States
#&gt; 2             13  United-States
#&gt; 3             40  United-States
#&gt; 4             40  United-States
#&gt; 5             40           Cuba
#&gt; 6             40  United-States</code></pre>
<div class="sourceCode" id="cb3"><html><body><pre class="r"># for this vignette data will be truncated</pre></body></html></div>
<p>We will use adult data to predict whether certain person has yearly salary exceeding 50 000 or not. Our protected variable will be sex. For this tutorial we will be using <code>gbm</code> and of course we will explain it with <code>DALEX</code>.</p>
<div class="sourceCode" id="cb4"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">gbm</span>)
<span class="fu"><a href="https://rdrr.io/r/base/library.html">library</a></span>(<span class="no">DALEX</span>)

<span class="no">adult</span>$<span class="no">salary</span>   <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span>(<span class="no">adult</span>$<span class="no">salary</span>) -<span class="fl">1</span> <span class="co"># 0 if bad and 1 if good risk</span>
<span class="no">protected</span>     <span class="kw">&lt;-</span> <span class="no">adult</span>$<span class="no">sex</span>
<span class="no">adult</span> <span class="kw">&lt;-</span> <span class="no">adult</span>[<span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span>(<span class="no">adult</span>) <span class="kw">!=</span> <span class="st">"sex"</span>] <span class="co"># sex not specified</span>

<span class="co"># making model</span>
<span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span>(<span class="fl">1</span>)
<span class="no">gbm_model</span> <span class="kw">&lt;-</span><span class="fu"><a href="https://rdrr.io/pkg/gbm/man/gbm.html">gbm</a></span>(<span class="no">salary</span> ~<span class="no">.</span> , <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>, <span class="kw">distribution</span> <span class="kw">=</span> <span class="st">"bernoulli"</span>)

<span class="co"># making explainer</span>
<span class="no">gbm_explainer</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">gbm_model</span>,
                         <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>[,-<span class="fl">1</span>],
                         <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>,
                         <span class="kw">colorize</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<pre><code>#&gt; Preparation of a new explainer is initiated
#&gt;   -&gt; model label       :  gbm  (  default  )
#&gt;   -&gt; data              :  32561  rows  13  cols 
#&gt;   -&gt; target variable   :  32561  values 
#&gt;   -&gt; predict function  :  yhat.gbm  will be used (  default  )
#&gt;   -&gt; predicted values  :  numerical, min =  0.0101498 , mean =  0.2409542 , max =  0.9864558  
#&gt;   -&gt; model_info        :  package gbm , ver. 2.1.5 , task classification (  default  ) 
#&gt;   -&gt; residual function :  difference between y and yhat (  default  )
#&gt;   -&gt; residuals         :  numerical, min =  -0.9790795 , mean =  -0.0001445991 , max =  0.9864904  
#&gt;   A new explainer has been created!</code></pre>
<div class="sourceCode" id="cb6"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/model_performance.html">model_performance</a></span>(<span class="no">gbm_explainer</span>)</pre></body></html></div>
<pre><code>#&gt; Measures for:  classification
#&gt; recall     : 0.5353909 
#&gt; precision  : 0.7999238 
#&gt; f1         : 0.6414547 
#&gt; accuracy   : 0.8558705 
#&gt; auc        : 0.9093789
#&gt; 
#&gt; Residuals:
#&gt;          0%         10%         20%         30%         40%         50% 
#&gt; -0.97907954 -0.31512711 -0.20921106 -0.12255213 -0.06480941 -0.04122486 
#&gt;         60%         70%         80%         90%        100% 
#&gt; -0.02832005 -0.01740186  0.13344943  0.53676046  0.98649038</code></pre>
<p>Our model has around 86% accuracy. And how about bias? Sex is our protected variable and we should suspect that men will be more frequently assigned better annual income.</p>
<div class="sourceCode" id="cb8"><html><body><pre class="r"><span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">gbm_explainer</span>,
                          <span class="kw">protected</span>  <span class="kw">=</span> <span class="no">protected</span>,
                          <span class="kw">privileged</span> <span class="kw">=</span> <span class="st">"Male"</span>,
                          <span class="kw">colorize</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<pre><code>#&gt; Creating fairness object
#&gt; -&gt; Privileged subgroup       : character ( Ok  )
#&gt; -&gt; Protected variable        : factor ( Ok  ) 
#&gt; -&gt; Cutoff values for explainers  : 0.5 ( for all subgroups ) 
#&gt; -&gt; Fairness objects      : 0 objects 
#&gt; -&gt; Checking explainers       : 1 in total (  compatible  )
#&gt; -&gt; Metric calculation        : 13/13 metrics calculated for all models
#&gt;  Fairness object created succesfully</code></pre>
<div class="sourceCode" id="cb10"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/print.html">print</a></span>(<span class="no">fobject</span>, <span class="kw">colorize</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<pre><code>#&gt; 
#&gt; Fairness check for models: gbm 
#&gt; 
#&gt; gbm passes 2/5 metrics
#&gt; Total loss:  0.504282</code></pre>
<p>Our model passes only few metrics, how big is the bias?</p>
<div class="sourceCode" id="cb12"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-6-1.png" width="720" style="display: block; margin: auto;"></p>
<p>The biggest bias is in <code>Statistical parity loss</code> metric. It is metric that is frequently look at because it gives answer how much difference is there in positive label rates in model within protected variable. Let’s say that it will be metric that we will try to mitigate.</p>
</div>
<div id="bias-mitigation-strategies" class="section level1">
<h1 class="hasAnchor">
<a href="#bias-mitigation-strategies" class="anchor"></a>Bias mitigation strategies</h1>
<div id="pre-processing-techniques" class="section level2">
<h2 class="hasAnchor">
<a href="#pre-processing-techniques" class="anchor"></a>Pre-processing techniques</h2>
<p>Pre-processing techniques focus on changing data <strong>before</strong> model is trained. This reduces bias in data.</p>
<div id="distribution-changing" class="section level3">
<h3 class="hasAnchor">
<a href="#distribution-changing" class="anchor"></a>Distribution changing</h3>
<p>Firs technique you will learn about is <code>disparate_impact_remover</code>. It is somehow limited as it works on ordinal, numeric data. This technique returns “fixed” data frame. Through parameter <code>lambda</code> we can manipulate with how much the distribution will be fixed. <code>lambda = 1</code> (default) will return data with identical distributions for all levels of protected variable whereas <code>lambda = 0</code> will barely change anything. We will transform a few features.</p>
<div class="sourceCode" id="cb13"><html><body><pre class="r"><span class="no">data_fixed</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/disparate_impact_remover.html">disparate_impact_remover</a></span>(<span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>, <span class="kw">protected</span> <span class="kw">=</span> <span class="no">protected</span>,
                            <span class="kw">features_to_transform</span> <span class="kw">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span>(<span class="st">"age"</span>, <span class="st">"hours_per_week"</span>,
                                                      <span class="st">"capital_loss"</span>,
                                                      <span class="st">"capital_gain"</span>))

<span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span>(<span class="fl">1</span>)
<span class="no">gbm_model</span>     <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/gbm/man/gbm.html">gbm</a></span>(<span class="no">salary</span> ~<span class="no">.</span> , <span class="kw">data</span> <span class="kw">=</span> <span class="no">data_fixed</span>, <span class="kw">distribution</span> <span class="kw">=</span> <span class="st">"bernoulli"</span>)
<span class="no">gbm_explainer_dir</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">gbm_model</span>,
                             <span class="kw">data</span> <span class="kw">=</span> <span class="no">data_fixed</span>[,-<span class="fl">1</span>],
                             <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>,
                             <span class="kw">label</span> <span class="kw">=</span> <span class="st">"gbm_dir"</span>,
                             <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<p>Now we will compare old explainer and new one.</p>
<div class="sourceCode" id="cb14"><html><body><pre class="r"><span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">gbm_explainer</span>, <span class="no">gbm_explainer_dir</span>,
                          <span class="kw">protected</span> <span class="kw">=</span> <span class="no">protected</span>,
                          <span class="kw">privileged</span> <span class="kw">=</span> <span class="st">"Male"</span>,
                          <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-8-1.png" width="720" style="display: block; margin: auto;"></p>
<p>As we can see bias has diminished. But it is not on acceptable level for us. Let’s try something else. As for now we will add explainers to existing <code>fairness_object</code> to compare methods among themselves.</p>
</div>
<div id="reweighting" class="section level3">
<h3 class="hasAnchor">
<a href="#reweighting" class="anchor"></a>Reweighting</h3>
<p>Reweighting is straightforward bias mitigation technique. It produces weights based on data to pass them to model so it can learn what to be careful about. As there can be multiple subgroups weights will come in form of vector.</p>
<div class="sourceCode" id="cb15"><html><body><pre class="r"><span class="no">weights</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/reweight.html">reweight</a></span>(<span class="kw">protected</span> <span class="kw">=</span> <span class="no">protected</span>, <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>)

<span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span>(<span class="fl">1</span>)
<span class="no">gbm_model</span>     <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/gbm/man/gbm.html">gbm</a></span>(<span class="no">salary</span> ~<span class="no">.</span> ,
                     <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>,
                     <span class="kw">weights</span> <span class="kw">=</span> <span class="no">weights</span>,
                     <span class="kw">distribution</span> <span class="kw">=</span> <span class="st">"bernoulli"</span>)

<span class="no">gbm_explainer_w</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">gbm_model</span>,
                           <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>[,-<span class="fl">1</span>],
                           <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>,
                           <span class="kw">label</span> <span class="kw">=</span> <span class="st">"gbm_weighted"</span>,
                           <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">fobject</span>, <span class="no">gbm_explainer_w</span>, <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-9-1.png" width="720" style="display: block; margin: auto;"></p>
<p>Our metric of interest (<code>Statistical parity loss</code>) has been diminished but in this process we exceeded acceptable limit for predictive parity loss. It is not something uncommon. To lower the statistical parity the model must classify some unfavorable cases from unprivileged subgroup as favorable and the opposite for privileged subgroup. Similar outcome will be visible in the next method</p>
</div>
<div id="resampling" class="section level3">
<h3 class="hasAnchor">
<a href="#resampling" class="anchor"></a>Resampling</h3>
<p>This method derives from reweighting the data but instead of weights it chooses observations from data the outcome of metrics. There is 2 types of resampling: 1. <strong>uniform</strong> - takes random observation from particular subgroup (in particular case - y == 1 or y == 0) 2. <strong>preferential</strong> - takes/omits observations either close to cutoff or as far from cutoff as possible. It needs probabilities (<code>probs</code>)</p>
<div class="sourceCode" id="cb16"><html><body><pre class="r"><span class="co"># to obtain probs we will use simple linear regression</span>
<span class="no">probs</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span>(<span class="no">salary</span> ~<span class="no">.</span>, <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>, <span class="kw">family</span> <span class="kw">=</span> <span class="fu"><a href="https://rdrr.io/r/stats/family.html">binomial</a></span>())$<span class="no">fitted.values</span>

<span class="no">uniform_indexes</span>      <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/resample.html">resample</a></span>(<span class="kw">protected</span> <span class="kw">=</span> <span class="no">protected</span>,
                                 <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>)
<span class="no">preferential_indexes</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/resample.html">resample</a></span>(<span class="kw">protected</span> <span class="kw">=</span> <span class="no">protected</span>,
                                 <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>,
                                 <span class="kw">type</span> <span class="kw">=</span> <span class="st">"preferential"</span>,
                                 <span class="kw">probs</span> <span class="kw">=</span> <span class="no">probs</span>)

<span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span>(<span class="fl">1</span>)
<span class="no">gbm_model</span>     <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/gbm/man/gbm.html">gbm</a></span>(<span class="no">salary</span> ~<span class="no">.</span> ,
                     <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>[<span class="no">uniform_indexes</span>,],
                     <span class="kw">distribution</span> <span class="kw">=</span> <span class="st">"bernoulli"</span>)

<span class="no">gbm_explainer_u</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">gbm_model</span>,
                           <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>[,-<span class="fl">1</span>],
                           <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>,
                           <span class="kw">label</span> <span class="kw">=</span> <span class="st">"gbm_uniform"</span>,
                           <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span>(<span class="fl">1</span>)
<span class="no">gbm_model</span>     <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/gbm/man/gbm.html">gbm</a></span>(<span class="no">salary</span> ~<span class="no">.</span> ,
                     <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>[<span class="no">preferential_indexes</span>,],
                     <span class="kw">distribution</span> <span class="kw">=</span> <span class="st">"bernoulli"</span>)

<span class="no">gbm_explainer_p</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">gbm_model</span>,
                           <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>[,-<span class="fl">1</span>],
                           <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>,
                           <span class="kw">label</span> <span class="kw">=</span> <span class="st">"gbm_preferential"</span>,
                           <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">fobject</span>, <span class="no">gbm_explainer_u</span>, <span class="no">gbm_explainer_p</span>,
                          <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)
<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-10-1.png" width="720" style="display: block; margin: auto;"></p>
<p>As we can see <code>preferential</code> sampling is best at mitigating statistical parity loss. As byproduct it also affects other metrics significantly.</p>
</div>
</div>
<div id="post-processing-techniques" class="section level2">
<h2 class="hasAnchor">
<a href="#post-processing-techniques" class="anchor"></a>Post-processing techniques</h2>
<p>Post-processing techniques focus on changing the output of model after model is generated</p>
<div id="roc-pivot" class="section level3">
<h3 class="hasAnchor">
<a href="#roc-pivot" class="anchor"></a>ROC pivot</h3>
<p>ROC stands for Reject Option based Classification and pivot for the behavior of <code>probs</code>/<code>y_hat</code> as it pivots around the cutoff when it is in close zone. Close zone is defined by the <code>theta</code> parameter equal to 0.1 by default. In simple words depending on whether subgroup is privileged or not certain observations will change their probabilities to the opposite (but in equal distance) side of cutoff. For example if observation is unprivileged and unfavorable with probability 0.45 (assuming the <code>cutoff = 0.5</code>) probability will change it’s value to 0.55.</p>
<div class="sourceCode" id="cb17"><html><body><pre class="r"><span class="co"># we will need normal explainer </span>
<span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span>(<span class="fl">1</span>)
<span class="no">gbm_model</span> <span class="kw">&lt;-</span><span class="fu"><a href="https://rdrr.io/pkg/gbm/man/gbm.html">gbm</a></span>(<span class="no">salary</span> ~<span class="no">.</span> , <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>, <span class="kw">distribution</span> <span class="kw">=</span> <span class="st">"bernoulli"</span>)
<span class="no">gbm_explainer</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">gbm_model</span>,
                         <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>[,-<span class="fl">1</span>],
                         <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>,
                         <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="no">gbm_explainer_r</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/roc_pivot.html">roc_pivot</a></span>(<span class="no">gbm_explainer</span>,
                             <span class="kw">protected</span> <span class="kw">=</span> <span class="no">protected</span>,
                             <span class="kw">privileged</span> <span class="kw">=</span> <span class="st">"Male"</span>)


<span class="no">fobject</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">fobject</span>, <span class="no">gbm_explainer_r</span>,
                          <span class="kw">label</span> <span class="kw">=</span> <span class="st">"gbm_roc"</span>,  <span class="co"># label as vector for explainers</span>
                          <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fobject</span>)</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-11-1.png" width="720" style="display: block; margin: auto;"></p>
<p><code>gmb_roc</code> is the best so far</p>
<div class="sourceCode" id="cb18"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/print.html">print</a></span>(<span class="no">fobject</span>, <span class="kw">colorize</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<pre><code>#&gt; 
#&gt; Fairness check for models: gbm_roc, gbm_uniform, gbm_preferential, gbm_weighted, gbm, gbm_dir 
#&gt; 
#&gt; gbm_roc passes 4/5 metrics
#&gt; Total loss:  0.3675947 
#&gt; 
#&gt; gbm_uniform passes 2/5 metrics
#&gt; Total loss:  0.5125624 
#&gt; 
#&gt; gbm_preferential passes 3/5 metrics
#&gt; Total loss:  0.7593104 
#&gt; 
#&gt; gbm_weighted passes 2/5 metrics
#&gt; Total loss:  0.4682607 
#&gt; 
#&gt; gbm passes 2/5 metrics
#&gt; Total loss:  0.504282 
#&gt; 
#&gt; gbm_dir passes 2/5 metrics
#&gt; Total loss:  0.4556241</code></pre>
</div>
<div id="cutoff-manipulation" class="section level3">
<h3 class="hasAnchor">
<a href="#cutoff-manipulation" class="anchor"></a>Cutoff manipulation</h3>
<p>Cutoff manipulation is simple technique that enables to set different cutoff for different subgroups.</p>
<div class="sourceCode" id="cb20"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span>(<span class="fl">1</span>)
<span class="no">gbm_model</span> <span class="kw">&lt;-</span><span class="fu"><a href="https://rdrr.io/pkg/gbm/man/gbm.html">gbm</a></span>(<span class="no">salary</span> ~<span class="no">.</span> , <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>, <span class="kw">distribution</span> <span class="kw">=</span> <span class="st">"bernoulli"</span>)
<span class="no">gbm_explainer</span> <span class="kw">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/DALEX/man/explain.html">explain</a></span>(<span class="no">gbm_model</span>,
                         <span class="kw">data</span> <span class="kw">=</span> <span class="no">adult</span>[,-<span class="fl">1</span>],
                         <span class="kw">y</span> <span class="kw">=</span> <span class="no">adult</span>$<span class="no">salary</span>,
                         <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="co"># test fairness object</span>
<span class="no">fobject_test</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">gbm_explainer</span>,
                          <span class="kw">protected</span> <span class="kw">=</span> <span class="no">protected</span>,
                          <span class="kw">privileged</span> <span class="kw">=</span> <span class="st">"Male"</span>,
                          <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="fu"><a href="../reference/ceteris_paribus_cutoff.html">ceteris_paribus_cutoff</a></span>(<span class="no">fobject_test</span>, <span class="kw">subgroup</span> <span class="kw">=</span> <span class="st">"Female"</span>))</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-13-1.png" width="720" style="display: block; margin: auto;"></p>
<p>It is possible to minimize all metrics or only few metrics of our interest. In this case it is also 0.35 but feel free to experiment!</p>
<div class="sourceCode" id="cb21"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="fu"><a href="../reference/ceteris_paribus_cutoff.html">ceteris_paribus_cutoff</a></span>(<span class="no">fobject_test</span>,
                            <span class="kw">subgroup</span> <span class="kw">=</span> <span class="st">"Female"</span>,
                            <span class="kw">fairness_metrics</span> <span class="kw">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span>(<span class="st">"ACC"</span>,<span class="st">"TPR"</span>,<span class="st">"STP"</span>)))</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-14-1.png" width="720" style="display: block; margin: auto;"></p>
<div class="sourceCode" id="cb22"><html><body><pre class="r"><span class="no">fc</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/fairness_check.html">fairness_check</a></span>(<span class="no">gbm_explainer</span>, <span class="no">fobject</span>,
                     <span class="kw">label</span> <span class="kw">=</span> <span class="st">"gbm_cutoff"</span>,
                     <span class="kw">cutoff</span> <span class="kw">=</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html">list</a></span>(<span class="kw">Female</span> <span class="kw">=</span> <span class="fl">0.35</span>),
                     <span class="kw">verbose</span> <span class="kw">=</span> <span class="fl">FALSE</span>)

<span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">fc</span>)</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-15-1.png" width="720" style="display: block; margin: auto;"></p>
<div class="sourceCode" id="cb23"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/print.html">print</a></span>(<span class="no">fc</span> , <span class="kw">colorize</span> <span class="kw">=</span> <span class="fl">FALSE</span>)</pre></body></html></div>
<pre><code>#&gt; 
#&gt; Fairness check for models: gbm_cutoff, gbm_roc, gbm_uniform, gbm_preferential, gbm_weighted, gbm, gbm_dir 
#&gt; 
#&gt; gbm_cutoff passes 3/5 metrics
#&gt; Total loss:  0.3555507 
#&gt; 
#&gt; gbm_roc passes 4/5 metrics
#&gt; Total loss:  0.3675947 
#&gt; 
#&gt; gbm_uniform passes 2/5 metrics
#&gt; Total loss:  0.5125624 
#&gt; 
#&gt; gbm_preferential passes 3/5 metrics
#&gt; Total loss:  0.7593104 
#&gt; 
#&gt; gbm_weighted passes 2/5 metrics
#&gt; Total loss:  0.4682607 
#&gt; 
#&gt; gbm passes 2/5 metrics
#&gt; Total loss:  0.504282 
#&gt; 
#&gt; gbm_dir passes 2/5 metrics
#&gt; Total loss:  0.4556241</code></pre>
</div>
</div>
</div>
<div id="tradeoff-between-bias-and-accuracy" class="section level1">
<h1 class="hasAnchor">
<a href="#tradeoff-between-bias-and-accuracy" class="anchor"></a>Tradeoff between bias and accuracy</h1>
<p>There is significant tradeoff between bias and accuracy one way to visualize it is to use <code>performance_and_fairness</code> function</p>
<div class="sourceCode" id="cb25"><html><body><pre class="r"><span class="no">paf</span> <span class="kw">&lt;-</span> <span class="fu"><a href="../reference/performance_and_fairness.html">performance_and_fairness</a></span>(<span class="no">fc</span>, <span class="kw">fairness_metric</span> <span class="kw">=</span> <span class="st">"STP"</span>,
                                 <span class="kw">performance_metric</span> <span class="kw">=</span> <span class="st">"accuracy"</span>)</pre></body></html></div>
<pre><code>#&gt; 
#&gt; Creating object with: 
#&gt; Fairness metric: STP 
#&gt; Performance metric: accuracy</code></pre>
<div class="sourceCode" id="cb27"><html><body><pre class="r"><span class="fu"><a href="https://rdrr.io/r/base/plot.html">plot</a></span>(<span class="no">paf</span>)</pre></body></html></div>
<p><img src="Advanced_tutorial_files/figure-html/unnamed-chunk-17-1.png" width="720" style="display: block; margin: auto;"></p>
<p>The tradeoff is significant and it should be always be taken into consideration.</p>
</div>
<div id="summary" class="section level1">
<h1 class="hasAnchor">
<a href="#summary" class="anchor"></a>Summary</h1>
<p>There is many ways to tackle bias in models. Thanks to <code>fairmodels</code> it is easy to compare changes and experiment with new models and bias mitigation techniques. It is also good idea to combine few techniques (for example minimizing once with weights and then with cutoff). <code>fairness_check</code> interface is flexible and allows combining models that were trained on different features, encodings etc. Please if you encounter some bug or you have an idea for new feature please write and issue <a href="https://github.com/ModelOriented/fairmodels/issues">here</a>.</p>
</div>
  </div>

  <div class="col-md-3 hidden-xs hidden-sm" id="pkgdown-sidebar">

        <nav id="toc" data-toggle="toc"><h2 data-toc-skip>Contents</h2>
    </nav>
</div>

</div>



      <footer><div class="copyright">
  <p>Developed by Jakub Wiśniewski, Przemysław Biecek.</p>
</div>

<div class="pkgdown">
  <p>Site built with <a href="https://pkgdown.r-lib.org/">pkgdown</a>.</p>
</div>

      </footer>
</div>

  


  </body>
</html>
