<!DOCTYPE html>
<!-- Generated by pkgdown: do not edit by hand --><html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>An introduction to specr • specr</title>
<!-- jquery --><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script><!-- Bootstrap --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/css/bootstrap.min.css" integrity="sha256-bZLfwXAP04zRMK2BjiO8iu9pf4FbLqX6zitd+tIvLhE=" crossorigin="anonymous">
<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/js/bootstrap.min.js" integrity="sha256-nuL8/2cJ5NDSSwnKD8VqreErSWHtnEP9E7AySL+1ev4=" crossorigin="anonymous"></script><!-- bootstrap-toc --><link rel="stylesheet" href="../bootstrap-toc.css">
<script src="../bootstrap-toc.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous">
<!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- pkgdown --><link href="../pkgdown.css" rel="stylesheet">
<script src="../pkgdown.js"></script><meta property="og:title" content="An introduction to specr">
<meta property="og:description" content="specr">
<!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body data-spy="scroll" data-target="#toc">
    

    <div class="container template-article">
      <header><div class="navbar navbar-default navbar-fixed-top" role="navigation">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false">
        <span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>
      <span class="navbar-brand">
        <a class="navbar-link" href="../index.html">specr</a>
        <span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="">1.0.0</span>
      </span>
    </div>

    <div id="navbar" class="navbar-collapse collapse">
      <ul class="nav navbar-nav">
<li>
  <a href="../articles/specr.html">Getting started</a>
</li>
<li>
  <a href="../reference/index.html">Reference</a>
</li>
<li class="dropdown">
  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" data-bs-toggle="dropdown" aria-expanded="false">
    Articles
     
    <span class="caret"></span>
  </a>
  <ul class="dropdown-menu" role="menu">
<li class="dropdown-header">Basics</li>
    <li>
      <a href="../articles/different-specifications.html">Setting up different types of specifications</a>
    </li>
    <li>
      <a href="../articles/custom-plot.html">Visualizing specification curve analyses</a>
    </li>
    <li>
      <a href="../articles/invest-spec.html">Investigating specific specifications</a>
    </li>
    <li>
      <a href="../articles/parallelization.html">Parallelization</a>
    </li>
    <li class="divider">
    </li>
<li class="dropdown-header">Incorporating specific models</li>
    <li>
      <a href="../articles/measurement_models.html">Structural equation models</a>
    </li>
    <li>
      <a href="../articles/random_effects.html">Multilevel models</a>
    </li>
    <li>
      <a href="../articles/bayesian-models.html">Bayesian Modeling</a>
    </li>
  </ul>
</li>
<li>
  <a href="../news/index.html">News</a>
</li>
      </ul>
<ul class="nav navbar-nav navbar-right">
<li>
  <a href="https://github.com/masurp/specr" class="external-link">
    <span class="fa fa-github fa-lg"></span>
     
  </a>
</li>
      </ul>
</div>
<!--/.nav-collapse -->
  </div>
<!--/.container -->
</div>
<!--/.navbar -->

      

      </header><div class="row">
  <div class="col-md-9 contents">
    <div class="page-header toc-ignore">
      <h1 data-toc-skip>An introduction to specr</h1>
            
      
      <small class="dont-index">Source: <a href="https://github.com/masurp/specr/blob/HEAD/vignettes/specr.rmd" class="external-link"><code>vignettes/specr.rmd</code></a></small>
      <div class="hidden name"><code>specr.rmd</code></div>

    </div>

    
    
<p>The following vignette is a more comprehensive version of the short
example on the homepage and exemplifies how to use the core functions of
this package. Although using <code>specr</code> strictly speaking does
not require any other package, we recommend to also load the
<code>tidyverse</code> as it provides valuable functions for data
wrangling and adapting outputs from <code>specr</code> functions.
Furthermore, the <code>tidyverse</code> includes <code>ggplot2</code>
which can be used to modify the layout of the plots.</p>
<div class="sourceCode" id="cb1"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># Load library</span></span>
<span><span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://masurp.github.io/specr/" class="external-link">specr</a></span><span class="op">)</span></span>
<span><span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://ggplot2.tidyverse.org" class="external-link">ggplot2</a></span><span class="op">)</span></span></code></pre></div>
<div class="section level2">
<h2 id="understanding-the-data">Understanding the data<a class="anchor" aria-label="anchor" href="#understanding-the-data"></a>
</h2>
<p>In order to understand what type of analytical choices exists and how
they could affect a “multiverse” of results, you need to understand your
data set. In a first step, you should hence investigate your data
closely. For this example, we are going to use a simulated data set
(included in specr).</p>
<div class="sourceCode" id="cb2"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># We have a look at the simulated data set that is included in the package</span></span>
<span><span class="fu"><a href="https://rdrr.io/r/utils/head.html" class="external-link">head</a></span><span class="op">(</span><span class="va">example_data</span><span class="op">)</span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># A tibble: 6 × 15</span></span></span>
<span><span class="co">#&gt;   group1 group2 group3     x1      x2      x3     x4     c1      c2     c3</span></span>
<span><span class="co">#&gt;   <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>   <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>   <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>   <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>   <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">1</span> middle female C       1.17  -<span style="color: #BB0000;">0.363</span>  -<span style="color: #BB0000;">0.999</span>  -<span style="color: #BB0000;">0.293</span>  0.736  0.171   0.244</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">2</span> middle male   B       0.765  0.034<span style="text-decoration: underline;">9</span>  0.129  -<span style="color: #BB0000;">0.155</span> -<span style="color: #BB0000;">0.720</span>  0.016<span style="text-decoration: underline;">5</span> -<span style="color: #BB0000;">0.771</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">3</span> old    female A      -<span style="color: #BB0000;">2.33</span>  -<span style="color: #BB0000;">0.998</span>  -<span style="color: #BB0000;">1.11</span>   -<span style="color: #BB0000;">1.27</span>   0.104 -<span style="color: #BB0000;">2.90</span>   -<span style="color: #BB0000;">6.95</span> </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">4</span> old    female B      -<span style="color: #BB0000;">0.507</span> -<span style="color: #BB0000;">0.970</span>  -<span style="color: #BB0000;">0.086</span><span style="color: #BB0000; text-decoration: underline;">7</span> -<span style="color: #BB0000;">0.978</span>  1.79  -<span style="color: #BB0000;">0.138</span>  -<span style="color: #BB0000;">0.431</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">5</span> old    male   A      -<span style="color: #BB0000;">0.999</span>  0.546   0.901   2.57  -<span style="color: #BB0000;">0.245</span>  2.02    2.83 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">6</span> middle male   A      -<span style="color: #BB0000;">0.177</span> -<span style="color: #BB0000;">1.37</span>    0.094<span style="text-decoration: underline;">7</span> -<span style="color: #BB0000;">2.53</span>   0.315 -<span style="color: #BB0000;">0.688</span>  -<span style="color: #BB0000;">1.33</span> </span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># … with 5 more variables: c4 &lt;dbl&gt;, y1 &lt;dbl&gt;, y2 &lt;dbl&gt;, y3 &lt;dbl&gt;, y4 &lt;dbl&gt;</span></span></span></code></pre></div>
<p>In this example, we assume that <code>x</code> represents independent
variables, <code>y</code> represents dependent variables, <code>c</code>
represents control variables, and <code>group</code> denotes potential
grouping variables that can be used for subsetting the data. Note that
these variables are <code>characters</code>. To use them as grouping
variables later, they need to be such character-variables. Numeric
variables will not work and need to be transformed beforehand.</p>
</div>
<div class="section level2">
<h2 id="defining-analytical-choices">Defining analytical choices<a class="anchor" aria-label="anchor" href="#defining-analytical-choices"></a>
</h2>
<p>The next step involves setting up all specifications. This step
involves an in-depth understanding of the research question and the
model(s) that will be specified. In this case, we assume simply that
<code>x</code> should be positively correlated with <code>y</code>. We
use the function <code><a href="../reference/setup.html">setup()</a></code> to specify different analytical
decisions.</p>
<p>One type of analytical choice that could additionally affect the
results refers to the type of model that is estimated. The function runs
traditional linear regression models by default (i.e. when
<code>model = "lm"</code> is provided as argument). However, customized
model functions can be passed to the function, too. The only requirement
is that the customized function has <code>formula</code> and
<code>data</code> as argument names. The self-made function needs to be
a model estimation function, that takes the formula and estimates the
model based on the data.</p>
<p>Once we have specified all analytic decisions, we run the function
and save the results in the object <code>specs</code>. We can then use
the summary function to get a quick overview of the multiverse.</p>
<div class="sourceCode" id="cb3"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># specific model fitting function</span></span>
<span><span class="va">lm_gaussian</span> <span class="op">&lt;-</span> <span class="kw">function</span><span class="op">(</span><span class="va">formula</span>, <span class="va">data</span><span class="op">)</span> <span class="op">{</span></span>
<span>  <span class="fu"><a href="https://rdrr.io/r/stats/glm.html" class="external-link">glm</a></span><span class="op">(</span>formula <span class="op">=</span> <span class="va">formula</span>, </span>
<span>      data <span class="op">=</span> <span class="va">data</span>, </span>
<span>      family <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/stats/family.html" class="external-link">gaussian</a></span><span class="op">(</span>link <span class="op">=</span> <span class="st">"identity"</span><span class="op">)</span><span class="op">)</span></span>
<span><span class="op">}</span></span>
<span></span>
<span><span class="co"># Specifying the analytic decisions</span></span>
<span><span class="va">specs</span> <span class="op">&lt;-</span> <span class="fu"><a href="../reference/setup.html">setup</a></span><span class="op">(</span>data <span class="op">=</span> <span class="va">example_data</span>,            <span class="co"># First, we provide the data set</span></span>
<span>               y <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"y1"</span>, <span class="st">"y2"</span><span class="op">)</span>,              <span class="co"># We choose only one dependent variable</span></span>
<span>               x <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"x1"</span>, <span class="st">"x2"</span><span class="op">)</span>,              <span class="co"># Two independent variables</span></span>
<span>               model <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"lm"</span>, <span class="st">"lm_gaussian"</span><span class="op">)</span>, <span class="co"># We estimate two types of models</span></span>
<span>               <span class="fu"><a href="https://dplyr.tidyverse.org/reference/distinct.html" class="external-link">distinct</a></span><span class="op">(</span><span class="va">example_data</span>, <span class="va">group1</span><span class="op">)</span>, <span class="co"># Subset 1</span></span>
<span>               <span class="fu"><a href="https://dplyr.tidyverse.org/reference/distinct.html" class="external-link">distinct</a></span><span class="op">(</span><span class="va">example_data</span>, <span class="va">group2</span><span class="op">)</span>, <span class="co"># Subset 2</span></span>
<span>               controls <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"c1"</span>, <span class="st">"c2"</span><span class="op">)</span><span class="op">)</span>       <span class="co"># We include two control variable </span></span>
<span></span>
<span><span class="co"># Overview of the specifications/multiverse</span></span>
<span><span class="fu"><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary</a></span><span class="op">(</span><span class="va">specs</span>, rows <span class="op">=</span> <span class="fl">20</span><span class="op">)</span></span>
<span><span class="co">#&gt; Setup for the Specification Curve Analysis</span></span>
<span><span class="co">#&gt; -------------------------------------------</span></span>
<span><span class="co">#&gt; Class:                      specr.setup -- version: 1.0.0 </span></span>
<span><span class="co">#&gt; Number of specifications:   384 </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Specifications:</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;   Independent variable:     x1, x2 </span></span>
<span><span class="co">#&gt;   Dependent variable:       y1, y2 </span></span>
<span><span class="co">#&gt;   Models:                   lm, lm_gaussian </span></span>
<span><span class="co">#&gt;   Covariates:               no covariates, c1, c2, c1 + c2 </span></span>
<span><span class="co">#&gt;   Subsets analyses:         middle &amp; female, old &amp; female, young &amp; female, female, middle &amp; male, old &amp; male, young &amp; male, male, middle, old, young, all </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Function used to extract parameters:</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;   function(x) broom::tidy(x, conf.int = TRUE)</span></span>
<span><span class="co">#&gt; &lt;environment: 0x7f980b12b768&gt;</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Head of specifications table (first 20 rows):</span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># A tibble: 20 × 8</span></span></span>
<span><span class="co">#&gt;    x     y     model controls      subsets         group1 group2 formula     </span></span>
<span><span class="co">#&gt;    <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>         <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>           <span style="color: #949494; font-style: italic;">&lt;fct&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;fct&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;glue&gt;</span>      </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 1</span> x1    y1    lm    no covariates middle &amp; female middle female y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 2</span> x1    y1    lm    no covariates old &amp; female    old    female y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 3</span> x1    y1    lm    no covariates young &amp; female  young  female y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 4</span> x1    y1    lm    no covariates female          <span style="color: #BB0000;">NA</span>     female y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 5</span> x1    y1    lm    no covariates middle &amp; male   middle male   y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 6</span> x1    y1    lm    no covariates old &amp; male      old    male   y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 7</span> x1    y1    lm    no covariates young &amp; male    young  male   y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 8</span> x1    y1    lm    no covariates male            <span style="color: #BB0000;">NA</span>     male   y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 9</span> x1    y1    lm    no covariates middle          middle <span style="color: #BB0000;">NA</span>     y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">10</span> x1    y1    lm    no covariates old             old    <span style="color: #BB0000;">NA</span>     y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">11</span> x1    y1    lm    no covariates young           young  <span style="color: #BB0000;">NA</span>     y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">12</span> x1    y1    lm    no covariates all             <span style="color: #BB0000;">NA</span>     <span style="color: #BB0000;">NA</span>     y1 ~ x1 + 1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">13</span> x1    y1    lm    c1            middle &amp; female middle female y1 ~ x1 + c1</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">14</span> x1    y1    lm    c1            old &amp; female    old    female y1 ~ x1 + c1</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">15</span> x1    y1    lm    c1            young &amp; female  young  female y1 ~ x1 + c1</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">16</span> x1    y1    lm    c1            female          <span style="color: #BB0000;">NA</span>     female y1 ~ x1 + c1</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">17</span> x1    y1    lm    c1            middle &amp; male   middle male   y1 ~ x1 + c1</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">18</span> x1    y1    lm    c1            old &amp; male      old    male   y1 ~ x1 + c1</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">19</span> x1    y1    lm    c1            young &amp; male    young  male   y1 ~ x1 + c1</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">20</span> x1    y1    lm    c1            male            <span style="color: #BB0000;">NA</span>     male   y1 ~ x1 + c1</span></span></code></pre></div>
<p>The resulting S3 class (“specr.setup”) includes a data frame with 96
different specifications. We can also see that the analysis will use the
function <code>tidy()</code> from the <code>broom</code>-package to
extract parameters from the models.</p>
<p>We can also get a “visual summary” of the specifications and the
resulting “garden of forking paths” using the generic
<code><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot()</a></code> function.</p>
<div class="sourceCode" id="cb4"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot</a></span><span class="op">(</span><span class="va">specs</span>, circular <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></code></pre></div>
<p><img src="specr_files/figure-html/unnamed-chunk-5-1.png" width="100%"></p>
</div>
<div class="section level2">
<h2 id="fit-models-across-specifications">Fit models across specifications<a class="anchor" aria-label="anchor" href="#fit-models-across-specifications"></a>
</h2>
<p>The main function of the package is <code><a href="../reference/specr.html">specr()</a></code>. We simply
pass the <code>specs</code>-object created above and specify the number
of cores that should be use in evaluating all models. A good first check
is to use the <code><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary()</a></code> function to get an overview of
results.</p>
<div class="sourceCode" id="cb5"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># Run specification curve analysis</span></span>
<span><span class="va">results</span> <span class="op">&lt;-</span> <span class="fu"><a href="../reference/specr.html">specr</a></span><span class="op">(</span><span class="va">specs</span><span class="op">)</span>  </span>
<span></span>
<span><span class="co"># Summarize results</span></span>
<span><span class="fu"><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary</a></span><span class="op">(</span><span class="va">results</span><span class="op">)</span></span>
<span><span class="co">#&gt; Results of the specification curve analysis</span></span>
<span><span class="co">#&gt; -------------------</span></span>
<span><span class="co">#&gt; Technical details:</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;   Class:                          specr.object -- version: 1.0.0 </span></span>
<span><span class="co">#&gt;   Cores used:                     1 </span></span>
<span><span class="co">#&gt;   Duration of fitting process:    5.103 sec elapsed </span></span>
<span><span class="co">#&gt;   Number of specifications:       384 </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Descriptive summary of the specification curve:</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;  median  mad   min  max   q25  q75</span></span>
<span><span class="co">#&gt;    0.13 0.45 -0.49 0.68 -0.16 0.44</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Descriptive summary of sample sizes: </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;  median min  max</span></span>
<span><span class="co">#&gt;     250 160 1000</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Head of the specification results (first 6 rows): </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># A tibble: 6 × 28</span></span></span>
<span><span class="co">#&gt;   x     y     model controls    subsets group1 group2 formula estimate std.error</span></span>
<span><span class="co">#&gt;   <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>       <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>   <span style="color: #949494; font-style: italic;">&lt;fct&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;fct&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;glue&gt;</span>     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">1</span> x1    y1    lm    no covaria… middle… middle female y1 ~ x…     0.6       0.1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">2</span> x1    y1    lm    no covaria… old &amp; … old    female y1 ~ x…     0.65      0.08</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">3</span> x1    y1    lm    no covaria… young … young  female y1 ~ x…     0.48      0.08</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">4</span> x1    y1    lm    no covaria… female  <span style="color: #BB0000;">NA</span>     female y1 ~ x…     0.61      0.05</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">5</span> x1    y1    lm    no covaria… middle… middle male   y1 ~ x…     0.61      0.09</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">6</span> x1    y1    lm    no covaria… old &amp; … old    male   y1 ~ x…     0.68      0.1 </span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># … with 18 more variables: statistic &lt;dbl&gt;, p.value &lt;dbl&gt;, conf.low &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   conf.high &lt;dbl&gt;, fit_r.squared &lt;dbl&gt;, fit_adj.r.squared &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   fit_sigma &lt;dbl&gt;, fit_statistic &lt;dbl&gt;, fit_p.value &lt;dbl&gt;, fit_df &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   fit_logLik &lt;dbl&gt;, fit_AIC &lt;dbl&gt;, fit_BIC &lt;dbl&gt;, fit_deviance &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   fit_df.residual &lt;dbl&gt;, fit_nobs &lt;dbl&gt;, fit_null.deviance &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   fit_df.null &lt;dbl&gt;</span></span></span></code></pre></div>
<p>The resulting S3 class (`specr.object”) includes relevant aspects of
the specification curve analysis and a tibble that includes results for
all specifications. We get a short descriptive summary of the
specification curve, the sample sizes and an overview of the first 6
rows of the actual results.</p>
</div>
<div class="section level2">
<h2 id="investigating-the-specification-curve">Investigating the specification curve<a class="anchor" aria-label="anchor" href="#investigating-the-specification-curve"></a>
</h2>
<p>To investigate the results, we can use some generic functions. The
simple <code><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary()</a></code> function has some additional functionality
that we can exploit here.</p>
<div class="sourceCode" id="cb6"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># basic descriptive summary of the entire specification curve</span></span>
<span><span class="fu"><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary</a></span><span class="op">(</span><span class="va">results</span>, </span>
<span>        type <span class="op">=</span> <span class="st">"curve"</span><span class="op">)</span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># A tibble: 1 × 7</span></span></span>
<span><span class="co">#&gt;   median   mad    min   max    q25   q75   obs</span></span>
<span><span class="co">#&gt;    <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">1</span>  0.129 0.452 -<span style="color: #BB0000;">0.490</span> 0.678 -<span style="color: #BB0000;">0.161</span> 0.440   250</span></span>
<span></span>
<span><span class="co"># basic descriptive summary per specific choices</span></span>
<span><span class="fu"><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary</a></span><span class="op">(</span><span class="va">results</span>, </span>
<span>        type <span class="op">=</span> <span class="st">"curve"</span>, </span>
<span>        group <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"x"</span>, <span class="st">"controls"</span><span class="op">)</span><span class="op">)</span>  <span class="co"># group analysis by choices</span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># A tibble: 8 × 9</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># Groups:   x [2]</span></span></span>
<span><span class="co">#&gt;   x     controls      median    mad      min   max     q25   q75   obs</span></span>
<span><span class="co">#&gt;   <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>          <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>    <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>   <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">1</span> x1    c1             0.117 0.709  -<span style="color: #BB0000;">0.490</span>   0.653 -<span style="color: #BB0000;">0.314</span>  0.600   250</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">2</span> x1    c1 + c2        0.160 0.595  -<span style="color: #BB0000;">0.292</span>   0.653 -<span style="color: #BB0000;">0.213</span>  0.591   250</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">3</span> x1    c2             0.178 0.609  -<span style="color: #BB0000;">0.271</span>   0.675 -<span style="color: #BB0000;">0.197</span>  0.604   250</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">4</span> x1    no covariates  0.146 0.699  -<span style="color: #BB0000;">0.462</span>   0.678 -<span style="color: #BB0000;">0.314</span>  0.616   250</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">5</span> x2    c1             0.101 0.251  -<span style="color: #BB0000;">0.204</span>   0.420 -<span style="color: #BB0000;">0.085</span><span style="color: #BB0000; text-decoration: underline;">5</span> 0.224   250</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">6</span> x2    c1 + c2        0.122 0.097<span style="text-decoration: underline;">0</span> -<span style="color: #BB0000;">0.006</span><span style="color: #BB0000; text-decoration: underline;">48</span> 0.438  0.080<span style="text-decoration: underline;">9</span> 0.207   250</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">7</span> x2    c2             0.138 0.130  -<span style="color: #BB0000;">0.015</span><span style="color: #BB0000; text-decoration: underline;">4</span>  0.432  0.066<span style="text-decoration: underline;">3</span> 0.240   250</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">8</span> x2    no covariates  0.102 0.251  -<span style="color: #BB0000;">0.194</span>   0.415 -<span style="color: #BB0000;">0.066</span><span style="color: #BB0000; text-decoration: underline;">9</span> 0.263   250</span></span>
<span></span>
<span><span class="co"># basic descriptive summary with other statistics</span></span>
<span><span class="fu"><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary</a></span><span class="op">(</span><span class="va">results</span>, </span>
<span>        type <span class="op">=</span> <span class="st">"curve"</span>, </span>
<span>        group <span class="op">=</span> <span class="st">"subsets"</span>, </span>
<span>        stats <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>mean <span class="op">=</span> <span class="va">mean</span>, </span>
<span>                     median <span class="op">=</span> <span class="va">median</span><span class="op">)</span><span class="op">)</span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># A tibble: 12 × 4</span></span></span>
<span><span class="co">#&gt;    subsets           mean median   obs</span></span>
<span><span class="co">#&gt;    <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>            <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>  <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 1</span> all             0.140  0.144   <span style="text-decoration: underline;">1</span>000</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 2</span> female          0.157  0.174    500</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 3</span> male            0.125  0.116    500</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 4</span> middle          0.125  0.130    331</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 5</span> middle &amp; female 0.149  0.130    168</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 6</span> middle &amp; male   0.091<span style="text-decoration: underline;">0</span> 0.108    163</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 7</span> old             0.184  0.204    323</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 8</span> old &amp; female    0.225  0.255    163</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;"> 9</span> old &amp; male      0.147  0.153    160</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">10</span> young           0.112  0.101    346</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">11</span> young &amp; female  0.086<span style="text-decoration: underline;">9</span> 0.110    169</span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">12</span> young &amp; male    0.135  0.084<span style="text-decoration: underline;">2</span>   177</span></span></code></pre></div>
<p>The output contains summary statistics such as the median, the median
absolute deviation, … as well as the number of observations that were
used for each model. Bear in mind that due to subsetting or missing
data, sample sizes can vary considerably which, in turn, affects the
results.</p>
</div>
<div class="section level2">
<h2 id="visualizing-results">Visualizing results<a class="anchor" aria-label="anchor" href="#visualizing-results"></a>
</h2>
<p>However, in order to grasp how the different analytical choices
affect the outcome of interest (in this case, the estimate refers to the
unstandardized regression coefficient <em>b</em>), it is reasonable to
plot a specification curve. The generic function <code><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot()</a></code> can
be used to produce the typical visualization of the specification curve
and how the analytical choices affected the obtained results.</p>
<div class="sourceCode" id="cb7"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># Plot entire visualization</span></span>
<span><span class="fu"><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot</a></span><span class="op">(</span><span class="va">results</span><span class="op">)</span></span></code></pre></div>
<p><img src="specr_files/figure-html/unnamed-chunk-8-1.png" width="100%"></p>
<p>The resulting plot includes the ranked specification curve (A) and an
overview about how the different analytical choices affect the estimate
of interest (B). Red represents negative and significant effects (based
on the chosen significance level, by default <span class="math inline">\(\alpha = .05\)</span>). Blue represents positive
and significant effects. Grey refers to non-significant effects.</p>
<p>We can also just plot the curve and adjust the style and layout using
<code>ggplot2</code>- functions. Here, I am also order the curve by
control variables and rank them in a descending order.</p>
<div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># Plot </span></span>
<span><span class="fu"><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot</a></span><span class="op">(</span><span class="va">results</span>, type <span class="op">=</span> <span class="st">"curve"</span>, group <span class="op">=</span> <span class="va">controls</span>, desc <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span> <span class="op">+</span></span>
<span>  <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/scale_brewer.html" class="external-link">scale_color_brewer</a></span><span class="op">(</span>palette <span class="op">=</span> <span class="st">"Dark2"</span><span class="op">)</span> <span class="op">+</span></span>
<span>  <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggtheme.html" class="external-link">theme_grey</a></span><span class="op">(</span><span class="op">)</span> <span class="op">+</span></span>
<span>  <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/theme.html" class="external-link">theme</a></span><span class="op">(</span>legend.position <span class="op">=</span> <span class="st">"none"</span><span class="op">)</span></span></code></pre></div>
<p><img src="specr_files/figure-html/unnamed-chunk-9-1.png" width="100%"></p>
<p>Sometimes, it can be useful to check the influence of specific
choices on the estimate of interest more precisely. We can again use the
generic function <code><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot()</a></code> to produce respective boxplots.</p>
<div class="sourceCode" id="cb9"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot</a></span><span class="op">(</span><span class="va">results</span>, type <span class="op">=</span> <span class="st">"boxplot"</span><span class="op">)</span></span></code></pre></div>
<p><img src="specr_files/figure-html/unnamed-chunk-10-1.png" width="100%"></p>
<p>We can see that the dependent variable (<code>y</code>) produces the
largest differences in the obtained estimates.</p>
</div>
<div class="section level2">
<h2 id="decomposing-the-variance-in-the-specification-curve">Decomposing the variance in the specification curve<a class="anchor" aria-label="anchor" href="#decomposing-the-variance-in-the-specification-curve"></a>
</h2>
<p>Finally, we can estimate how much variance in the specification curve
is related to which analytical decisions. We again can simply use the
<code><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot()</a></code> function with the argument
<code>type = "variance"</code> and receive a barplot that shows which
analytical choice accounts for which amount of variance in the
curve.</p>
<div class="sourceCode" id="cb10"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot</a></span><span class="op">(</span><span class="va">results</span>, type <span class="op">=</span> <span class="st">"variance"</span><span class="op">)</span></span>
<span><span class="co">#&gt;        grp vcov  icc percent</span></span>
<span><span class="co">#&gt; 1  subsets 0.00 0.00    0.28</span></span>
<span><span class="co">#&gt; 2 controls 0.00 0.01    0.87</span></span>
<span><span class="co">#&gt; 3    model 0.00 0.00    0.00</span></span>
<span><span class="co">#&gt; 4        y 0.16 0.83   83.01</span></span>
<span><span class="co">#&gt; 5        x 0.00 0.01    0.73</span></span>
<span><span class="co">#&gt; 6 Residual 0.03 0.15   15.11</span></span></code></pre></div>
<p><img src="specr_files/figure-html/unnamed-chunk-11-1.png" width="100%"></p>
</div>
  </div>

  <div class="col-md-3 hidden-xs hidden-sm" id="pkgdown-sidebar">

        <nav id="toc" data-toggle="toc"><h2 data-toc-skip>Contents</h2>
    </nav>
</div>

</div>



      <footer><div class="copyright">
  <p></p>
<p>Developed by Philipp K. Masur, Michael Scharkow.</p>
</div>

<div class="pkgdown">
  <p></p>
<p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link">pkgdown</a> 2.0.6.</p>
</div>

      </footer>
</div>

  


  

  </body>
</html>
