<!DOCTYPE html>
<!-- Generated by pkgdown: do not edit by hand --><html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Inference with specification curve analysis • specr</title>
<!-- jquery --><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script><!-- Bootstrap --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/css/bootstrap.min.css" integrity="sha256-bZLfwXAP04zRMK2BjiO8iu9pf4FbLqX6zitd+tIvLhE=" crossorigin="anonymous">
<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/js/bootstrap.min.js" integrity="sha256-nuL8/2cJ5NDSSwnKD8VqreErSWHtnEP9E7AySL+1ev4=" crossorigin="anonymous"></script><!-- bootstrap-toc --><link rel="stylesheet" href="../bootstrap-toc.css">
<script src="../bootstrap-toc.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous">
<!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- pkgdown --><link href="../pkgdown.css" rel="stylesheet">
<script src="../pkgdown.js"></script><meta property="og:title" content="Inference with specification curve analysis">
<meta property="og:description" content="specr">
<!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body data-spy="scroll" data-target="#toc">
    

    <div class="container template-article">
      <header><div class="navbar navbar-default navbar-fixed-top" role="navigation">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false">
        <span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>
      <span class="navbar-brand">
        <a class="navbar-link" href="../index.html">specr</a>
        <span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="">1.0.1</span>
      </span>
    </div>

    <div id="navbar" class="navbar-collapse collapse">
      <ul class="nav navbar-nav">
<li>
  <a href="../articles/getting-started.html">Getting started</a>
</li>
<li>
  <a href="../reference/index.html">Reference</a>
</li>
<li class="dropdown">
  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" data-bs-toggle="dropdown" aria-expanded="false">
    Articles
     
    <span class="caret"></span>
  </a>
  <ul class="dropdown-menu" role="menu">
<li class="dropdown-header">Basics</li>
    <li>
      <a href="../articles/different-specifications.html">Setting up different types of specifications</a>
    </li>
    <li>
      <a href="../articles/custom-plot.html">Visualizing specification curve analyses</a>
    </li>
    <li>
      <a href="../articles/invest-spec.html">Investigating specific specifications</a>
    </li>
    <li>
      <a href="../articles/inferences.html">Inferences with specification curve analysis</a>
    </li>
    <li>
      <a href="../articles/parallelization.html">Parallelization</a>
    </li>
    <li class="divider">
    </li>
<li class="dropdown-header">Incorporating specific models</li>
    <li>
      <a href="../articles/measurement-models.html">Structural equation models</a>
    </li>
    <li>
      <a href="../articles/multilevel-models.html">Multilevel models</a>
    </li>
    <li>
      <a href="../articles/parallel-bayesian-models.html">Bayesian models</a>
    </li>
  </ul>
</li>
<li>
  <a href="../news/index.html">News</a>
</li>
      </ul>
<ul class="nav navbar-nav navbar-right">
<li>
  <a href="https://github.com/masurp/specr" class="external-link">
    <span class="fa fa-github fa-lg"></span>
     
  </a>
</li>
      </ul>
</div>
<!--/.nav-collapse -->
  </div>
<!--/.container -->
</div>
<!--/.navbar -->

      

      </header><div class="row">
  <div class="col-md-9 contents">
    <div class="page-header toc-ignore">
      <h1 data-toc-skip>Inference with specification curve analysis</h1>
            
      
      <small class="dont-index">Source: <a href="https://github.com/masurp/specr/blob/HEAD/vignettes/inferences.Rmd" class="external-link"><code>vignettes/inferences.Rmd</code></a></small>
      <div class="hidden name"><code>inferences.Rmd</code></div>

    </div>

    
    
<p>In their paper, Simonsohn et al. (2020) propose a third step within
the logic of specification curve analysis, which refers to conducting
inference tests across the entire range of the specification curve. This
step aims to address the following question: <em>Considering the full
set of reasonable specifications jointly, how inconsistent are the
results with the null hypothesis of no effect?</em></p>
<p>We have been contemplating the implementation of this third step in
our package. Until recently, we have concluded that there remains
significant uncertainty regarding when such joint inferences are
appropriate (see, for example, the recent discussion by Del Giudice
&amp; Gangestad, 2021), who highlight:</p>
<blockquote>
<p>“In principle, multiverse-style analyses can be highly instructive.
At the same time, analyses that explore multiverse spaces that are not
homogeneous can produce misleading results and interpretations, lead
scholars to dismiss the robustness of theoretically important findings
that do exist, and discourage them from following fruitful avenues of
research. This can hinder scientific progress just as much as the
proliferation of false, unreplicable findings does” (p. 2).</p>
</blockquote>
<p>The issue is as follows: Multiverse or specification curve analyses
can serve different purposes. In the simplest form, it may serve to
explore and discover differences in effect sizes that relate to specific
theoretical or analytical decisions. Yet, the method is also suggestive
of being a “somehow better approach to testing a relationship”. The aim
of this second purpose, which is akin to a robustness test, is to
exhaust the multiverse of truly arbitrary decisions in order to gain
some sort of aggregate finding that can be use to refute or support a
hypothesis. As Del Giudice and Gangestad point out very convincingly,
however, it is often not straight-forward to decide whether
specification is truly arbitrary (or in their terms: exhibits principled
equivalence) compared to another specification. Yet, if this is not the
case, running any inference tests is futile as “Just a few decisions
incorrectly treated as arbitrary can quickly explode the size of the
multiverse, drowning reasonable effect estimates in a sea of unjustified
alternatives.” (Del Giudice &amp; Gangestad, 2021, p. 2).</p>
<p>For these reasons, we have been, and continue to be, concerned that
tools for such an inference test could be misused or lead to improperly
conducted robustness tests. Nevertheless, due to frequent inquiries, we
have now integrated a function called <code><a href="../reference/boot_null.html">boot_null()</a></code> in the
latest development version of <code>specr</code>. For details on the
procedure, please refer to the paper by Simonsohn et al. (2020). Please
note that the implemented procedure pertains to the section “Inference
with non-experimental data” in the paper.</p>
<div class="section level2">
<h2 id="preparations">Preparations<a class="anchor" aria-label="anchor" href="#preparations"></a>
</h2>
<div class="section level3">
<h3 id="loading-packages-and-data">Loading packages and data<a class="anchor" aria-label="anchor" href="#loading-packages-and-data"></a>
</h3>
<div class="sourceCode" id="cb1"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># Load packages</span></span>
<span><span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://masurp.github.io/specr/" class="external-link">specr</a></span><span class="op">)</span></span>
<span><span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://tidyverse.tidyverse.org" class="external-link">tidyverse</a></span><span class="op">)</span></span></code></pre></div>
<p>For this tutorial, we will simple use the data set included in the
package, which we can call using <code>example_data</code>.</p>
<div class="sourceCode" id="cb2"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="fu"><a href="https://pillar.r-lib.org/reference/glimpse.html" class="external-link">glimpse</a></span><span class="op">(</span><span class="va">example_data</span><span class="op">)</span></span>
<span><span class="co">#&gt; Rows: 1,000</span></span>
<span><span class="co">#&gt; Columns: 15</span></span>
<span><span class="co">#&gt; $ group1 <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> "middle", "middle", "old", "old", "old", "middle", "middle", "y…</span></span>
<span><span class="co">#&gt; $ group2 <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> "female", "male", "female", "female", "male", "male", "male", "…</span></span>
<span><span class="co">#&gt; $ group3 <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> "C", "B", "A", "B", "A", "A", "C", "B", "C", "A", "C", "B", "A"…</span></span>
<span><span class="co">#&gt; $ x1     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> 1.16616502, 0.76481253, -2.32781072, -0.50729975, -0.99903467, …</span></span>
<span><span class="co">#&gt; $ x2     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> -0.36337844, 0.03490195, -0.99786005, -0.97004472, 0.54629183, …</span></span>
<span><span class="co">#&gt; $ x3     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> -0.99899919, 0.12896300, -1.11167238, -0.08670481, 0.90082978, …</span></span>
<span><span class="co">#&gt; $ x4     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> -0.29290911, -0.15539508, -1.27298782, -0.97830572, 2.56842916,…</span></span>
<span><span class="co">#&gt; $ c1     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> 0.73628873, -0.72031278, 0.10362937, 1.78665608, -0.24463403, 0…</span></span>
<span><span class="co">#&gt; $ c2     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> 0.17137360, 0.01651493, -2.89579197, -0.13769598, 2.02092928, -…</span></span>
<span><span class="co">#&gt; $ c3     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> 0.2438919, -0.7708723, -6.9505025, -0.4307226, 2.8327862, -1.32…</span></span>
<span><span class="co">#&gt; $ c4     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> 1.17397012, 0.00271097, 0.11513596, 0.84856292, -1.40719578, 0.…</span></span>
<span><span class="co">#&gt; $ y1     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> -0.83896230, -1.19442262, -1.63184657, 0.20794834, -1.06889092,…</span></span>
<span><span class="co">#&gt; $ y2     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> -1.15339449, -1.40708441, 0.46620798, 0.17421825, 2.89071414, 2…</span></span>
<span><span class="co">#&gt; $ y3     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> -0.20911315, 0.17605870, -1.07664735, -0.37805585, -0.35389962,…</span></span>
<span><span class="co">#&gt; $ y4     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span> -0.10135448, -0.08908509, -0.27064865, 0.73415456, -0.20267287,…</span></span></code></pre></div>
</div>
<div class="section level3">
<h3 id="creating-a-custom-function-to-extract-full-model">Creating a custom function to extract full model<a class="anchor" aria-label="anchor" href="#creating-a-custom-function-to-extract-full-model"></a>
</h3>
<p>To prepare the inference under-the-null bootstrapping procedure, we
need to run the standard specification curve analysis, but make sure
that we keep the entire model object. By default, the function
<code><a href="../reference/specr.html">specr()</a></code> only keeps the relevant coefficient, but by
creating a customized fitting function, we can add a column that
contains the full model.</p>
<div class="sourceCode" id="cb3"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="co"># Requires to keep full model</span></span>
<span><span class="va">tidy_full</span> <span class="op">&lt;-</span> <span class="kw">function</span><span class="op">(</span><span class="va">x</span><span class="op">)</span> <span class="op">{</span></span>
<span>  <span class="va">fit</span> <span class="op">&lt;-</span> <span class="fu">broom</span><span class="fu">::</span><span class="fu"><a href="https://generics.r-lib.org/reference/tidy.html" class="external-link">tidy</a></span><span class="op">(</span><span class="va">x</span>, conf.int <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span>
<span>  <span class="va">fit</span><span class="op">$</span><span class="va">res</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span>  <span class="co"># Store model object</span></span>
<span>  <span class="kw"><a href="https://rdrr.io/r/base/function.html" class="external-link">return</a></span><span class="op">(</span><span class="va">fit</span><span class="op">)</span></span>
<span><span class="op">}</span></span></code></pre></div>
</div>
<div class="section level3">
<h3 id="setup-specifications">Setup specifications<a class="anchor" aria-label="anchor" href="#setup-specifications"></a>
</h3>
<p>When setting up the specifications, we simply pass the customized
function via the argument <code>fun1</code>.</p>
<div class="sourceCode" id="cb4"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="va">specs</span> <span class="op">&lt;-</span> <span class="fu"><a href="../reference/setup.html">setup</a></span><span class="op">(</span>data <span class="op">=</span> <span class="va">example_data</span>,</span>
<span>   y <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"y1"</span>, <span class="st">"y2"</span><span class="op">)</span>,</span>
<span>   x <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"x1"</span>, <span class="st">"x2"</span><span class="op">)</span>,</span>
<span>   model <span class="op">=</span> <span class="st">"lm"</span>,</span>
<span>   controls <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"c1"</span>, <span class="st">"c2"</span><span class="op">)</span>,</span>
<span>   fun1 <span class="op">=</span> <span class="va">tidy_full</span><span class="op">)</span>          <span class="co"># Important to use the custom function here!</span></span></code></pre></div>
</div>
<div class="section level3">
<h3 id="run-standard-specification-curve-analysis">Run standard specification curve analysis<a class="anchor" aria-label="anchor" href="#run-standard-specification-curve-analysis"></a>
</h3>
<p>Next, we simply ran the standard specification curve analysis using
the core function <code>specr</code>.</p>
<div class="sourceCode" id="cb5"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="va">results</span> <span class="op">&lt;-</span> <span class="fu"><a href="../reference/specr.html">specr</a></span><span class="op">(</span><span class="va">specs</span><span class="op">)</span></span>
<span><span class="fu"><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary</a></span><span class="op">(</span><span class="va">results</span><span class="op">)</span></span>
<span><span class="co">#&gt; Results of the specification curve analysis</span></span>
<span><span class="co">#&gt; -------------------</span></span>
<span><span class="co">#&gt; Technical details:</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;   Class:                          specr.object -- version: 1.0.1 </span></span>
<span><span class="co">#&gt;   Cores used:                     1 </span></span>
<span><span class="co">#&gt;   Duration of fitting process:    0.265 sec elapsed </span></span>
<span><span class="co">#&gt;   Number of specifications:       16 </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Descriptive summary of the specification curve:</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;  median  mad   min  max   q25  q75</span></span>
<span><span class="co">#&gt;    0.14 0.44 -0.34 0.62 -0.13 0.35</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Descriptive summary of sample sizes: </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;  median  min  max</span></span>
<span><span class="co">#&gt;    1000 1000 1000</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Head of the specification results (first 6 rows): </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># A tibble: 6 × 25</span></span></span>
<span><span class="co">#&gt;   x     y     model controls      subsets formula   estimate std.error statistic</span></span>
<span><span class="co">#&gt;   <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span> <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>         <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>   <span style="color: #949494; font-style: italic;">&lt;glue&gt;</span>       <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span>     <span style="color: #949494; font-style: italic;">&lt;dbl&gt;</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">1</span> x1    y1    lm    no covariates all     y1 ~ x1 …     0.62      0.04     16.4 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">2</span> x1    y1    lm    c1            all     y1 ~ x1 …     0.6       0.04     15.6 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">3</span> x1    y1    lm    c2            all     y1 ~ x1 …     0.61      0.04     15.9 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">4</span> x1    y1    lm    c1 + c2       all     y1 ~ x1 …     0.59      0.04     15.1 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">5</span> x1    y2    lm    no covariates all     y2 ~ x1 …    -<span style="color: #BB0000;">0.33</span>      0.04     -<span style="color: #BB0000;">7.68</span></span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">6</span> x1    y2    lm    c1            all     y2 ~ x1 …    -<span style="color: #BB0000;">0.34</span>      0.04     -<span style="color: #BB0000;">7.72</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># ℹ 16 more variables: p.value &lt;dbl&gt;, conf.low &lt;dbl&gt;, conf.high &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   res &lt;list&gt;, fit_r.squared &lt;dbl&gt;, fit_adj.r.squared &lt;dbl&gt;, fit_sigma &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   fit_statistic &lt;dbl&gt;, fit_p.value &lt;dbl&gt;, fit_df &lt;dbl&gt;, fit_logLik &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   fit_AIC &lt;dbl&gt;, fit_BIC &lt;dbl&gt;, fit_deviance &lt;dbl&gt;, fit_df.residual &lt;dbl&gt;,</span></span></span>
<span><span class="co">#&gt; <span style="color: #949494;">#   fit_nobs &lt;dbl&gt;</span></span></span></code></pre></div>
<p>So far, nothing new. We get the results of the specification curve
analysis and can explore them descriptively or visually.</p>
</div>
</div>
<div class="section level2">
<h2 id="refit-the-models-under-the-null">Refit the models under-the-null<a class="anchor" aria-label="anchor" href="#refit-the-models-under-the-null"></a>
</h2>
<p>The idea behind this inference approach is that one forces the null
on the data. Have a look at the following section of the Simonsohn et
al. (2020) paper (p. 1213):</p>
<p><img src="https://raw.githubusercontent.com/masurp/specr/master/docs/articles/simonsohn_inference.png" width="45%"></p>
<div class="section level3">
<h3 id="run-bootstrap-sampling-procedure">Run bootstrap sampling procedure<a class="anchor" aria-label="anchor" href="#run-bootstrap-sampling-procedure"></a>
</h3>
<p>To run this procedure, we simply use the function
<code><a href="../reference/boot_null.html">boot_null()</a></code>, which requires the results, the specification
setup, and the number of samples that should be drawn (Simonsohn et
al. suggest n_samples = 500, here I am only demonstrating it with 10
resamples). Technically, we would suggest to run at least 1,000 or
better 10,000 samples to obtain robust results (as is common in
bootstrapping procedures or Monte Carlo simulations).</p>
<div class="sourceCode" id="cb6"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/Random.html" class="external-link">set.seed</a></span><span class="op">(</span><span class="fl">42</span><span class="op">)</span></span>
<span><span class="va">boot_models</span> <span class="op">&lt;-</span> <span class="fu"><a href="../reference/boot_null.html">boot_null</a></span><span class="op">(</span><span class="va">results</span>, <span class="va">specs</span>, n_samples <span class="op">=</span> <span class="fl">10</span><span class="op">)</span> <span class="co"># better 1,000 - 10,000!</span></span>
<span><span class="va">boot_models</span></span>
<span><span class="co">#&gt; Results of bootstrapping 'under-the-null' procedure</span></span>
<span><span class="co">#&gt; -------------------</span></span>
<span><span class="co">#&gt; Technical details:</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;   Class:                           specr.boot -- version: 1.0.1 </span></span>
<span><span class="co">#&gt;   Cores used:                      1 </span></span>
<span><span class="co">#&gt;   Duration of fitting process:     20.001 sec elapsed </span></span>
<span><span class="co">#&gt;   Number of bootstrapped samples:  10 </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Descriptive summary of the specification curves 'under-the-null' (head):</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;           id median   min  max</span></span>
<span><span class="co">#&gt;  Bootstrap01  -0.01 -0.14 0.06</span></span>
<span><span class="co">#&gt;  Bootstrap02   0.00 -0.08 0.02</span></span>
<span><span class="co">#&gt;  Bootstrap03  -0.02 -0.12 0.02</span></span>
<span><span class="co">#&gt;  Bootstrap04  -0.03 -0.10 0.02</span></span>
<span><span class="co">#&gt;  Bootstrap05   0.00 -0.05 0.09</span></span>
<span><span class="co">#&gt;  Bootstrap06   0.00 -0.08 0.06</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt; Overall median across all resamples (should be close to NULL):</span></span>
<span><span class="co">#&gt; </span></span>
<span><span class="co">#&gt;  median   min   max</span></span>
<span><span class="co">#&gt;   -0.01 -0.01 -0.01</span></span></code></pre></div>
<p>The resulting fit object includes all resamples under the null (the
output shows the first 6 curves summarized).</p>
</div>
<div class="section level3">
<h3 id="summarize-findings">Summarize findings<a class="anchor" aria-label="anchor" href="#summarize-findings"></a>
</h3>
<p>Based on these resamples, we can compute several test statistics.
Simonsohn et al. propose three, but we implemented the first two for
now:</p>
<ol style="list-style-type: decimal">
<li><p>Obtaining the median effect estimated across all specifications,
and then testing whether this median estimated effect is more extreme
than would be expected if all specifications had a true effect of
zero.</p></li>
<li><p>The share of specifications that obtain a statistically
significant effect in the predicted direction, testing whether such
share is more extreme (higher) than would be expected if all
specifications had an effect of zero.</p></li>
</ol>
<p>These test statistics can be obtained by simply using the generic
function <code><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary()</a></code> around the fitted bootstrap object:</p>
<div class="sourceCode" id="cb7"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/summary.html" class="external-link">summary</a></span><span class="op">(</span><span class="va">boot_models</span><span class="op">)</span></span>
<span><span class="co">#&gt; <span style="color: #949494;"># A tibble: 3 × 3</span></span></span>
<span><span class="co">#&gt;   type           estimate p.value</span></span>
<span><span class="co">#&gt;   <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>          <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>    <span style="color: #949494; font-style: italic;">&lt;chr&gt;</span>  </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">1</span> median         0.14     &lt; .001 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">2</span> share positive 8 / 16   &lt; .001 </span></span>
<span><span class="co">#&gt; <span style="color: #BCBCBC;">3</span> share negative 6 / 16   &lt; .001</span></span></code></pre></div>
<p>As we can see here, first, the likelihood of obtaining a median of
0.14 under the assumption that all specifications had a true effect of
zero is very low (p &lt; .001). Furthermore, 8 out of 16 specifications
are actually in the right direction and the likelihood of this share of
specifications that obtains a statistically significant effect in the
predicted direction (here: positive) to be obtained if the true effect
was zero is again low (p &lt; .001).</p>
</div>
<div class="section level3">
<h3 id="plot-inference-curve">Plot inference curve<a class="anchor" aria-label="anchor" href="#plot-inference-curve"></a>
</h3>
<p>Simonsohn et al. also suggest that the observed specification curve
can be plotted against the expected under-the-null specification curves.
In the figure, we see three expected curves that are based on the 10
resamples under-the-null (if more resamples are drawn, they are based on
those resamples). The three curves here represent the 2.5th (low), the
50th (mid) and the 97.5th percentiles of these resamples.</p>
<div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/graphics/plot.default.html" class="external-link">plot</a></span><span class="op">(</span><span class="va">boot_models</span><span class="op">)</span></span></code></pre></div>
<p><img src="inferences_files/figure-html/unnamed-chunk-9-1.png" width="100%"></p>
<p>As we can see, the observed curve is clearly different than the
simulated curves unter the assumption that the true effect is zero.</p>
</div>
</div>
<div class="section level2">
<h2 id="references">References<a class="anchor" aria-label="anchor" href="#references"></a>
</h2>
<ul>
<li><p>Del Giudice, M., &amp; Gangestad, S. W. (2021). A traveler’s
guide to the multiverse: Promises, pitfalls, and a framework for the
evaluation of analytic decisions. Advances in Methods and Practices in
Psychological Science, 4(1), <a href="https://journals.sagepub.com/doi/full/10.1177/2515245920954925" class="external-link uri">https://journals.sagepub.com/doi/full/10.1177/2515245920954925</a>.</p></li>
<li><p>Simonsohn, U., Simmons, J.P. &amp; Nelson, L.D. (2020).
Specification curve analysis. Nature Human Behaviour, 4, 1208–1214. <a href="https://doi.org/10.1038/s41562-020-0912-z" class="external-link uri">https://doi.org/10.1038/s41562-020-0912-z</a></p></li>
</ul>
</div>
  </div>

  <div class="col-md-3 hidden-xs hidden-sm" id="pkgdown-sidebar">

        <nav id="toc" data-toggle="toc"><h2 data-toc-skip>Contents</h2>
    </nav>
</div>

</div>



      <footer><div class="copyright">
  <p></p>
<p>Developed by Philipp K. Masur, Michael Scharkow.</p>
</div>

<div class="pkgdown">
  <p></p>
<p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link">pkgdown</a> 2.0.7.</p>
</div>

      </footer>
</div>

  


  

  </body>
</html>
