<!DOCTYPE html>
<!-- Generated by pkgdown: do not edit by hand --><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Getting started with <code>castarter</code> • castarter</title>
<!-- jquery --><script src="https://code.jquery.com/jquery-3.1.0.min.js" integrity="sha384-nrOSfDHtoPMzJHjVTdCopGqIqeYETSXhZDFyniQ8ZHcVy08QesyHcnOUpMpqnmWq" crossorigin="anonymous"></script><!-- Bootstrap --><link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script><!-- Font Awesome icons --><link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/font-awesome.min.css" rel="stylesheet" integrity="sha384-T8Gy5hrqNKT+hzMclPo118YTQO6cYprQmhrYwIiQ/3axmI1hQomh7Ud2hPOy8SP1" crossorigin="anonymous">
<!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/1.7.1/clipboard.min.js" integrity="sha384-cV+rhyOuRHc9Ub/91rihWcGmMmCXDeksTtCihMupQHSsi8GIIRDG0ThDc3HGQFJ3" crossorigin="anonymous"></script><!-- sticky kit --><script src="https://cdnjs.cloudflare.com/ajax/libs/sticky-kit/1.1.3/sticky-kit.min.js" integrity="sha256-c4Rlo1ZozqTPE2RLuvbusY3+SU1pQaJC0TjuhygMipw=" crossorigin="anonymous"></script><!-- pkgdown --><link href="../pkgdown.css" rel="stylesheet">
<script src="../pkgdown.js"></script><meta property="og:title" content="Getting started with &lt;code&gt;castarter&lt;/code&gt;">
<meta property="og:description" content="">
<meta name="twitter:card" content="summary">
<!-- mathjax --><script src="https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script><!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
    <div class="container template-article">
      <header><div class="navbar navbar-default navbar-fixed-top" role="navigation">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar">
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>
      <span class="navbar-brand">
        <a class="navbar-link" href="../index.html">castarter</a>
        <span class="label label-default" data-toggle="tooltip" data-placement="bottom" title="Released package">0.2.6.4</span>
      </span>
    </div>

    <div id="navbar" class="navbar-collapse collapse">
      <ul class="nav navbar-nav">
<li>
  <a href="../index.html">
    <span class="fa fa-home fa-lg"></span>
     
  </a>
</li>
<li>
  <a href="../reference/index.html">Reference</a>
</li>
<li class="dropdown">
  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
    Articles
     
    <span class="caret"></span>
  </a>
  <ul class="dropdown-menu" role="menu">
<li>
      <a href="../articles/getting-started-with-castarter.html">Getting started with `castarter`</a>
    </li>
    <li>
      <a href="../articles/install.html">Install `castarter`</a>
    </li>
  </ul>
</li>
      </ul>
<ul class="nav navbar-nav navbar-right">
<li>
  <a href="https://github.com/giocomai/castarter">
    <span class="fa fa-github fa-lg"></span>
     
  </a>
</li>
      </ul>
</div>
<!--/.nav-collapse -->
  </div>
<!--/.container -->
</div>
<!--/.navbar -->

      
      </header><div class="row">
  <div class="col-md-9 contents">
    <div class="page-header toc-ignore">
      <h1>Getting started with <code>castarter</code>
</h1>
                        <h4 class="author">Giorgio Comai</h4>
            
            <h4 class="date">2018-10-07</h4>
      
      <small class="dont-index">Source: <a href="https://github.com/giocomai/castarter/blob/master/vignettes/getting-started-with-castarter.Rmd"><code>vignettes/getting-started-with-castarter.Rmd</code></a></small>
      <div class="hidden name"><code>getting-started-with-castarter.Rmd</code></div>

    </div>

    
    
<div id="setting-up-the-environment" class="section level2">
<h2 class="hasAnchor">
<a href="#setting-up-the-environment" class="anchor"></a>Setting up the environment</h2>
<p><code>castarter</code> facilitates downloading and extracting textual contents from a website, or a given section of a website. <code>castarter</code> can be used to analyse a single website, or more websites as part of a project, facilitating comparison among cases. In order to smoothen replication and prevent data losses, by default <code>castarter</code> stores a copy of downloaded web pages in html format. To do so, it includes a function that creates the relevant folder structure, and another function that facilitates archiving the html files in compressed format. In <code>castarter</code> every dataset is expected to be extracted from a website, and each website is expected to be part of a project. This is useful to give a consistent way of storing the materials gathered and to facilitate comparison among cases, but it does not imply additional limitations: it is possible to have projects composed of a single website, and it is possible to make comparisons among websites that are part of different projects.</p>
<p>As an example, this vignette will demonstrate how to extract press releases published by the Kremlin. When starting a new <code>castarter</code> project, the first step is usally that of defining the name of the project and the name of website. In this case, the project will be called “Presidents”, assuming other websites to be analysed later may be of presidents of other countries.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb1-1" data-line-number="1"><span class="kw"><a href="../reference/SetCastarter.html">SetCastarter</a></span>(<span class="dt">project =</span> <span class="st">"Presidents"</span>, <span class="dt">website =</span> <span class="st">"Kremlin_en"</span>)</a></code></pre></div>
<p>This stores the names of project and website as options that can be retrieved by all <code>castarter</code> functions automatically, without the need to inputting them each time a function requires to save or load files.</p>
<p>Then, it is time to create the folder structure where all outputs will be saved:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb2-1" data-line-number="1"><span class="kw"><a href="../reference/CreateFolders.html">CreateFolders</a></span>()</a></code></pre></div>
<p>This creates the following folders in the current working directory:</p>
<ul>
<li>Presidents
<ul>
<li>Kremlin_en
<ul>
<li>Archives</li>
<li>Dataset</li>
<li>Html</li>
<li>IndexHtml</li>
<li>Logs</li>
<li>Outputs</li>
<li>Txt</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>Notice that there is no need to provide names of project and website to the function <code><a href="../reference/CreateFolders.html">CreateFolders()</a></code>, since they are retrieved from the previously set options.</p>
</div>
<div id="download-index-pages" class="section level2">
<h2 class="hasAnchor">
<a href="#download-index-pages" class="anchor"></a>Download index pages</h2>
<p>From the Kremlin’s website, as in many other websites, it is easy to understand the structure of the archive: it is composed by <code>http://en.kremlin.ru/events/president/news/page/</code> plus a number. By clicking on the “previous” link at the bottom of the page, as it is customary, the user is given the possibility to see older press released. In <code>castarter</code>, it is possible to get the links to all the pages we would see by clicking subsequent times on the ‘previous page’ link, with the following command function:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb3-1" data-line-number="1">indexLinks &lt;-<span class="st"> </span><span class="kw"><a href="../reference/CreateLinks.html">CreateLinks</a></span>(<span class="dt">linkFirstChunk =</span> <span class="st">"http://en.kremlin.ru/events/president/news/page/"</span>,</a>
<a class="sourceLine" id="cb3-2" data-line-number="2">                          <span class="dt">startPage =</span> <span class="dv">1</span>,</a>
<a class="sourceLine" id="cb3-3" data-line-number="3">                          <span class="dt">endPage =</span> <span class="dv">1082</span>)</a>
<a class="sourceLine" id="cb3-4" data-line-number="4"><span class="kw">head</span>(indexLinks)</a></code></pre></div>
<p>N.B. Another frequently found way for websites to provide access to their archives is by creating pages with dates directly in their URL. If, for example, a website offers its contents in the form of <a href="http://www.example.com/archive/2015-10-22" class="uri">http://www.example.com/archive/2015-10-22</a>, then it would be possible to generate the links for all the dates, say, between January 2012 and December 2015, with the following parameters: <code><a href="../reference/CreateLinks.html">CreateLinks(linkFirstChunk = "http://www.example.com/", startDate = "2012-01-01", endDate = "2015-12-31", dateSeparator = "-")</a></code></p>
<p>This command creates a vector (<code>indexLinks</code>) including direct links to the index pages.</p>
<p>It is now necessary to download these index pages. The following command downloads the relevant index pages and saves them in html format in the <code>/Presidents/Kremlin_en/IndexHtml/</code> folder.</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb4-1" data-line-number="1"><span class="kw"><a href="../reference/DownloadContents.html">DownloadContents</a></span>(<span class="dt">links =</span> indexLinks, <span class="dt">type =</span> <span class="st">"index"</span>)</a></code></pre></div>
<p>If the process is interrupted for any reason, it is possible simply to run again the same function, which will check which files have already been downloaded, and will download the missing ones. Sometimes, due to a number of reasons, one or a few pages may not download correctly. It is usually appropriate to run the following function, which will re-download files that are too small to contain meaningful contents (if all pages have been downloaded correctly, the following will not do anything).</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb5-1" data-line-number="1"><span class="kw"><a href="../reference/DownloadContents.html">DownloadContents</a></span>(<span class="dt">links =</span> indexLinks, <span class="dt">type =</span> <span class="st">"index"</span>, <span class="dt">missingPages =</span> <span class="ot">FALSE</span>)</a></code></pre></div>
</div>
<div id="extracting-links" class="section level2">
<h2 class="hasAnchor">
<a href="#extracting-links" class="anchor"></a>Extracting links</h2>
<p>Now that we have the index pages, we need to extract direct links to the individual press releases. There are various ways to do this, but the easiest approach is looking at the URLs of individual articles, and see if there is a pattern (this is very often the case in modern websites). Hovering over the titles of an index page - you can open one with <code>browseURL(url = sample(x = indexLinks, size = 1))</code> - it is easy to see the links to indivudal news items.</p>
<p>In this case, we see that the URL to each news item includes in the URL the following string: “president/news/”. We can have a quick look at the links to see if the resulting list makes sense. In this case, it appears that also links to photo and videos are included, so we should remove them in order to have a clean list of links to press releases. The following command extracts links to individual articles.</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb6-1" data-line-number="1"></a>
<a class="sourceLine" id="cb6-2" data-line-number="2">links &lt;-<span class="st"> </span><span class="kw"><a href="../reference/ExtractLinks.html">ExtractLinks</a></span>(<span class="dt">domain =</span> <span class="st">"http://en.kremlin.ru/"</span>,</a>
<a class="sourceLine" id="cb6-3" data-line-number="3">                      <span class="dt">partOfLink =</span> <span class="st">"president/news/"</span>,</a>
<a class="sourceLine" id="cb6-4" data-line-number="4">                      <span class="dt">partOfLinkToExclude =</span> <span class="kw">c</span>(<span class="st">"page"</span>,</a>
<a class="sourceLine" id="cb6-5" data-line-number="5">                                              <span class="st">"photos"</span>,</a>
<a class="sourceLine" id="cb6-6" data-line-number="6">                                              <span class="st">"videos"</span>,</a>
<a class="sourceLine" id="cb6-7" data-line-number="7">                                              <span class="st">"special"</span>))</a></code></pre></div>
<p>We use this to extract relevant links, and discard all links to other pages or sections of the website scattered around the page that we are not interested in. For details on more advanced options, e.g. extract only links that are inside a given html container in the page, see <code><a href="../reference/ExtractLinks.html">?ExtractLinks</a></code>.</p>
<p>A common approach to find if all relevant links have been extracted, no more no less, is to see how many links per index page have been extracted.</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb7-1" data-line-number="1"><span class="kw">length</span>(links)<span class="op">/</span><span class="kw">length</span>(indexLinks)</a></code></pre></div>
<p>This should usually give a number that is very close to the number of links per index page.</p>
</div>
<div id="download-pages" class="section level2">
<h2 class="hasAnchor">
<a href="#download-pages" class="anchor"></a>Download pages</h2>
<p>Now it’s time to download the actual press releases.</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb8-1" data-line-number="1"><span class="kw"><a href="../reference/DownloadContents.html">DownloadContents</a></span>(<span class="dt">links =</span> links)</a></code></pre></div>
<p>This command downloads html files for each page, and stores them as a numbered html file in the <code>Html</code> subfolder. If for any reason you have to interrupt the process, you can re-run the command and by default it will download only pages that have not been previously downloaded.</p>
<p>When downloading thousands of pages, it sometimes happens that a few of them are not downloaded correctly due to server errors or connection issues. It is advised to re-run the above command at least once to ensure that all files have been properly downloaded. In other cases, however, an empty or almost-empty page has been downloaded (e.g. a web page with only an error message). Such pages can be downloaded with the following command (it is possible to adjust the minimum size with the <code>size</code> parameter).</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb9-1" data-line-number="1"><span class="kw"><a href="../reference/DownloadContents.html">DownloadContents</a></span>(<span class="dt">links =</span> links, <span class="dt">missingPages =</span> <span class="ot">FALSE</span>)</a></code></pre></div>
</div>
<div id="extract-data-and-metadata" class="section level2">
<h2 class="hasAnchor">
<a href="#extract-data-and-metadata" class="anchor"></a>Extract data and metadata</h2>
<p>[vignette do be completed]</p>
<hr>
<p>Here’s a whole script that outlines the whole procedure, that can be copy pasted and adapted to different websites:</p>
<div class="sourceCode" id="cb10"><pre class="sourceCode r"><code class="sourceCode r"><a class="sourceLine" id="cb10-1" data-line-number="1">#### 1. Preliminary steps ####</a>
<a class="sourceLine" id="cb10-2" data-line-number="2"></a>
<a class="sourceLine" id="cb10-3" data-line-number="3">## Install castarter (devtools required for installing from github)</a>
<a class="sourceLine" id="cb10-4" data-line-number="4"><span class="co"># install.packages("devtools")</span></a>
<a class="sourceLine" id="cb10-5" data-line-number="5">devtools<span class="op">::</span><span class="kw"><a href="http://www.rdocumentation.org/packages/devtools/topics/install_github">install_github</a></span>(<span class="dt">repo =</span> <span class="st">"giocomai/castarter"</span>)</a>
<a class="sourceLine" id="cb10-6" data-line-number="6"></a>
<a class="sourceLine" id="cb10-7" data-line-number="7">## Load castarter</a>
<a class="sourceLine" id="cb10-8" data-line-number="8"><span class="kw">library</span>(<span class="st">"castarter"</span>)</a>
<a class="sourceLine" id="cb10-9" data-line-number="9"></a>
<a class="sourceLine" id="cb10-10" data-line-number="10">## Set project and website name.</a>
<a class="sourceLine" id="cb10-11" data-line-number="11"><span class="co"># They will remain in memory in the current R session,</span></a>
<a class="sourceLine" id="cb10-12" data-line-number="12"><span class="co"># so need need to include them in each function call.</span></a>
<a class="sourceLine" id="cb10-13" data-line-number="13"><span class="kw"><a href="../reference/SetCastarter.html">SetCastarter</a></span>(<span class="dt">project =</span> <span class="st">"Presidents"</span>, <span class="dt">website =</span> <span class="st">"Kremlin_en"</span>)</a>
<a class="sourceLine" id="cb10-14" data-line-number="14"></a>
<a class="sourceLine" id="cb10-15" data-line-number="15">## Creates the folder structure where files will be saved within the</a>
<a class="sourceLine" id="cb10-16" data-line-number="16">## current working directory</a>
<a class="sourceLine" id="cb10-17" data-line-number="17"><span class="kw"><a href="../reference/CreateFolders.html">CreateFolders</a></span>()</a>
<a class="sourceLine" id="cb10-18" data-line-number="18"></a>
<a class="sourceLine" id="cb10-19" data-line-number="19">#### 2. Download index/archive pages ####</a>
<a class="sourceLine" id="cb10-20" data-line-number="20"></a>
<a class="sourceLine" id="cb10-21" data-line-number="21"><span class="co"># Check out the archive page, e.g. http://en.kremlin.ru/events/president/news/</span></a>
<a class="sourceLine" id="cb10-22" data-line-number="22"><span class="co"># See what is the last archive page, if they are defined by consecutive numbers</span></a>
<a class="sourceLine" id="cb10-23" data-line-number="23"></a>
<a class="sourceLine" id="cb10-24" data-line-number="24">indexLinks &lt;-<span class="st"> </span><span class="kw"><a href="../reference/CreateLinks.html">CreateLinks</a></span>(<span class="dt">linkFirstChunk =</span> <span class="st">"http://en.kremlin.ru/events/president/news/page/"</span>,</a>
<a class="sourceLine" id="cb10-25" data-line-number="25">                          <span class="dt">startPage =</span> <span class="dv">1</span>,</a>
<a class="sourceLine" id="cb10-26" data-line-number="26">                          <span class="dt">endPage =</span> <span class="dv">1082</span>)</a>
<a class="sourceLine" id="cb10-27" data-line-number="27"></a>
<a class="sourceLine" id="cb10-28" data-line-number="28">## Downloads all html files of the archive pages</a>
<a class="sourceLine" id="cb10-29" data-line-number="29"><span class="co"># This can be interrupted: by default, if the command is issued again it will download only missing pages (i.e. pages that have not been downloaded previously)</span></a>
<a class="sourceLine" id="cb10-30" data-line-number="30"><span class="kw"><a href="../reference/DownloadContents.html">DownloadContents</a></span>(<span class="dt">links =</span> indexLinks, <span class="dt">type =</span> <span class="st">"index"</span>)</a>
<a class="sourceLine" id="cb10-31" data-line-number="31"><span class="co"># Downloads again files with oddly small size (usually, errors), if any.</span></a>
<a class="sourceLine" id="cb10-32" data-line-number="32"><span class="kw"><a href="../reference/DownloadContents.html">DownloadContents</a></span>(<span class="dt">links =</span> indexLinks, <span class="dt">type =</span> <span class="st">"index"</span>, <span class="dt">missingPages =</span> <span class="ot">FALSE</span>)</a>
<a class="sourceLine" id="cb10-33" data-line-number="33"></a>
<a class="sourceLine" id="cb10-34" data-line-number="34">#### 3. Extract links to individual pages ####</a>
<a class="sourceLine" id="cb10-35" data-line-number="35"></a>
<a class="sourceLine" id="cb10-36" data-line-number="36"><span class="co"># Find criteria to filter only direct links to news items</span></a>
<a class="sourceLine" id="cb10-37" data-line-number="37"><span class="co"># Open an archive page at random.</span></a>
<a class="sourceLine" id="cb10-38" data-line-number="38"><span class="co"># browseURL(url = sample(x = indexLinks, size = 1))</span></a>
<a class="sourceLine" id="cb10-39" data-line-number="39"></a>
<a class="sourceLine" id="cb10-40" data-line-number="40">links &lt;-<span class="st"> </span><span class="kw"><a href="../reference/ExtractLinks.html">ExtractLinks</a></span>(<span class="dt">domain =</span> <span class="st">"http://en.kremlin.ru/"</span>,</a>
<a class="sourceLine" id="cb10-41" data-line-number="41">                      <span class="dt">partOfLink =</span> <span class="st">"president/news/"</span>,</a>
<a class="sourceLine" id="cb10-42" data-line-number="42">                      <span class="dt">partOfLinkToExclude =</span> <span class="kw">c</span>(<span class="st">"page"</span>,</a>
<a class="sourceLine" id="cb10-43" data-line-number="43">                                              <span class="st">"photos"</span>,</a>
<a class="sourceLine" id="cb10-44" data-line-number="44">                                              <span class="st">"videos"</span>,</a>
<a class="sourceLine" id="cb10-45" data-line-number="45">                                              <span class="st">"special"</span>))</a>
<a class="sourceLine" id="cb10-46" data-line-number="46"><span class="co"># length(links)/length(indexLinks)</span></a>
<a class="sourceLine" id="cb10-47" data-line-number="47"><span class="co"># Explore links and check if more or less alright, if number realistic</span></a>
<a class="sourceLine" id="cb10-48" data-line-number="48"><span class="co"># View(links)</span></a>
<a class="sourceLine" id="cb10-49" data-line-number="49"><span class="co"># head(links)</span></a>
<a class="sourceLine" id="cb10-50" data-line-number="50"></a>
<a class="sourceLine" id="cb10-51" data-line-number="51">#### 4. Download pages ####</a>
<a class="sourceLine" id="cb10-52" data-line-number="52"></a>
<a class="sourceLine" id="cb10-53" data-line-number="53"><span class="kw"><a href="../reference/DownloadContents.html">DownloadContents</a></span>(<span class="dt">links =</span> links, <span class="dt">wait =</span> <span class="dv">3</span>)</a>
<a class="sourceLine" id="cb10-54" data-line-number="54"><span class="kw"><a href="../reference/DownloadContents.html">DownloadContents</a></span>(<span class="dt">links =</span> links, <span class="dt">missingPages =</span> <span class="ot">FALSE</span>, <span class="dt">wait =</span> <span class="dv">3</span>)</a>
<a class="sourceLine" id="cb10-55" data-line-number="55"></a>
<a class="sourceLine" id="cb10-56" data-line-number="56">#### 5. Extract metadata and text ####</a>
<a class="sourceLine" id="cb10-57" data-line-number="57"></a>
<a class="sourceLine" id="cb10-58" data-line-number="58"><span class="co"># open an article at random to find out where metadata are located</span></a>
<a class="sourceLine" id="cb10-59" data-line-number="59"><span class="co"># browseURL(url = sample(x = links, size = 1))</span></a>
<a class="sourceLine" id="cb10-60" data-line-number="60"></a>
<a class="sourceLine" id="cb10-61" data-line-number="61">id &lt;-<span class="st"> </span><span class="kw"><a href="../reference/ExtractId.html">ExtractId</a></span>()</a>
<a class="sourceLine" id="cb10-62" data-line-number="62"></a>
<a class="sourceLine" id="cb10-63" data-line-number="63">titles &lt;-<span class="st"> </span><span class="kw"><a href="../reference/ExtractTitles.html">ExtractTitles</a></span>(<span class="dt">removeEverythingAfter =</span> <span class="st">" •"</span>,</a>
<a class="sourceLine" id="cb10-64" data-line-number="64">                        <span class="dt">id =</span> id)</a>
<a class="sourceLine" id="cb10-65" data-line-number="65"></a>
<a class="sourceLine" id="cb10-66" data-line-number="66"></a>
<a class="sourceLine" id="cb10-67" data-line-number="67">dates &lt;-<span class="st"> </span><span class="kw"><a href="../reference/ExtractDates.html">ExtractDates</a></span>(<span class="dt">container =</span> <span class="st">"time"</span>,</a>
<a class="sourceLine" id="cb10-68" data-line-number="68">                      <span class="dt">attribute =</span> <span class="st">"datetime"</span>,</a>
<a class="sourceLine" id="cb10-69" data-line-number="69">                      <span class="dt">containerInstance =</span> <span class="dv">1</span>,</a>
<a class="sourceLine" id="cb10-70" data-line-number="70">                      <span class="dt">dateFormat =</span> <span class="st">"Ymd"</span>,</a>
<a class="sourceLine" id="cb10-71" data-line-number="71">                      <span class="dt">id =</span> id)</a>
<a class="sourceLine" id="cb10-72" data-line-number="72"></a>
<a class="sourceLine" id="cb10-73" data-line-number="73"><span class="co"># check how many dates were not captured</span></a>
<a class="sourceLine" id="cb10-74" data-line-number="74"><span class="kw">sum</span>(<span class="kw">is.na</span>(dates))</a>
<a class="sourceLine" id="cb10-75" data-line-number="75">links[<span class="kw">is.na</span>(dates)]</a>
<a class="sourceLine" id="cb10-76" data-line-number="76"></a>
<a class="sourceLine" id="cb10-77" data-line-number="77">language &lt;-<span class="st"> "english"</span></a>
<a class="sourceLine" id="cb10-78" data-line-number="78"></a>
<a class="sourceLine" id="cb10-79" data-line-number="79">metadata &lt;-<span class="st"> </span><span class="kw"><a href="../reference/ExportMetadata.html">ExportMetadata</a></span>(<span class="dt">id =</span> id,</a>
<a class="sourceLine" id="cb10-80" data-line-number="80">                           <span class="dt">dates =</span> dates,</a>
<a class="sourceLine" id="cb10-81" data-line-number="81">                           <span class="dt">titles =</span> titles,</a>
<a class="sourceLine" id="cb10-82" data-line-number="82">                           <span class="dt">language =</span> language,</a>
<a class="sourceLine" id="cb10-83" data-line-number="83">                           <span class="dt">links =</span> links)</a>
<a class="sourceLine" id="cb10-84" data-line-number="84"></a>
<a class="sourceLine" id="cb10-85" data-line-number="85">## Extract text</a>
<a class="sourceLine" id="cb10-86" data-line-number="86">text &lt;-<span class="st"> </span><span class="kw"><a href="../reference/ExtractText.html">ExtractText</a></span>(<span class="dt">container =</span> <span class="st">"div"</span>,</a>
<a class="sourceLine" id="cb10-87" data-line-number="87">                    <span class="dt">containerClass =</span> <span class="st">"read__content"</span>,</a>
<a class="sourceLine" id="cb10-88" data-line-number="88">                    <span class="dt">subElement =</span> <span class="st">"p"</span>,</a>
<a class="sourceLine" id="cb10-89" data-line-number="89">                    <span class="dt">removeEverythingAfter =</span> <span class="st">"</span><span class="ch">\n</span><span class="st">Published in:"</span>,</a>
<a class="sourceLine" id="cb10-90" data-line-number="90">                    <span class="dt">id =</span> id)</a>
<a class="sourceLine" id="cb10-91" data-line-number="91"></a>
<a class="sourceLine" id="cb10-92" data-line-number="92">i &lt;-<span class="st"> </span><span class="kw">sample</span>(<span class="dt">x =</span> <span class="dv">1</span><span class="op">:</span><span class="kw">length</span>(text), <span class="dv">1</span>)</a>
<a class="sourceLine" id="cb10-93" data-line-number="93">titles[i]</a>
<a class="sourceLine" id="cb10-94" data-line-number="94">dates[i]</a>
<a class="sourceLine" id="cb10-95" data-line-number="95">text[i]</a>
<a class="sourceLine" id="cb10-96" data-line-number="96">links[id][i]</a>
<a class="sourceLine" id="cb10-97" data-line-number="97">#### 6. Save and export ####</a>
<a class="sourceLine" id="cb10-98" data-line-number="98"></a>
<a class="sourceLine" id="cb10-99" data-line-number="99"><span class="kw"><a href="../reference/SaveWebsite.html">SaveWebsite</a></span>(<span class="dt">dataset =</span> <span class="ot">TRUE</span>)</a></code></pre></div>
</div>
  </div>

  <div class="col-md-3 hidden-xs hidden-sm" id="sidebar">
        <div id="tocnav">
      <h2 class="hasAnchor">
<a href="#tocnav" class="anchor"></a>Contents</h2>
      <ul class="nav nav-pills nav-stacked">
<li><a href="#setting-up-the-environment">Setting up the environment</a></li>
      <li><a href="#download-index-pages">Download index pages</a></li>
      <li><a href="#extracting-links">Extracting links</a></li>
      <li><a href="#download-pages">Download pages</a></li>
      <li><a href="#extract-data-and-metadata">Extract data and metadata</a></li>
      </ul>
</div>
      </div>

</div>


      <footer><div class="copyright">
  <p>Developed by Giorgio Comai.</p>
</div>

<div class="pkgdown">
  <p>Site built with <a href="http://pkgdown.r-lib.org/">pkgdown</a>.</p>
</div>

      </footer>
</div>

  

  </body>
</html>
