---
title: "Analyzing R-Bloggers' posts via Twitter"
output:
  html_document:
    toc: true
    css: ../css/main.css
author: Dean Attali
date: "2015-05-15"
runtime: shiny

---

## Overview

This document is a supplement to [my blog post about analyzing R-Bloggers' posts via Twitter](http://deanattali.com/2015/05/17/analyzing-rbloggers-posts-via-twitter/).  That blog post shows and discusses the results, while this document contains the source code that generated the results and figures.

I'm also providing a few interactive plots/tables to explore the data before showcasing the source code. These are all elements that are included in the blog post as static images, but I thought it could be nice to be able to explore a few of them interactively as well.

The source code that comes later is silenced and not actually ran as R code, because I didn't want to make the loading of this document take a long time. Much of the source code section will include comments in code instead of nicely formatted markdown text - again, that's because the purpose of this document is mostly to give out the code. For a nicer read, [read the blog post](http://deanattali.com/2015/05/17/analyzing-rbloggers-posts-via-twitter/) :)

This document is also available [on GitHub](https://github.com/daattali/shiny-server/tree/master/rbloggers-twitter), where you can also download the data if you want to look at it yourself without having to go through the long process of generating it.

# Interactive results

Just a few plots/tables that I figured would be nice to explore interactively. 

```{r interactive-load-pkgs, echo = FALSE, warning=FALSE, message=FALSE}
library(DT)
library(ggvis)
library(plyr)
library(dplyr)

# read tweets file
tweets_file <- file.path("results", "tweets.rds")
if (!file.exists(tweets_file)) {
  print("Could not find tweets file - not showing interactive elements.")
  show_interactive <- FALSE
} else {
  tweets <- readRDS(tweets_file)
  
  # this code is copied from the main section
  tweets_by_author <-
    tweets %>% 
    ddply(~ author, function(x) {
      data.frame(num_tweets = nrow(x),
                 avg_favorites = mean(x$favorites) %>% round(digits = 1),
                 avg_retweets = mean(x$retweets) %>% round(digits = 1),
                 avg_score = mean(x$score) %>% round(digits = 1)
      )}) %>%
      arrange(desc(avg_score))
  top_authors <- tweets_by_author %>% arrange(desc(avg_score)) %>% .$author %>% head(10)
  tweets_top_authors <- tweets %>% filter(author %in% top_authors)
  
  show_interactive <- TRUE
}
```

### Posts by top 10 best-scoring authors

Hover over a data point to see more info.

```{r interactive-plot-top, fig.width=10, echo = FALSE, warning=FALSE, message=FALSE}
if (show_interactive) {
  # build a tooltip text when the mouse hovers over a data point
  tooltip_value <- function(x) {
    if(is.null(x)) return(NULL)
    
    idx <- which(tweets_top_authors$favorites == x$favorites &
                 tweets_top_authors$retweets == x$retweets &
                 tweets_top_authors$author == x$author)
    tweet <- tweets_top_authors[idx, ]
    res <-
      paste0("<strong>", tweet$title, "</strong><br/>",
             tweet$date, "<br/>",
             paste0(names(x), ": ", x, collapse = "<br />"))
    res
  }
  
  tweets_top_authors %>%
    ggvis(x = ~ favorites, y = ~ retweets, fill = ~ author, stroke := "black") %>%
    layer_points(size := 100) %>%
    add_tooltip(tooltip_value) %>%
    add_axis("x",
             title = "# favorites",
             properties = axis_props(
               title = list(fontSize = 25),
               labels = list(fontSize = 25)
             ),
             title_offset = 50
    ) %>%
    add_axis("y",
             title = "# retweets" ,
             properties = axis_props(
               title = list(fontSize = 25),
               labels = list(fontSize = 25)    
             ),
             title_offset = 50
    ) %>%
    add_legend("fill",
               title = "Author",
               properties = legend_props(
                 title = list(fontSize = 22),
                 symbols = list(size = 150),
                 labels = list(fontSize = 20))
               )
}
```

### Summary of posts by each author

```{r interactive-table-author, fig.width=10, echo = FALSE, warning=FALSE, message=FALSE}
if (show_interactive) {
  datatable(
    tweets_by_author,
    rownames = FALSE,
    escape = TRUE,
    class = 'cell-border stripe hover',
    options = list(searching = FALSE),
    caption = htmltools::tags$caption(
      style = 'caption-side: bottom; text-align: center; color: #777777;',
      "Searching and filtering has been disabled since this is hosted on a weak machine and I don't want to crash the poor guy"
    )
  )
}
```

### Summary of all posts

```{r interactive-table-tweets, fig.width=10, echo = FALSE, warning=FALSE, message=FALSE}
if (show_interactive) {
  datatable(
    tweets %>% select(date, title, author, retweets, favorites, score, url),
    rownames = FALSE,
    escape = TRUE,
    class = 'cell-border stripe hover',
    caption = htmltools::tags$caption(
      style = 'caption-side: bottom; text-align: center; color: #777777;',
      "Searching and filtering has been disabled since this is hosted on a weak machine and I don't want to crash the poor guy"
    ),
    options = list(
      searching = FALSE,
      pageLength = 10,
      columnDefs = list(
        # replace - in dates to . so that the string won't get cut into multiple lines
        list(targets = 0,
             render = JS(
               "function(data, type, row, meta) {",
                 "return data.replace(/-/g, '.');",
               "}"
        )),
        # make the title link to the blog pots
        list(targets = 6,
             render = JS(
               "function(data, type, row, meta) {",
                 "return '<a target=\"_blank\" href=\"' + data + '\">' + data + '</a>';",
               "}"
        ))
      )
    )
  )
}
```

---

# Source code

## Data prep

As I said, the source code is not actually getting run because it would put a lot of strain on my server to run it every time the page is opened. Usually with Rmarkdown that's not a problem because I would be able to use the `cache` chunk option, but cached code chunks don't work in interactive documents or documents hosted on a shiny server.  Feel free to copy the code of go to [the GitHub repo where it's hosted](https://github.com/daattali/shiny-server/tree/master/rbloggers-twitter) if you want to run the analysis yourself.

```{r knitr-setup}
knitr::opts_chunk$set(eval = FALSE, message=FALSE, warning=FALSE)
```

Load required packages

```{r load-pkgs}
library(httr)
library(XML)
library(plyr)
library(dplyr)
library(magrittr)
library(twitteR)
library(ggplot2)

# for wordcloud
library(SnowballC)
library(wordcloud)
library(tm)
library(stringr)
```

### Get data from Twitter

First we need to get authenticated. These private keys are stored in my .Rprofile. To get this to work, get your own tokens from Twitter and set them in your `.Rprofile` using `options(twitter_consumer_key = "YOUR_CONSUMER_KEY")`.

```{r setup-twitter}
setup_twitter_oauth(getOption('twitter_consumer_key'),
                    getOption('twitter_consumer_secret'),
                    getOption('twitter_access_token'),
                    getOption('twitter_access_secret'))
```

Now we grab the last 3200 tweets (restriction set by Twitter) and prepare a nice data.frame with the info we want.

I'm keeping:  

- tweet ID  

- tweet date  

- day of the week that the tweet was made  

- \# of times tweet was favorited  

- \# of times tweet was retweeted  

- tweet text (= blog post title)  

- last URL in each tweet, becuse that's the URL that points to the r-bloggers post

```{r get-tweets}
MAX_TWEETS <- 3200
tweets_raw <- userTimeline('Rbloggers', n = MAX_TWEETS,
                           includeRts = FALSE, excludeReplies = TRUE)

tweets <- 
  ldply(tweets_raw, function(x) {
    data_frame(id = x$id,
               date = as.Date(x$created),
               day = weekdays(date),
               favorites = x$favoriteCount,
               retweets = x$retweetCount,
               title = x$text,
               url = x$urls %>% .[['url']] %>% tail(1)
    )
  })

rm(tweets_raw)  # being extremely memory conscious
```

### Scrape R-Bloggers to get more info

#### Add author name to each post.

This is achieved by following the URL of each tweet and scraping the resulting R-bloggers post to find the author tag. If the author did not provide a name and is just an email address, R-Bloggers tried to hide their email address and we have to jump through a tiny hoop to grab it.

```{r add-authors}
# Get the author of a single post given an R-Bloggers post URL
get_post_author_single <- function(url) {
  if (is.null(url)) {
    return(NA)
  }

  # get author HTML node
  author_node <- 
    GET(url) %>%
    httr::content("parsed") %>%
    getNodeSet("//a[@rel='author']")
  if (author_node %>% length != 1) {
    return(NA)
  }

  # get author name
  author <- author_node %>% .[[1]] %>% xmlValue

  # r-bloggers hides email address names so grab the email a different way
  if (nchar(author) > 100 && grepl("document\\.getElementsByTagName", author)) {
    author <- author_node %>% .[[1]] %>% xmlGetAttr("title")
  }

  author  
}

# Get a list of URL --> author for a list of R-Bloggers post URLs
get_post_author <- function(urls) {
  lapply(urls, get_post_author_single) %>% unlist
}

# Add the author to each tweet.
# This will take several minutes because we're scraping r-bloggers 3200 times (don't run this frequently - we don't want to overwork our beloved R-Bloggers server)
tweets %<>% mutate(author = get_post_author(url))  

# Remove NA author (these are mostly jobs postings, plus a few blog posts that have been deleted)
tweets %<>% na.omit
```

### Clean up data

It's time for a bit of cleanup:  

- Remove the URL and `#rstats` hashtag from every tweet's title  

- Older posts all contain the text "This article was originally posted on ... and kindly contributed by ..." - try to remove that as well

- Order the day factor levels in order from Monday - Sunday

- Truncate very long author names with an ellipsis

- Merge duplicate tweets (tweets with the same author and title that are posted within a week)

```{r cleanup-tweets}
# remove redundant URL+hashtag from title
tweets$title <- mapply(gsub, sprintf(" %s #rstats", tweets$url), "", tweets$title)

# remove redundant "This article was originally published..." from title
tweets$title <- gsub(": \n\n\\(This article.*$", "", tweets$title)

# order days of the week
tweets$day %<>% factor(levels = c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"))

# truncate long author names
ellipsis <- function(x, n) {
  idx <- nchar(x) > n
  x[idx] <- paste0(substr(x[idx], 1, n-3), "...")
  x
}
tweets$author %<>% ellipsis(25)

# Merge duplicate tweets.
# There are some tweets with the exact same title and author that are a day apart for some reason, and they have different URLs because they do appear on r-bloggers twice, so here I'm trying to merge these cases. It's not the most efficient way to do it, but it's readable so I'll stick with it.
days_btwn_posts <- function(x) {
  (x$date %>% head(1) - x$date %>% tail(1)) %>% as.integer
}
tweets %<>%
  ddply(.(author, title), function(x) {
    if (nrow(x) > 1 && days_btwn_posts(x) <= 7) {
      x$favorites <- sum(x$favorites)
      x$retweets <- sum(x$retweets)
      x <- x[1, , drop = FALSE]
    }
    x
  }) %>%
  arrange(desc(date))
```

### Add score to each tweet

The last thing we need to do is to calculate a score metric for every tweet using the little bit of information we have (# of favorites and # of retweets). This is of course very arbitrary - I chose to score a tweet's success as a linear combination of its # of favorites and retweets.  Since there are roughly twice as many favorites as retweets in total, retweets get twice the weight. Very simple formula :)

```{r add-score}
sum(tweets$favorites) / sum(tweets$retweets)   # = 2.1
tweets$score <- tweets$favorites + tweets$retweets * 2
```

#### Save results for later

Save the results so that we can load them up later without doing the slow API calls + scraping again. Save it both as a csv so that humans can easily take a peek and as rds to load it into R faster and more accurately.

```{r save-tweets}
resultsDir <- "results"
dir.create(resultsDir, showWarnings = FALSE)
write.csv(tweets, file.path(resultsDir, "tweets.csv"), quote = TRUE, row.names = FALSE)
saveRDS(tweets, file.path(resultsDir, "tweets.rds"))
```

## Data exloration/visualization

### Preliminary look at data

Let's see how many favorites+retweets posts get, and which posts had the highest score.

```{r explore}
ggplot(tweets, aes(favorites, retweets)) +
    geom_jitter(size = 3, shape = 21, fill = "#444444", alpha = 0.4) +
    theme_bw(30) + xlab("# favorites") + ylab("# retweets") +
  ggtitle("Score of each tweet of @rbloggers")
# Looks like most posts are close to the (0, 0) area, with 20 favorites and 10 retweets being the maximum boundary for most. A very small fraction of tweets make it past the 40 favorites or 20 retweets. 

tweets %>% arrange(desc(score)) %>% head(10)
# Looks like DataCamp and David Smith have some of the most successful posts. Also note how 9/10 top posts have "R" in their title... Correlation or causation or random? Maybe I should start doing that too then :)
```

### Summary of posts by each author

I wanted to see which authors contribute the most, and which authors contribute the most-shared posts. I'll also look at the posts made by the top 10 highest scoring authors, and see how each of their posts compares to all other posts.

```{r by-author}
tweets_by_author <-
  tweets %>% 
  ddply(~ author, function(x) {
    data.frame(num_tweets = nrow(x),
               avg_favorites = mean(x$favorites) %>% round(digits = 1),
               avg_retweets = mean(x$retweets) %>% round(digits = 1),
               avg_score = mean(x$score) %>% round(digits = 1)
    )}) %>%
    arrange(desc(avg_score)) 

tweets_by_author %>% head(10)
# Woo I'm in the top 10! But it looks like the top 10 is dominated by one-hit wonders, so let's try again and only consider blogs that contributed more than one post.

tweets_by_author %>% filter(num_tweets > 1) %>% head(10)
# DataCamp managed to stay up there with over 30 posts, that's impressive. Generally the more posts you have, the harder it is to maintain a high average.

nrow(tweets_by_author)  # 420 unique authors since Sept 2013, so about 1/4 of the authors on r-bloggers haven't posted since

ggplot(tweets_by_author, aes(num_tweets)) +
  geom_histogram(binwidth = 1, fill = "#888888", color = "#444444") +
  theme_classic(30) +
  scale_x_continuous(limits = c(1, 50), breaks = c(1, seq(10, 50, 10))) +
  xlab("# of posts") + ylab("# of blogs who have\nexactly x posts") +
  ggtitle("How much do blogs contribute?") +
  coord_flip()
# looks like a lot of people only posted once since Sept 2013

# who are the top contributors?
tweets_by_author %>% arrange(desc(num_tweets)) %>% head(10)

# I wonder if users who post more also tend to post higher quality content?
cor(tweets_by_author$num_tweets, tweets_by_author$avg_score)

# Let's see what all the posts looks like for the top scorers (an interactive version of this plot is available at the top of this document)
top_authors <- tweets_by_author %>% arrange(desc(avg_score)) %>% .$author %>% head(10)
tweets_top_authors <- tweets %>% filter(author %in% top_authors)
ggplot(tweets_top_authors) +
    geom_point(aes(favorites, retweets, fill = author), size = 5, shape = 21) +
    theme_bw(30) +
    scale_fill_brewer("Author", type = "qual", palette = 3) +
    ggtitle("Score of each @Rbloggers tweet of\ntop-10 best scoring authors") +
    xlab("# favorites") + ylab("# retweets")

# And let's see them again, in perspective to all tweets
ggplot(tweets) +
  geom_jitter(aes(favorites, retweets), size = 3, shape = 21, fill = "#444444", alpha=0.4) +
  theme_bw(30) +
  geom_point(data = tweets_top_authors,
             aes(favorites, retweets, fill = author), size = 6, shape = 21) +
  scale_fill_brewer("Author", type = "qual", palette = 3) +
  xlab("# favorites") + ylab("# retweets") +
  ggtitle("Score of each @Rbloggers tweet (Coloured points\nare posts from top-10 best scoring authors)")
```

### Post success by day of week

I wanted to see if there is a correlation between when a post is posted and how successful it is. I also wanted to see if there are certain days of the week that are more/less active.

```{r by-day}
tweets_by_day <-
  tweets %>%
  ddply(~ day, function(x) {
    data.frame(num_tweets = nrow(x),
               favorites_per_post = mean(x$favorites) %>% round(digits = 1),
               retweets_per_post = mean(x$retweets) %>% round(digits = 1),
               avg_score = mean(x$score) %>% round(digits = 1)
    )})
tweets_by_day
# cool! looks like weekend (sat/sunday) get the least posts, BUT they both have the highest number of favorites and retweets!

ggplot(tweets, aes(x = day, y = score)) +
    geom_point(aes(fill = day),
               position = position_jitter(width = 0.3),
               show_guide = FALSE, shape = 21, color = "#333333",
               size = 3) +
  geom_line(data = tweets_by_day,
            aes(day, avg_score, group = 1),
            color = "#333333", size = 1) +
  geom_point(data = tweets_by_day,
             aes(day, avg_score),
             size = 9,
             col = "#333333") +
  theme_classic(30) +
  scale_y_continuous(limits=c(-5,100)) +
  scale_x_discrete(labels = levels(tweets_by_day$day) %>% substr(1, 3)) +
  xlab("Day of week") + ylab("Tweet score") +
  ggtitle("Tweet score vs day of tweet\n(Large point = average)")
```

### Wordcloud

I must admit I'm not the biggest fan of wordclouds, but it feels like no amateur R analysis can be complete without one of these bad boys.

```{r word-cloud}
# wordcloud of 100 most popular words in r-bloggers post titles
topwords <- 
    tweets$title %>%
    paste(collapse = " ") %>%
    str_split("\\s") %>%
    unlist %>%
    tolower %>%
    removePunctuation %>%
    removeWords(stopwords("english")) %>%
    wordStem %>%
    .[. != ""] %>% 
    .[. != "r"] %>%
    table %>%
    sort(decreasing = TRUE) %>%
    head(100)

wordcloud(names(topwords), topwords, min.freq = 10)
```

Remember to check out the [accompanying blog post](http://deanattali.com/2015/05/17/analyzing-rbloggers-posts-via-twitter/)!
