{"word_count": 646, "character_count": 5738}
Two days ago, in my post to extract automatically my own tweets, and to generate some html list, I mentioned that it would be great if there were a function that could be used to distinguish tweets in English, and tweets in French (usually, I tweet in one of those two languages). And one more time, @3wen came to rescue me! In my previous post, I spent some time to extract mentions to other twitter accounts, and urls. This time, I have to remove those two informations, that are not relevant to get the language of a tweet.&#62; text_only &#60;- function(x){+ split_x &#60;- strsplit(x,"@")[[1]]+ x_id &#60;- paste(split_x,collapse="http://twitter.com/",sep="")+ split_x_id &#60;- strsplit(x_id,"http")+ n &#60;- length(split_x_id[[1]])+ tweet_x &#60;- strsplit(split_x_id[[1]]," ")+ + if(n==1) rt &#60;- x_id+ if(n&#62;1){+ for(i in 2:n){+ tweet_x[[i]] &#60;- tweet_x[[i]][-1]+ }}+ tweet_x_text &#60;- paste(unlist(tweet_x),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text," rt")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text," ht")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,", rt")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,", ht")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,", via")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,", poke")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,",rt")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,",ht")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,",via")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,",poke")),collapse=" ")+ tweet_x_text &#60;- paste(unlist(strsplit(tweet_x_text,"\"")),collapse=" ")+ return(tweet_x_text)+ }The function is very simple. The only point here is that I wanted to exclude typical tweet words, such as "RT", "HT", or words such as "via", or "poke", that can be used in any language. To get the text of my own tweets, we can use&#62; tweets_freak_text &#60;- lapply(tweets_freak, text_only)Now, to get the language of a tweet, the function I was looking for is the following,&#62; library("textcat")&#62; library("rvest")&#62; library("stringr")&#62; text_tweets &#60;- str_trim(unlist(tweets_freak_text))&#62; lang &#60;- textcat(text_tweets)For instance&#62; text_tweets[1][1] "De la surveillance de masse à la paranoïa généralisée  par passionant !"&#62; textcat(text_tweets[1])[1] "french"or&#62; text_tweets[6][1] "Life Expectancy improved in every country of the world  on blog"&#62; textcat(text_tweets[6])[1] "english"All that makes sense. Of course, sometimes, the prediction is not the one I expected&#62; text_tweets[2][1] "You're more likely to die on your birthday   see also"&#62; textcat(text_tweets[2])[1] "afrikaans"(note that https://translate.google.com/ detected English, here). Consider the following code now, to get a data frame, with the language, as well as a flag if the language is neither English or French,&#62; base_tweets &#60;- data.frame(text=substr(text_tweets,1,20),+ lang=textcat(text_tweets),+ check=1-textcat(text_tweets) %in% c("english","french"))&#62; head(base_tweets,20)                        text         lang check1  De la surveillance de mas       french     02  You're more likely to die    afrikaans     13  Universités: guerre ouver       french     04  [free ebook]  Statistique       french     05  (nice viz, even if I bel       english     06  Life Expectancy improved       english     07  Gigabyte gourmet: AI robo      english     08  interview with Damon Murr      english     09  Roundup and Gluten Intol       english     010 Measuring Inequality  by       english     011 Pour une révolution fisca       french     012 How statisticians changed      english     013 The Death and Life of the      english     014 “Le bénéficiaire de la lé       french     015 Traffic Accidents Are Mor      english     016 Flight Reliability When F      english     017 How good are out-of-sampl      english     018 Entrepreneurship, down-si      english     019                true story slovak-ascii     120 An NSA Big Graph experime         &#60;NA&#62;     1That's not (so) bad... if we consider that we have less than 140 characters, i.e. less than 20 words, to recognize a language. I will have to check less than 15% of the tweets. Now, I can use the following lines to generate a text file with the html code I can use in my posts, with tweets in English first, then French, and finally those that were neither in English, nor in French&#62; list_tweets_start &#60;- unlist(tweets_freak_sub)&#62; list_tweets &#60;- c(list_tweets_start[base_tweets$lang=="english"],+ list_tweets_start[base_tweets$lang=="french"],+list_tweets_start[base_tweets$check==1])&#62; write.table(list_tweets,file="tweets_somewhere_else.txt",quote=FALSE,row.names=FALSE)For instance, the first tweets in English are(nice viz', even if I believe that the title is maybe not appropriate)"Life Expectancy improved in every country of the world"  http://maxroser.com/everyone-is-better-off-life-expectancy-increased/  on @MaxCRoser's  blog  http://twitter.com/freakonometrics/status/551981550070693888/photo/1 "Gigabyte gourmet: AI robot learns to cook just by watching YouTube videos"  http://rt.com/news/219687-robot-learns-watching-video/  ht @aussietorres interview with Damon Murray,  http://www.vice.com/read/russian-criminal-tattoo-fuel-damon-murray-interview-876  who published the Russian Criminal Tattoo Encyclopaedia series, ht @fdastous That's a good start... Then I have to include pictures, graphs, and maps, but at least, that's easy to spot here (from the url).