ReplicationWiki - Information on empirical studies in the Social Sciences
Höffler, Jan H.
The ReplicationWiki provides an overview of published empirical studies in the social sciences with information on data and code availability, data sources, and software. One can search for keywords, Journal of Economic Literature codes, and geographical origin of data. It informs about 670 replications, that is studies reanalyzing previously published results, as well as corrections and retractions. The wiki helps researchers to compare their results to those of previous studies. It is a resource that helps to identify useful teaching examples for statistical methods, replication and studies of social science. It allows advanced students and practicing researchers to find guidance on how to publish their replication research in various journals. A collection of teaching resources, useful tools, and literature helps instructors to integrate replication into their teaching and students to integrate open science practices into their own research. With the ongoing expansion of the wiki, currently covering more than 4,500 empirical studies, it is becoming an ever more powerful tool for social science research and education. It is a crowd-based platform where users can add their own replication results, suggest studies that should be replicated, and identify for example further data sources used in the empirical studies, especially ones from countries underrepresented in the literature and for whom economic policies are thus difficult to investigate. The Wiki uses Semantic Media Wiki technology that is evolving, and a number of technical improvements in terms of usability and database structure are planned. A massive expansion of the content is planned based on machine learning and natural language processing techniques identifying the relevant information from the available data. For the technical improvements further expertise is welcome, and for the content expansion developers and researchers from all fields of the empirical social sciences are invited to join.
 Welcome to this presentation about the Replication Wiki, which informs on Empirical Studies in the Social Sciences. I founded the Wiki in 2013 during my PhD studies in Economics. To motivate the stock, first let me try to convince you why we need this Wiki, then I'll show you how it works and which stock we use, and finally what we plan for the future and thus where you could contribute. To give you an example of what this is all about, let's think of something that is really relevant to each of us at the moment. The Corona Crisis. It's not only a health crisis. To protect the population we have lockdowns in many countries. This has the side effect that we cannot admit in person now, which also has good sides as like that more people can participate and form anywhere. But in general it freezes the economy, it causes unemployment and a massive loss of income. There are many social science aspects to the question how to best react to this crisis. From an economics perspective some may say let there be creative destruction, survival of the fittest, those that manage challenges like a crisis best are probably the most able leaders in other circumstances as well, so they should be awarded for their success and prospering and for the others there will be some trickle down effect. Others may find such an approach would involve too much hardship to bear, there might even be unrest going much further than mass demonstrations of yellow vests or people storming parliament buildings as was seen during the Great Depression in Argentina. One proposed solution is the universal basic income. Something it would be too costly, people might just become lazy. Others say it saves an enormous amount of administrative cost because no one has to check who actually leads her. If you better when not under pressure or social stigma so they can do what they really want and might even become more productive. Like in sports and politics and economic policy everyone has an opinion. Pope Francis who saw in his home country what severe crisis can do to people suggested in a recent book that the universal basic income could be part of a solution. But how do we find a scientific answer to the question whether a universal basic income can happen in times of crisis? What economists do is build a model of the world and try to argue why different factors are related in certain ways and how different policies would have different effects. Some say that such models can always be tweaked in a way that they predict what the researcher wants to show. So an additional approach is to investigate numbers from the real world, work with data, that's what we call empirical work. There are still many different approaches, what numbers should be used and how they should be investigated and many studies trying to answer the same question come to different conclusions. That can be frustrating. What many people may not know is that traditionally social scientists just made their calculations, published the results in academic journals and then the readers had to believe them or make their own calculations. But the data and the code of the calculations were not shared. So in case different studies came to different results on the same question it was difficult to identify the exact reasons for the discrepancies. So politicians could just pick studies that came to their preferred result and use them to justify the introduction of the policies that they wanted to implement anyway. In 1986 then researchers used an archive that an editor of a journal had kept of all the data and code of empirical studies published and they tried to re-run the calculations to see if they could get the same results. That's what we call a replication of the study. It turned out that in most cases the original results could not be replicated and a big discussion started why this was the case and what could be done about it. And this interesting thing the internet became accessible to more and more people and the first journal started to just put the data online so anyone could check results. It took many years until more and more journals followed that example and there are still many journals that do not enforce rules that empirical work should be accompanied by the underlying data and code. There are also cases of confidential data, think of medical data or census data. Many people would not want their income data to be available to anyone in an online appendix or some academic journal article. There's also the case of proprietary data. If a firm collects valuable data and sells it to researchers these may get valuable scientific insights from the data but will not be allowed to share them. This is basically why I set up the replication wiki. I wanted to give an overview of empirical studies and the data and code availability and also of replication studies and their results. The case that attracted a lot of attention to replication centered around a study by two economists of Harvard University on the relationship between government debt and economic growth. It became influential in politics as it presented an easy to understand stylized fact that can be used to support the case for austerity policies. The economies of countries with government debt higher than 90% of GDP grow much less. Thomas Hinton, who you see here, was a PhD student at the time and found that in the exercise used for the analysis certain data had been left out. He argued that after correcting for this and making other adjustments that he regarded necessary the stylized fact disappeared. Thomas gave lots of media interviews around the world and was invited to present his work at many institutions. We gave a workshop together on replication in San Francisco. He became economics professor at Loyola Merriman University in Los Angeles by now. One of the authors of the original study now is chief economist of the World Bank. There is still an ongoing discussion on how the data on growth and debt should be interpreted and what we can conclude for our current situation. What we can certainly learn from this is the following. Every non-economist can find errors in exo sheets. If it can be as easy as that to initiate a worldwide discussion about pivotal questions in economics then more transparency by making such material available can definitely help to improve the quality of empirical social science research. So this is why I founded the replication wiki, whose main page you see here. It is a database of by now more than 4500 studies from the social sciences for which empirical methods were used. It lists which of the studies have data and code available online. In cases where replications are known, they are classified by their type and results. At the bottom here you see which software was used. Media wiki is the software that is also used for Wikipedia. It works well for text-based collaborative pages with images. What we wanted is a kind of database structure that allows for complex searches. Typical example is an instructor who searches for studies that have replication material available, employ a method that is truly taught, use the software that is available to the students and cover a topic that is regarded interesting. To research like that in Wikipedia, say you have seen a band with keyboard, guitar and drums and the single first name was Jim, all the information is there but you can only use full text search and categories and won't get a nice list of all measures. Semantic media wiki allows such a structure. For example, what software was used is stored as a property of each page for a study and you can easily search for all other studies that have the same property. The number of extensions help us. For example, a form extension allows users to just add information to predefined fields. No markup language is needed that deters unexperienced users. Some of the extensions could benefit from adjustment to our purposes. For example here you see this form that we use to enter new data that allows easy editing but can only produce one new page at a time. It would be good for us if there was an option to create several pages at once with such a form. And here you see why. This is a typical page for a replication. On the top you see bibliographical information, the second line informs about availability of replication material, what methods, data and software were used and the third line informs about the original study that was replicated. That study also has its page on the wiki with the same information about it and potentially more, for example if there was also another replication on it. It would be great if a form could change information on both these pages at once because currently everything has to be entered twice. That causes redundancy, potentially contradicting information if errors are made when entering the data. It's probably not difficult to find a solution but we would need someone to work on this with us. On top of what I just explained we also face other technical challenges. One is that our forms cannot be used for relations between more than two studies. I already mentioned that some studies get replicated more than once, some replications also replicate more than one original study and some replications get also replicated. So instead of one-to-one relationships we need M-to-N relationships and we do that with templates and markup code but that is user unfriendly and creates redundancies. We already found an extension for multiple instance templates but we need to implement what we want here. We would like to create automated links for selected content of properties. For our authors you can just click on them and then see all their other studies that are covered by the wiki. For data sources for example we also have information like which years they refer to in the same property field. For the years it doesn't make sense to create links but for the data sources it would be useful so it would be great to have a solution for this. Finally we would like to work on our search options. This is our advanced search form which has many options and allows for complex searches. Works great for me but it's great to complicate it with its syntax and very user friendly for others who just want to try out if this is a user service for them. This year is based on a very useful extension called drill down. You can just click on categories and select values of properties. Unfortunately it's not always possible to click on multiple values of properties. Say you search for all studies from 2014 and those of 2015. I found out you can type that into the URL then it works but again that is very user unfriendly. I wrote a bug report and the developer who created the extension even showed personal interest in our wiki and interviewed me about it but he's busy with other stuff and at the moment no one is working on fixing this. In spite of some technical imperfections the wiki already helps to build on existing research. You can compare your results to those that were already published and find out what material is available. If more than one study already addressed the same question you can see if the results were the same and if not what the reasons for this were. For example different methods or data or if there was an error in the study or if different authors just interpret results differently. The wiki helps to identify practical examples for social science education as one no longer has to do the tedious work to go through all the single journal archives to find studies that could be useful for our course. We already had teaching corporations with universities around the world. We see online that the wiki is mentioned in instructions for courses also at institutions that we did not have direct contact with and we were recommended by learned societies and libraries. We also used the wiki for some of our own research for example here I showed how the use of software is distributed in economics. In our overview of databases used you can see that the economics literature is very much dominated by the red ones from the United States so there's a lot of opportunity to check if results generalize if you check them with data from other countries. At the beginning we acquired about 200,000 dollars for this project. We have had more than 6.7 million page views and more than 280 users registered. I was invited to present the project at many places various blogs and the media reported about it. We have a link exchanged with a very widely used project research papers in economics. I could publish about it and the wiki was cited numerous times in academic works across social science disciplines. We are cooperating with a number of associations and institutions and are open to further partnerships. For the initial phase we had student assistants generating the content. That worked very well to show that this is a user service. It is however costly and not easily scalable. Now that we have explored what kind of data we can generate we would like to use machine learning and natural language processing techniques to generate content. First we need to identify the empirical studies in the literature. There are also theoretical studies, book reviews and other kinds of content in academic journals that we are not interested in so we need to sort them out. Then we would like to classify studies by whether code and data are available. We already have code written in R for this. Next we would like to identify what data was used. The witch context project has been working on this for many years and with many partners internationally not only from academia but also from the software industry like Google, Facebook, artificial intelligence. The results are still far from perfect but they started a competition. If you want a real challenge I encourage you to participate. It might be easier for us to at least classify studies by a geographical origin of data. Next we would like to classify studies as replications, corrections or retractions. All this has turned out to be very useful to the community. If there is data out there there are usually also scientists who try to analyze them and publish about this even if it does not make too much sense. There have already been studies that use data from the beginning. In the first one here computer scientists try to predict replicability of empirical studies based on the words used in them. The reasoning behind that could be that if researchers are aware that their results are not strong they might use language that is a bit vague and this could be detected. In this case I could have told them right away that this would probably not work with our data. First, because there is so much bias towards replications we exclusively getting published if they find something problematic in the original study that the sample is heavily influenced by this. Second, the types of replications are so different, some use different data, some different methods, if the results are not the same that does not necessarily mean that there was anything problematic about the original study. One would need a much cleaner kind of experiment where the conditions are the same for each replication to get a clear picture about which kind of studies are replicable and which are not. There was another article for which studies were used that had been retracted from manipulations. That is a clear criterion. Although even here, especially as long as we don't have open data and code for all empirical research just because a study was not retracted from manipulations doesn't mean they're random. It is however an interesting idea to check if our retracted studies different language was used and maybe like that one could even detect further ones that might be worth checking. With the replications I am more skeptical. What use does it really have if some algorithm says based on the language compared to previous replicated studies there are some 70% chance that a study is replicable or not? In the second case here, researchers search for replications. They are not very open about how exactly they did that, but at least they could identify some further replications. Unfortunately, they then use the data they had obtained to estimate determinants of which studies receive a published replication and drew such an ill-suited sample for this that they ended up with the opposite of the result that can be quite easily seen from the data. If you click on the link to their study it brings you to the respective page in the wiki where my comments on this are listed as a replication. There is this debate whether open data leads to researchers rewriting on the data of others that they do not fully understand and then producing questionable research with it. It is my experience, yes this happens, but in my opinion this should not be used as an excuse not to open data because that leads to the much worse state of science in which practically no published result is easily testable and thus reliable. What is needed is better post-publication review and discussion to root out errors and build consensus. We now plan to massively expand our project to include all empirical work in various social sciences fields that was published in major journals that keep online archives of data in code. For a related project, colleagues used machine learning techniques to collect information about studies in machine learning. They made quite some money with that. So we got inspired and now plan to read in data from journal websites and bibliographical databases. We already have code written in Python to create wiki pages in an automated way. What we now need is self-learning algorithms that detect relevant content from the available information and bring it into a form that is helpful to our users. What we are looking for is developers who have us to improve the wiki architecture or to generate content with machine learning techniques. We are currently raising funding and turn towards foundations, academic institutions, but also investors, as we saw with the related project, that work like ours has quite some value on the market, as one can also see from the usage numbers. You as a first step could just register to show your support. Ideally you could add content or work on the software, or just make a small contribution and vote on which studies you think should be replicated. I thank you very much for your attention. If you are interested, I have some further slides with literature that can be downloaded, but now I hope for a lively discussion.
 Welcome to this presentation about the Replication Wiki, which informs on Empirical Studies in  the Social Sciences.  I founded the Wiki in 2013 during my PhD studies in Economics.  To motivate the stock, first let me try to convince you why we need this Wiki, then I'll  show you how it works and which stock we use, and finally what we plan for the future and  thus where you could contribute.  To give you an example of what this is all about, let's think of something that is really  relevant to each of us at the moment.  The Corona Crisis.  It's not only a health crisis.  To protect the population we have lockdowns in many countries.  This has the side effect that we cannot admit in person now, which also has good sides as  like that more people can participate and form anywhere.  But in general it freezes the economy, it causes unemployment and a massive loss of  income.  There are many social science aspects to the question how to best react to this crisis.  From an economics perspective some may say let there be creative destruction, survival  of the fittest, those that manage challenges like a crisis best are probably the most able  leaders in other circumstances as well, so they should be awarded for their success and  prospering and for the others there will be some trickle down effect.  Others may find such an approach would involve too much hardship to bear, there might even  be unrest going much further than mass demonstrations of yellow vests or people storming parliament  buildings as was seen during the Great Depression in Argentina.  One proposed solution is the universal basic income.  Something it would be too costly, people might just become lazy.  Others say it saves an enormous amount of administrative cost because no one has to  check who actually leads her.  If you better when not under pressure or social stigma so they can do what they really want  and might even become more productive.  Like in sports and politics and economic policy everyone has an opinion.  Pope Francis who saw in his home country what severe crisis can do to people suggested in  a recent book that the universal basic income could be part of a solution.  But how do we find a scientific answer to the question whether a universal basic income  can happen in times of crisis?  What economists do is build a model of the world and try to argue why different factors  are related in certain ways and how different policies would have different effects.  Some say that such models can always be tweaked in a way that they predict what the researcher  wants to show.  So an additional approach is to investigate numbers from the real world, work with data,  that's what we call empirical work.  There are still many different approaches, what numbers should be used and how they should  be investigated and many studies trying to answer the same question come to different  conclusions.  That can be frustrating.  What many people may not know is that traditionally social scientists just made their calculations,  published the results in academic journals and then the readers had to believe them or  make their own calculations.  But the data and the code of the calculations were not shared.  So in case different studies came to different results on the same question it was difficult  to identify the exact reasons for the discrepancies.  So politicians could just pick studies that came to their preferred result and use them  to justify the introduction of the policies that they wanted to implement anyway.  In 1986 then researchers used an archive that an editor of a journal had kept of all the  data and code of empirical studies published and they tried to re-run the calculations  to see if they could get the same results.  That's what we call a replication of the study.  It turned out that in most cases the original results could not be replicated and a big discussion  started why this was the case and what could be done about it.  And this interesting thing the internet became accessible to more and more people and the  first journal started to just put the data online so anyone could check results.  It took many years until more and more journals followed that example and there are still many  journals that do not enforce rules that empirical work should be accompanied by the underlying  data and code.  There are also cases of confidential data, think of medical data or census data.  Many people would not want their income data to be available to anyone in an online appendix  or some academic journal article.  There's also the case of proprietary data.  If a firm collects valuable data and sells it to researchers these may get valuable scientific  insights from the data but will not be allowed to share them.  This is basically why I set up the replication wiki.  I wanted to give an overview of empirical studies and the data and code availability and also  of replication studies and their results.  The case that attracted a lot of attention to replication centered around a study by  two economists of Harvard University on the relationship between government debt and economic  growth.  It became influential in politics as it presented an easy to understand stylized fact that can  be used to support the case for austerity policies.  The economies of countries with government debt higher than 90% of GDP grow much less.  Thomas Hinton, who you see here, was a PhD student at the time and found that in the  exercise used for the analysis certain data had been left out.  He argued that after correcting for this and making other adjustments that he regarded necessary  the stylized fact disappeared.  Thomas gave lots of media interviews around the world and was invited to present his work  at many institutions.  We gave a workshop together on replication in San Francisco.  He became economics professor at Loyola Merriman University in Los Angeles by now.  One of the authors of the original study now is chief economist of the World Bank.  There is still an ongoing discussion on how the data on growth and debt should be interpreted  and what we can conclude for our current situation.  What we can certainly learn from this is the following.  Every non-economist can find errors in exo sheets.  If it can be as easy as that to initiate a worldwide discussion about pivotal questions  in economics then more transparency by making such material available can definitely help  to improve the quality of empirical social science research.  So this is why I founded the replication wiki, whose main page you see here.  It is a database of by now more than 4500 studies from the social sciences for which  empirical methods were used.  It lists which of the studies have data and code available online.  In cases where replications are known, they are classified by their type and results.  At the bottom here you see which software was used.  Media wiki is the software that is also used for Wikipedia.  It works well for text-based collaborative pages with images.  What we wanted is a kind of database structure that allows for complex searches.  Typical example is an instructor who searches for studies that have replication material  available, employ a method that is truly taught, use the software that is available to the  students and cover a topic that is regarded interesting.  To research like that in Wikipedia, say you have seen a band with keyboard, guitar and  drums and the single first name was Jim, all the information is there but you can only  use full text search and categories and won't get a nice list of all measures.  Semantic media wiki allows such a structure.  For example, what software was used is stored as a property of each page for a study and  you can easily search for all other studies that have the same property.  The number of extensions help us.  For example, a form extension allows users to just add information to predefined fields.  No markup language is needed that deters unexperienced users.  Some of the extensions could benefit from adjustment to our purposes.  For example here you see this form that we use to enter new data that allows easy editing  but can only produce one new page at a time.  It would be good for us if there was an option to create several pages at once with such  a form.  And here you see why.  This is a typical page for a replication.  On the top you see bibliographical information, the second line informs about availability  of replication material, what methods, data and software were used and the third line  informs about the original study that was replicated.  That study also has its page on the wiki with the same information about it and potentially  more, for example if there was also another replication on it.  It would be great if a form could change information on both these pages at once because  currently everything has to be entered twice.  That causes redundancy, potentially contradicting information if errors are made when entering  the data.  It's probably not difficult to find a solution but we would need someone to work on this  with us.  On top of what I just explained we also face other technical challenges.  One is that our forms cannot be used for relations between more than two studies.  I already mentioned that some studies get replicated more than once, some replications  also replicate more than one original study and some replications get also replicated.  So instead of one-to-one relationships we need M-to-N relationships and we do that with  templates and markup code but that is user unfriendly and creates redundancies.  We already found an extension for multiple instance templates but we need to implement  what we want here.  We would like to create automated links for selected content of properties.  For our authors you can just click on them and then see all their other studies that  are covered by the wiki.  For data sources for example we also have information like which years they refer to  in the same property field.  For the years it doesn't make sense to create links but for the data sources it would be  useful so it would be great to have a solution for this.  Finally we would like to work on our search options.  This is our advanced search form which has many options and allows for complex searches.  Works great for me but it's great to complicate it with its syntax and very user friendly  for others who just want to try out if this is a user service for them.  This year is based on a very useful extension called drill down.  You can just click on categories and select values of properties.  Unfortunately it's not always possible to click on multiple values of properties.  Say you search for all studies from 2014 and those of 2015.  I found out you can type that into the URL then it works but again that is very user  unfriendly.  I wrote a bug report and the developer who created the extension even showed personal  interest in our wiki and interviewed me about it but he's busy with other stuff and at  the moment no one is working on fixing this.  In spite of some technical imperfections the wiki already helps to build on existing research.  You can compare your results to those that were already published and find out what material  is available.  If more than one study already addressed the same question you can see if the results  were the same and if not what the reasons for this were.  For example different methods or data or if there was an error in the study or if different  authors just interpret results differently.  The wiki helps to identify practical examples for social science education as one no longer  has to do the tedious work to go through all the single journal archives to find studies  that could be useful for our course.  We already had teaching corporations with universities around the world.  We see online that the wiki is mentioned in instructions for courses also at institutions  that we did not have direct contact with and we were recommended by learned societies and  libraries.  We also used the wiki for some of our own research for example here I showed how the  use of software is distributed in economics.  In our overview of databases used you can see that the economics literature is very much  dominated by the red ones from the United States so there's a lot of opportunity to  check if results generalize if you check them with data from other countries.  At the beginning we acquired about 200,000 dollars for this project.  We have had more than 6.7 million page views and more than 280 users registered.  I was invited to present the project at many places various blogs and the media reported  about it.  We have a link exchanged with a very widely used project research papers in economics.  I could publish about it and the wiki was cited numerous times in academic works across  social science disciplines.  We are cooperating with a number of associations and institutions and are open to further partnerships.  For the initial phase we had student assistants generating the content.  That worked very well to show that this is a user service.  It is however costly and not easily scalable.  Now that we have explored what kind of data we can generate we would like to use machine  learning and natural language processing techniques to generate content.  First we need to identify the empirical studies in the literature.  There are also theoretical studies, book reviews and other kinds of content in academic journals  that we are not interested in so we need to sort them out.  Then we would like to classify studies by whether code and data are available.  We already have code written in R for this.  Next we would like to identify what data was used.  The witch context project has been working on this for many years and with many partners  internationally not only from academia but also from the software industry like Google,  Facebook, artificial intelligence.  The results are still far from perfect but they started a competition.  If you want a real challenge I encourage you to participate.  It might be easier for us to at least classify studies by a geographical origin of data.  Next we would like to classify studies as replications, corrections or retractions.  All this has turned out to be very useful to the community.  If there is data out there there are usually also scientists who try to analyze them and  publish about this even if it does not make too much sense.  There have already been studies that use data from the beginning.  In the first one here computer scientists try to predict replicability of empirical studies  based on the words used in them.  The reasoning behind that could be that if researchers are aware that their results are  not strong they might use language that is a bit vague and this could be detected.  In this case I could have told them right away that this would probably not work with  our data.  First, because there is so much bias towards replications we exclusively getting published  if they find something problematic in the original study that the sample is heavily influenced  by this.  Second, the types of replications are so different, some use different data, some different methods,  if the results are not the same that does not necessarily mean that there was anything  problematic about the original study.  One would need a much cleaner kind of experiment where the conditions are the same for each  replication to get a clear picture about which kind of studies are replicable and which are  not.  There was another article for which studies were used that had been retracted from manipulations.  That is a clear criterion.  Although even here, especially as long as we don't have open data and code for all empirical  research just because a study was not retracted from manipulations doesn't mean they're  random.  It is however an interesting idea to check if our retracted studies different language  was used and maybe like that one could even detect further ones that might be worth checking.  With the replications I am more skeptical.  What use does it really have if some algorithm says based on the language compared to previous  replicated studies there are some 70% chance that a study is replicable or not?  In the second case here, researchers search for replications.  They are not very open about how exactly they did that, but at least they could identify  some further replications.  Unfortunately, they then use the data they had obtained to estimate determinants of which  studies receive a published replication and drew such an ill-suited sample for this that  they ended up with the opposite of the result that can be quite easily seen from the data.  If you click on the link to their study it brings you to the respective page in the wiki  where my comments on this are listed as a replication.  There is this debate whether open data leads to researchers rewriting on the data of others  that they do not fully understand and then producing questionable research with it.  It is my experience, yes this happens, but in my opinion this should not be used as an  excuse not to open data because that leads to the much worse state of science in which  practically no published result is easily testable and thus reliable.  What is needed is better post-publication review and discussion to root out errors and build  consensus.  We now plan to massively expand our project to include all empirical work in various social  sciences fields that was published in major journals that keep online archives of data  in code.  For a related project, colleagues used machine learning techniques to collect information  about studies in machine learning.  They made quite some money with that.  So we got inspired and now plan to read in data from journal websites and bibliographical  databases.  We already have code written in Python to create wiki pages in an automated way.  What we now need is self-learning algorithms that detect relevant content from the available  information and bring it into a form that is helpful to our users.  What we are looking for is developers who have us to improve the wiki architecture or  to generate content with machine learning techniques.  We are currently raising funding and turn towards foundations, academic institutions,  but also investors, as we saw with the related project, that work like ours has quite some  value on the market, as one can also see from the usage numbers.  You as a first step could just register to show your support.  Ideally you could add content or work on the software, or just make a small contribution  and vote on which studies you think should be replicated.  I thank you very much for your attention.  If you are interested, I have some further slides with literature that can be downloaded,  but now I hope for a lively discussion.
open_research_replication_wiki_0_755.png open_research_replication_wiki_756_935.png open_research_replication_wiki_936_2219.png open_research_replication_wiki_2220_3455.png open_research_replication_wiki_3456_3851.png open_research_replication_wiki_3852_4967.png open_research_replication_wiki_4968_5771.png open_research_replication_wiki_5772_6815.png open_research_replication_wiki_6816_8051.png open_research_replication_wiki_8052_9359.png open_research_replication_wiki_9360_11339.png open_research_replication_wiki_11340_12371.png open_research_replication_wiki_12372_12479.png open_research_replication_wiki_12480_14207.png open_research_replication_wiki_14208_15035.png open_research_replication_wiki_15036_16871.png open_research_replication_wiki_16872_17519.png open_research_replication_wiki_17520_17675.png open_research_replication_wiki_17676_18155.png open_research_replication_wiki_18156_19415.png open_research_replication_wiki_19416_24215.png open_research_replication_wiki_24216_24731.png open_research_replication_wiki_24732_25019.png open_research_replication_wiki_25020_25671.png
Welcome to this presentation about the Replication Wiki, which informs on Empirical Studies in the Social Sciences. I founded the Wiki in 2013 during my PhD studies in Economics. To motivate the stock, first let me try to convince you why we need this Wiki, then I'll show you how it works and which stock we use, and finally what we plan for the future and thus where you could contribute. To give you an example of what this is all about, let's think of something that is really relevant to each of us at the moment. The Corona Crisis. It's not only a health crisis. To protect the population we have lockdowns in many countries. This has the side effect that we cannot admit in person now, which also has good sides as like that more people can participate and form anywhere. But in general it freezes the economy, it causes unemployment and a massive loss of income. There are many social science aspects to the question how to best react to this crisis. From an economics perspective some may say let there be creative destruction, survival of the fittest, those that manage challenges like a crisis best are probably the most able leaders in other circumstances as well, so they should be awarded for their success and prospering and for the others there will be some trickle down effect. Others may find such an approach would involve too much hardship to bear, there might even be unrest going much further than mass demonstrations of yellow vests or people storming parliament buildings as was seen during the Great Depression in Argentina. One proposed solution is the universal basic income. Something it would be too costly, people might just become lazy. Others say it saves an enormous amount of administrative cost because no one has to check who actually leads her. If you better when not under pressure or social stigma so they can do what they really want and might even become more productive. Like in sports and politics and economic policy everyone has an opinion. Pope Francis who saw in his home country what severe crisis can do to people suggested in a recent book that the universal basic income could be part of a solution. But how do we find a scientific answer to the question whether a universal basic income can happen in times of crisis? What economists do is build a model of the world and try to argue why different factors are related in certain ways and how different policies would have different effects. Some say that such models can always be tweaked in a way that they predict what the researcher wants to show. So an additional approach is to investigate numbers from the real world, work with data, that's what we call empirical work. There are still many different approaches, what numbers should be used and how they should be investigated and many studies trying to answer the same question come to different conclusions. That can be frustrating. What many people may not know is that traditionally social scientists just made their calculations, published the results in academic journals and then the readers had to believe them or make their own calculations. But the data and the code of the calculations were not shared. So in case different studies came to different results on the same question it was difficult to identify the exact reasons for the discrepancies. So politicians could just pick studies that came to their preferred result and use them to justify the introduction of the policies that they wanted to implement anyway. In 1986 then researchers used an archive that an editor of a journal had kept of all the data and code of empirical studies published and they tried to re-run the calculations to see if they could get the same results. That's what we call a replication of the study. It turned out that in most cases the original results could not be replicated and a big discussion started why this was the case and what could be done about it. And this interesting thing the internet became accessible to more and more people and the first journal started to just put the data online so anyone could check results. It took many years until more and more journals followed that example and there are still many journals that do not enforce rules that empirical work should be accompanied by the underlying data and code. There are also cases of confidential data, think of medical data or census data. Many people would not want their income data to be available to anyone in an online appendix or some academic journal article. There's also the case of proprietary data. If a firm collects valuable data and sells it to researchers these may get valuable scientific insights from the data but will not be allowed to share them. This is basically why I set up the replication wiki. I wanted to give an overview of empirical studies and the data and code availability and also of replication studies and their results. The case that attracted a lot of attention to replication centered around a study by two economists of Harvard University on the relationship between government debt and economic growth. It became influential in politics as it presented an easy to understand stylized fact that can be used to support the case for austerity policies. The economies of countries with government debt higher than 90% of GDP grow much less. Thomas Hinton, who you see here, was a PhD student at the time and found that in the exercise used for the analysis certain data had been left out. He argued that after correcting for this and making other adjustments that he regarded necessary the stylized fact disappeared. Thomas gave lots of media interviews around the world and was invited to present his work at many institutions. We gave a workshop together on replication in San Francisco. He became economics professor at Loyola Merriman University in Los Angeles by now. One of the authors of the original study now is chief economist of the World Bank. There is still an ongoing discussion on how the data on growth and debt should be interpreted and what we can conclude for our current situation. What we can certainly learn from this is the following. Every non-economist can find errors in exo sheets. If it can be as easy as that to initiate a worldwide discussion about pivotal questions in economics then more transparency by making such material available can definitely help to improve the quality of empirical social science research. So this is why I founded the replication wiki, whose main page you see here. It is a database of by now more than 4500 studies from the social sciences for which empirical methods were used. It lists which of the studies have data and code available online. In cases where replications are known, they are classified by their type and results. At the bottom here you see which software was used. Media wiki is the software that is also used for Wikipedia. It works well for text-based collaborative pages with images. What we wanted is a kind of database structure that allows for complex searches. Typical example is an instructor who searches for studies that have replication material available, employ a method that is truly taught, use the software that is available to the students and cover a topic that is regarded interesting. To research like that in Wikipedia, say you have seen a band with keyboard, guitar and drums and the single first name was Jim, all the information is there but you can only use full text search and categories and won't get a nice list of all measures. Semantic media wiki allows such a structure. For example, what software was used is stored as a property of each page for a study and you can easily search for all other studies that have the same property. The number of extensions help us. For example, a form extension allows users to just add information to predefined fields. No markup language is needed that deters unexperienced users. Some of the extensions could benefit from adjustment to our purposes. For example here you see this form that we use to enter new data that allows easy editing but can only produce one new page at a time. It would be good for us if there was an option to create several pages at once with such a form. And here you see why. This is a typical page for a replication. On the top you see bibliographical information, the second line informs about availability of replication material, what methods, data and software were used and the third line informs about the original study that was replicated. That study also has its page on the wiki with the same information about it and potentially more, for example if there was also another replication on it. It would be great if a form could change information on both these pages at once because currently everything has to be entered twice. That causes redundancy, potentially contradicting information if errors are made when entering the data. It's probably not difficult to find a solution but we would need someone to work on this with us. On top of what I just explained we also face other technical challenges. One is that our forms cannot be used for relations between more than two studies. I already mentioned that some studies get replicated more than once, some replications also replicate more than one original study and some replications get also replicated. So instead of one-to-one relationships we need M-to-N relationships and we do that with templates and markup code but that is user unfriendly and creates redundancies. We already found an extension for multiple instance templates but we need to implement what we want here. We would like to create automated links for selected content of properties. For our authors you can just click on them and then see all their other studies that are covered by the wiki. For data sources for example we also have information like which years they refer to in the same property field. For the years it doesn't make sense to create links but for the data sources it would be useful so it would be great to have a solution for this. Finally we would like to work on our search options. This is our advanced search form which has many options and allows for complex searches. Works great for me but it's great to complicate it with its syntax and very user friendly for others who just want to try out if this is a user service for them. This year is based on a very useful extension called drill down. You can just click on categories and select values of properties. Unfortunately it's not always possible to click on multiple values of properties. Say you search for all studies from 2014 and those of 2015. I found out you can type that into the URL then it works but again that is very user unfriendly. I wrote a bug report and the developer who created the extension even showed personal interest in our wiki and interviewed me about it but he's busy with other stuff and at the moment no one is working on fixing this. In spite of some technical imperfections the wiki already helps to build on existing research. You can compare your results to those that were already published and find out what material is available. If more than one study already addressed the same question you can see if the results were the same and if not what the reasons for this were. For example different methods or data or if there was an error in the study or if different authors just interpret results differently. The wiki helps to identify practical examples for social science education as one no longer has to do the tedious work to go through all the single journal archives to find studies that could be useful for our course. We already had teaching corporations with universities around the world. We see online that the wiki is mentioned in instructions for courses also at institutions that we did not have direct contact with and we were recommended by learned societies and libraries. We also used the wiki for some of our own research for example here I showed how the use of software is distributed in economics. In our overview of databases used you can see that the economics literature is very much dominated by the red ones from the United States so there's a lot of opportunity to check if results generalize if you check them with data from other countries. At the beginning we acquired about 200,000 dollars for this project. We have had more than 6.7 million page views and more than 280 users registered. I was invited to present the project at many places various blogs and the media reported about it. We have a link exchanged with a very widely used project research papers in economics. I could publish about it and the wiki was cited numerous times in academic works across social science disciplines. We are cooperating with a number of associations and institutions and are open to further partnerships. For the initial phase we had student assistants generating the content. That worked very well to show that this is a user service. It is however costly and not easily scalable. Now that we have explored what kind of data we can generate we would like to use machine learning and natural language processing techniques to generate content. First we need to identify the empirical studies in the literature. There are also theoretical studies, book reviews and other kinds of content in academic journals that we are not interested in so we need to sort them out. Then we would like to classify studies by whether code and data are available. We already have code written in R for this. Next we would like to identify what data was used. The witch context project has been working on this for many years and with many partners internationally not only from academia but also from the software industry like Google, Facebook, artificial intelligence. The results are still far from perfect but they started a competition. If you want a real challenge I encourage you to participate. It might be easier for us to at least classify studies by a geographical origin of data. Next we would like to classify studies as replications, corrections or retractions. All this has turned out to be very useful to the community. If there is data out there there are usually also scientists who try to analyze them and publish about this even if it does not make too much sense. There have already been studies that use data from the beginning. In the first one here computer scientists try to predict replicability of empirical studies based on the words used in them. The reasoning behind that could be that if researchers are aware that their results are not strong they might use language that is a bit vague and this could be detected. In this case I could have told them right away that this would probably not work with our data. First, because there is so much bias towards replications we exclusively getting published if they find something problematic in the original study that the sample is heavily influenced by this. Second, the types of replications are so different, some use different data, some different methods, if the results are not the same that does not necessarily mean that there was anything problematic about the original study. One would need a much cleaner kind of experiment where the conditions are the same for each replication to get a clear picture about which kind of studies are replicable and which are not. There was another article for which studies were used that had been retracted from manipulations. That is a clear criterion. Although even here, especially as long as we don't have open data and code for all empirical research just because a study was not retracted from manipulations doesn't mean they're random. It is however an interesting idea to check if our retracted studies different language was used and maybe like that one could even detect further ones that might be worth checking. With the replications I am more skeptical. What use does it really have if some algorithm says based on the language compared to previous replicated studies there are some 70% chance that a study is replicable or not? In the second case here, researchers search for replications. They are not very open about how exactly they did that, but at least they could identify some further replications. Unfortunately, they then use the data they had obtained to estimate determinants of which studies receive a published replication and drew such an ill-suited sample for this that they ended up with the opposite of the result that can be quite easily seen from the data. If you click on the link to their study it brings you to the respective page in the wiki where my comments on this are listed as a replication. There is this debate whether open data leads to researchers rewriting on the data of others that they do not fully understand and then producing questionable research with it. It is my experience, yes this happens, but in my opinion this should not be used as an excuse not to open data because that leads to the much worse state of science in which practically no published result is easily testable and thus reliable. What is needed is better post-publication review and discussion to root out errors and build consensus. We now plan to massively expand our project to include all empirical work in various social sciences fields that was published in major journals that keep online archives of data in code. For a related project, colleagues used machine learning techniques to collect information about studies in machine learning. They made quite some money with that. So we got inspired and now plan to read in data from journal websites and bibliographical databases. We already have code written in Python to create wiki pages in an automated way. What we now need is self-learning algorithms that detect relevant content from the available information and bring it into a form that is helpful to our users. What we are looking for is developers who have us to improve the wiki architecture or to generate content with machine learning techniques. We are currently raising funding and turn towards foundations, academic institutions, but also investors, as we saw with the related project, that work like ours has quite some value on the market, as one can also see from the usage numbers. You as a first step could just register to show your support. Ideally you could add content or work on the software, or just make a small contribution and vote on which studies you think should be replicated. I thank you very much for your attention. If you are interested, I have some further slides with literature that can be downloaded, but now I hope for a lively discussion.
should be used and how they should be difficult to identify the exact reasons for the wanted to give an overview of empirical studies and the data and code availability and also of replication studies and their results. The case an ongoing discussion on how the data on growth quality of empirical social science research. So now more than 4500 studies from the social software was used. Media wiki is the software What we wanted is a kind of database structure that is available to the students and cover a measures. Semantic media wiki allows such a you can easily search for all other studies that information, the second line informs about methods, data and software were used and the technical challenges. One is that our forms other studies that are covered by the wiki. For data sources for example we also have information to those that were already published and find out results differently. The wiki helps to identify practical examples for social science education for some of our own research for example here I more than 280 users registered. I was invited to We are cooperating with a number of associations and institutions and are open to further learning and natural language processing identify the empirical studies in the literature. least classify studies by a geographical origin as replications, corrections or retractions. All replicability of empirical studies based on the ones that might be worth checking. With the the language compared to previous replicated questionable research with it. It is my social sciences fields that was published in content with machine learning techniques. We are