Arcas - Using Python to access open research literature
Glynatsi, Nikoleta E.
Literature collection is a crucial process in all studies. It allows for the comparison in the historical context of the research as well as how the research is different or original from what others have done. Not several years ago, assembling sources was not an easy task. A major part of the problem was overcome with the creation of various scholarly databases and collections that live on the web. The beauty of programming is that it allows repeated tasks such as harvesting the web for data to be automatised; that is how the web scraping or web data extraction was created. Similarly, enabling an automatised process to scrape through the scholarly databases would greatly benefit researchers. This has inspired the development of an open source library called Arcas which allows for web scraping of academic articles using open access APIs. APIs such PLOS, IEEE, arXiv etc. My proposal for EuroScipy 2017 is to introduce the library Arcas and how scraping online APIs was implemented in a sustainable and reproducible manner using Python. Furthermore, to test the abilities of the library a data set was collected on a specific topic. I am also proposing to give a very brief analysis on this data set using both supervised and unsupervised machine learning algorithms.
 Hello. Hi. My name is Nicoletta Glinazzi and I'm here today to talk to you about ARCAS and this is ARCAS's logo. Obviously, there's another Euroseye by one. And I'm going to be talking to you about using Python to access Open Research Literature. I would like to start off by just saying a few things about myself. I am a first year PhD student at Cardiff University and there I study game theory, which is what this logo stands for. I'm here today not only representing Cardiff University but also the Software Sustainability Institute, which is an institute in the UK that tries to make sure that all researchers such as myself use reproducible and sustainable software in the UK region. So I started my PhD studies nearly a year ago. Yeah. And one of the first things that I came across was an illustrated guide to a PhD by Mike. Just to summarize what the guide is saying, I'll be using this picture. So I would like you to imagine that all the human knowledge is between the gray outer circle. So that's where the bounds of the human knowledge lies. And what Matt said was okay, in the first years of education, when you go to primary school and then to high school, the way our education is expanding our knowledge is in a circular way. And that is because we've been taught various subjects from literature to science and even gymnastics, right? But once you start your undergraduate level degree, you start deviating towards a very specific direction. And this little first arrow is just me choosing to do my undergraduate in business analytics and information systems. So then I managed to push just a bit further when I did my master's degree. And then it was a big question, how do I get from there all the way to the bounds? And that is what a PhD is. And the big difference with the PhD and the rest years of education is the first time you don't have someone there to teach you. And how do you show how do we do that? How do we do this leap of faith? And that's actually, you do that by doing literature review. So literature review is going back, reading what other researchers have done in your field, understanding the historical concepts of your field, and maybe just maybe at the end of the day, you will manage to boost that bound just a bit further. But how do we conduct a literature review? So we read articles. And though I'm all, I'm so you're all familiar with the process, please let me go through it again. A researcher, a researcher comes and he, he or she develops writes a paper and the paper will go through the journal. The journal is not only in charge for reviewing the article, but also publishing it. The review process is a back and forward process between the journal and the researcher. And once a journal gets a thumbs up by the reviewers, then he publishes the paper. So once a paper has been published, how do we get to find it? Well, in 2017, and there are many journals have the websites online and the databases and so are preprint, uh, uh, Greenpeace service as well. So just a demonstration, let's just say that I've been looking for articles that have to do with sustainable software. I'm going to demonstrate what I would do. I would, I would open my browser. I would go to archive, which is a print pitch event is just an example. I would go at the search here. I would type what I want. I want sustainable software. I would present her and I would have some answers back and you can see I can get various articles. One of them is visit experiences with sustainable software. Let's say I want to keep track of what I'm doing. I want to keep track of the articles. I would open up a date and editor and I will start copy pasting the information that I want. And let's just say that I want the metadata for now. And the metadata is the title is the name of the authors is the year is the journal. So I want all this information. Let's just say I want to do that for a hundred articles. So it would take me approximately zero point five minutes to open my web browser and do my search. But I don't take me one point five minutes to copy for each article all the information that I want into an editor. And that would be one point five times 10, uh, times a hundred. And then just imagine that the journal has pages, right? So you have to click to the next page and keep it reloads together next article. So it would take me two hours and three and a half and 35 minutes to get the information that I wanted. Is there a way to do this faster? And yes, there is. There's something that we call API. So an API is just a way from the user to communicate with the database of the journal straight away. And you do that by writing a query. And so you talk to the API. So back in my example again, I want to talk to the archive API. Another archive API has a standard URL that you have to answer. I have to type there. And then all I'm just saying is, okay, I want in the title, the sustainable software to exist. 100% is just the space character here. So the API will say, okay, this is your results. And we can see visit experiences with sustainable software. The article is still there. So this is an XML file. And the problem with that is that it will take me some time to actually take the information that I want from the XML file to parse it. So let's say I'm just going to write a script and it's going to do it for me. So it will take me 15 minutes to learn to speak to the API. It will take me one minute to write the query. But it will take me 15 minutes approximately to understand what the answer looks like was the information I want and write my scripts. So we're down to one hour and six minutes, which is way better than before. But there's another question now. What if I'm not only looking for articles from archive, but I want to go to various other journals or reprint service as well? Well, I'm going to have to learn to speak to the other ones. So that's the query for the archive. And this one is for plus. So we can see the difference is small, right? Instead of the standard URL still exists. And then instead of TI, now write title. But there are other APIs and the way to speak to them is even more complicated. But it's not just three APIs, right? It's more than six. And it's not just the way you speak to them that is different, but it's also the answer you get back that is different. So that's the problem that I came across when I was contacting my literature review. And I thought, oh, what am I going to do now? And I decided to build a library for this. So Archis is a library that works as a translator between the user and the APIs. You can pip install Archis and all you do is I can import it. I'll say again, in our example, create an object that knows how to speak to archive. Then I'm going to talk to our cause like I'm a human being. So I'm just going to say title is this, I want one record and I want to be the first record in line. Our case going to create the URL in the way he knows our archive speaks, is going to get the request is going to make the request going to get the answer and is going to translate it again in the way he knows our kind of speech. And I'm saving to a JSON file. JSON file just because I think it's more readable and it works good with large data. So here's the answer I'm getting. And once again, we can see the visit experiences with some state of the software is there. But what happens now if you want to speak to more, we said Archis is a translator, right? So it's quite easy to speak to the other APIs. All I have to do is write a list with the APIs I want to create objects from. And the only piece of code that changes is this one because now I'm writing to a different file and the only thing, there's a try and expect, except, because if the API doesn't have software with the articles with this title, it's going to bring a Boolean variable so the type is going to be wrong. So it's going to pass. So now it will take me 15 minutes to read about Archis, write my script, it will take me five minutes to run it. So I'm down to 20 minutes to collect the articles that I wanted. So I actually did that. I did run a script and I did look for articles that have sustainable software in the title because one of the questions you can get is, okay, you're not actually collecting the article itself. I'm just collecting metadata. So what's the value of these metadata? So here we can see that I collected 87 articles in total and I created a time plot. Why? Because well, thing is interesting. Now, you can see that the first article that I collected was in 2004, which is quite recent. So the articles that contain sustainable software are quite recent. Also we can look at the provenance. And here we can see that I tried four different journals, but I only had answers for my triple E, archive and plus. And another thing we can also do, which I find quite interesting, we can look at the core authors' connections. So this plot doesn't really look really good, but it's actually really interesting because the cycles are the authors that are in the data set that I collected. And if the two circles have a line between, it means that these two people wrote together. Why is quite interesting here that we can see that some clusters exist, like this one and this one here. And also what's really interesting, we can see that this person is connecting to two clusters. It would be nice to see who influenced who and how. And that person happens to be Beatrice. I'm not familiar with the lady, unfortunately. And going back to the workshop that we attended these days, things like NetworkX, all these, we measure the degree centrality and all these things or machine learning algorithms to categorize the article, all the abstract, all these methods could be applied to the data that you are collecting. Just to finish off, I would like to tell you a few things about how ARCAS work. So each API is implemented in a folder itself and it has the methods that let ARCAS know how to speak to each API. If there are similar methods, they will be written in tools.py. And what is actually nice is really easy to contribute. So if you wanted just to implement an extra API, you would create a folder here and you would make sure you test it. And just following best practice, of course, ARCAS has a documentation and you can find our line at ARCAS read the docs.io. Just finish off, ARCAS also works as a command line too. So here again in our example, I'm just saying take the API archive with the title to be sustainable software and I want you to return me one record. All the information about ARCAS and myself, you can find here and thank you very much. Thanks. Thank you for the talk. What about Google Scholar? If I want to implement Google Scholar, comparing Google. The API that you can include in the project? So yes, there are various APIs. There is an issue on ARCAS with all the listing of the APIs. Google Scholar is one of them. Last time, which was nearly a year ago, I tried there were some issues with Google Scholar that I couldn't get through. But it isn't the issue. So I'll be glad to take a look if I could implement. Yeah, it looks like it's really cool. I've also been having a lot of issues doing better literature research. Do you have any means of looking at citation metrics or something like that? Like could I say, okay, show me the most cited articles containing these keywords or something like that. There's many services where you can do that from your browser, but they're all super slow and super annoying. It would be so much cooler if you could do this from your terminal. So the only issue with this, because I was trying to score my articles, it's not all APIs return the number of citations. PLOS actually returns a score that they conduct themselves. But it's not easy to get the number because the API itself doesn't return it. And that would be something else that I would have to do. But it doesn't come back as a result if I answer your questions. Is a question up. For recording, it would help if you use the mic. Thanks. Hi. Thank you very much for your nice talk. Just a quick question. I've been using, through you at my degree program, Mendeley. So I do not really see the difference because there I can basically do exactly what you do, but with a nice GUI. With a nice GUI instead of using Python and this command line interface. So I use Mendeley to get my citations. But I've never, I didn't see how I could be wrong. Mendeley does not return me the data and the format that I like, that I want to have this big data set to collect to then analyze my data sets. And again, Mendeley, I have to find the article and then say, okay, Mendeley, he's an article, take it. With our because I can just randomly look for a specific title and it just saves everything for me. You can do that. You can search in Mendeley for sustainable software and just add everything to your library and export it to Leitach, to your Bibtech file or whatever. So a good argument. Can I just go software sustainable, save me 100 articles in my library? Some of these articles are obviously behind a paywall, but at least you got the meta information and then can have a look into you. That would be something to look at. And that's, sorry. Thanks.
 Hello. Hi. My name is Nicoletta Glinazzi and I'm here today to talk to you about ARCAS  and this is ARCAS's logo. Obviously, there's another Euroseye by one. And I'm going to  be talking to you about using Python to access Open Research Literature. I would like to start  off by just saying a few things about myself. I am a first year PhD student at Cardiff University  and there I study game theory, which is what this logo stands for. I'm here today not only  representing Cardiff University but also the Software Sustainability Institute, which is  an institute in the UK that tries to make sure that all researchers such as myself use  reproducible and sustainable software in the UK region. So I started my PhD studies  nearly a year ago. Yeah. And one of the first things that I came across was an illustrated  guide to a PhD by Mike. Just to summarize what the guide is saying, I'll be using this  picture. So I would like you to imagine that all the human knowledge is between the gray  outer circle. So that's where the bounds of the human knowledge lies. And what Matt said  was okay, in the first years of education, when you go to primary school and then to  high school, the way our education is expanding our knowledge is in a circular way. And that  is because we've been taught various subjects from literature to science and even gymnastics,  right? But once you start your undergraduate level degree, you start deviating towards  a very specific direction. And this little first arrow is just me choosing to do my  undergraduate in business analytics and information systems. So then I managed to push just a  bit further when I did my master's degree. And then it was a big question, how do I get  from there all the way to the bounds? And that is what a PhD is. And the big difference  with the PhD and the rest years of education is the first time you don't have someone there  to teach you. And how do you show how do we do that? How do we do this leap of faith?  And that's actually, you do that by doing literature review. So literature review is going back,  reading what other researchers have done in your field, understanding the historical concepts  of your field, and maybe just maybe at the end of the day, you will manage to boost that  bound just a bit further. But how do we conduct a literature review? So we read articles.  And though I'm all, I'm so you're all familiar with the process, please let me go through  it again. A researcher, a researcher comes and he, he or she develops writes a paper  and the paper will go through the journal. The journal is not only in charge for reviewing  the article, but also publishing it. The review process is a back and forward process between  the journal and the researcher. And once a journal gets a thumbs up by the reviewers,  then he publishes the paper. So once a paper has been published, how do we get to find  it? Well, in 2017, and there are many journals have the websites online and the databases  and so are preprint, uh, uh, Greenpeace service as well. So just a demonstration, let's just  say that I've been looking for articles that have to do with sustainable software. I'm  going to demonstrate what I would do. I would, I would open my browser. I would go to archive,  which is a print pitch event is just an example. I would go at the search here. I would type  what I want. I want sustainable software. I would present her and I would have some answers  back and you can see I can get various articles. One of them is visit experiences with sustainable  software. Let's say I want to keep track of what I'm doing. I want to keep track of the  articles. I would open up a date and editor and I will start copy pasting the information  that I want. And let's just say that I want the metadata for now. And the metadata is  the title is the name of the authors is the year is the journal. So I want all this information.  Let's just say I want to do that for a hundred articles. So it would take me approximately  zero point five minutes to open my web browser and do my search. But I don't take me one point  five minutes to copy for each article all the information that I want into an editor.  And that would be one point five times 10, uh, times a hundred. And then just imagine  that the journal has pages, right? So you have to click to the next page and keep it  reloads together next article. So it would take me two hours and three and a half and  35 minutes to get the information that I wanted. Is there a way to do this faster? And yes,  there is. There's something that we call API. So an API is just a way from the user to communicate  with the database of the journal straight away. And you do that by writing a query. And so  you talk to the API. So back in my example again, I want to talk to the archive API.  Another archive API has a standard URL that you have to answer. I have to type there.  And then all I'm just saying is, okay, I want in the title, the sustainable software to  exist. 100% is just the space character here. So the API will say, okay, this is your results.  And we can see visit experiences with sustainable software. The article is still there. So this  is an XML file. And the problem with that is that it will take me some time to actually  take the information that I want from the XML file to parse it. So let's say I'm just  going to write a script and it's going to do it for me. So it will take me 15 minutes  to learn to speak to the API. It will take me one minute to write the query. But it will  take me 15 minutes approximately to understand what the answer looks like was the information  I want and write my scripts. So we're down to one hour and six minutes, which is way  better than before. But there's another question now. What if I'm not only looking for articles  from archive, but I want to go to various other journals or reprint service as well?  Well, I'm going to have to learn to speak to the other ones. So that's the query for  the archive. And this one is for plus. So we can see the difference is small, right?  Instead of the standard URL still exists. And then instead of TI, now write title. But there  are other APIs and the way to speak to them is even more complicated. But it's not just  three APIs, right? It's more than six. And it's not just the way you speak to them that  is different, but it's also the answer you get back that is different. So that's the  problem that I came across when I was contacting my literature review. And I thought, oh, what  am I going to do now? And I decided to build a library for this. So Archis is a library  that works as a translator between the user and the APIs. You can pip install Archis and  all you do is I can import it. I'll say again, in our example, create an object that knows  how to speak to archive. Then I'm going to talk to our cause like I'm a human being.  So I'm just going to say title is this, I want one record and I want to be the first  record in line. Our case going to create the URL in the way he knows our archive speaks,  is going to get the request is going to make the request going to get the answer and is  going to translate it again in the way he knows our kind of speech. And I'm saving to  a JSON file. JSON file just because I think it's more readable and it works good with  large data. So here's the answer I'm getting. And once again, we can see the visit experiences  with some state of the software is there. But what happens now if you want to speak  to more, we said Archis is a translator, right? So it's quite easy to speak to the other  APIs. All I have to do is write a list with the APIs I want to create objects from. And  the only piece of code that changes is this one because now I'm writing to a different  file and the only thing, there's a try and expect, except, because if the API doesn't  have software with the articles with this title, it's going to bring a Boolean variable  so the type is going to be wrong. So it's going to pass. So now it will take me 15  minutes to read about Archis, write my script, it will take me five minutes to run it. So  I'm down to 20 minutes to collect the articles that I wanted. So I actually did that. I did  run a script and I did look for articles that have sustainable software in the title because  one of the questions you can get is, okay, you're not actually collecting the article  itself. I'm just collecting metadata. So what's the value of these metadata? So here  we can see that I collected 87 articles in total and I created a time plot. Why? Because  well, thing is interesting. Now, you can see that the first article that I collected was  in 2004, which is quite recent. So the articles that contain sustainable software are quite  recent. Also we can look at the provenance. And here we can see that I tried four different  journals, but I only had answers for my triple E, archive and plus. And another thing we  can also do, which I find quite interesting, we can look at the core authors' connections.  So this plot doesn't really look really good, but it's actually really interesting because  the cycles are the authors that are in the data set that I collected. And if the two  circles have a line between, it means that these two people wrote together. Why is quite  interesting here that we can see that some clusters exist, like this one and this one  here. And also what's really interesting, we can see that this person is connecting to  two clusters. It would be nice to see who influenced who and how. And that person happens  to be Beatrice. I'm not familiar with the lady, unfortunately. And going back to the  workshop that we attended these days, things like NetworkX, all these, we measure the degree  centrality and all these things or machine learning algorithms to categorize the article,  all the abstract, all these methods could be applied to the data that you are collecting.  Just to finish off, I would like to tell you a few things about how ARCAS work. So each  API is implemented in a folder itself and it has the methods that let ARCAS know how  to speak to each API. If there are similar methods, they will be written in tools.py.  And what is actually nice is really easy to contribute. So if you wanted just to implement  an extra API, you would create a folder here and you would make sure you test it. And just  following best practice, of course, ARCAS has a documentation and you can find our line  at ARCAS read the docs.io. Just finish off, ARCAS also works as a command line too. So  here again in our example, I'm just saying take the API archive with the title to be  sustainable software and I want you to return me one record. All the information about ARCAS  and myself, you can find here and thank you very much.  Thanks. Thank you for the talk. What about Google Scholar?  If I want to implement Google Scholar, comparing Google.  The API that you can include in the project?  So yes, there are various APIs. There is an issue on ARCAS with all the listing of the  APIs. Google Scholar is one of them. Last time, which was nearly a year ago, I tried  there were some issues with Google Scholar that I couldn't get through. But it isn't  the issue. So I'll be glad to take a look if I could implement.  Yeah, it looks like it's really cool. I've also been having a lot of issues doing better  literature research. Do you have any means of looking at citation metrics or something  like that? Like could I say, okay, show me the most cited articles containing these keywords  or something like that. There's many services where you can do that from your browser, but  they're all super slow and super annoying. It would be so much cooler if you could do  this from your terminal.  So the only issue with this, because I was trying to score my articles, it's not all  APIs return the number of citations. PLOS actually returns a score that they conduct themselves.  But it's not easy to get the number because the API itself doesn't return it. And that  would be something else that I would have to do. But it doesn't come back as a result  if I answer your questions. Is a question up.  For recording, it would help if you use the mic. Thanks.  Hi. Thank you very much for your nice talk. Just a quick question. I've been using, through  you at my degree program, Mendeley. So I do not really see the difference because there  I can basically do exactly what you do, but with a nice GUI.  With a nice GUI instead of using Python and this command line interface.  So I use Mendeley to get my citations. But I've never, I didn't see how I could be wrong.  Mendeley does not return me the data and the format that I like, that I want to have this  big data set to collect to then analyze my data sets. And again, Mendeley, I have to  find the article and then say, okay, Mendeley, he's an article, take it. With our because  I can just randomly look for a specific title and it just saves everything for me.  You can do that. You can search in Mendeley for sustainable software and just add everything  to your library and export it to Leitach, to your Bibtech file or whatever.  So a good argument. Can I just go software sustainable, save me 100 articles in my library?  Some of these articles are obviously behind a paywall, but at least you got the meta information  and then can have a look into you. That would be something to look at.  And that's, sorry. Thanks.
EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_0_3863.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_3864_4979.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_4980_6107.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_6108_6995.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_6996_7751.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_7752_8243.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_8244_8879.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_8880_9455.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_9456_9923.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_9924_10019.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_10020_12959.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_12960_13343.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_13344_15527.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_15528_15887.png EuroSciPy_2017__Arcas_-_Using_Python_to_access_open_research_literature_MvZt5dmoJ5M_15888_21434.png
Hello. Hi. My name is Nicoletta Glinazzi and I'm here today to talk to you about ARCAS and this is ARCAS's logo. Obviously, there's another Euroseye by one. And I'm going to be talking to you about using Python to access Open Research Literature. I would like to start off by just saying a few things about myself. I am a first year PhD student at Cardiff University and there I study game theory, which is what this logo stands for. I'm here today not only representing Cardiff University but also the Software Sustainability Institute, which is an institute in the UK that tries to make sure that all researchers such as myself use reproducible and sustainable software in the UK region. So I started my PhD studies nearly a year ago. Yeah. And one of the first things that I came across was an illustrated guide to a PhD by Mike. Just to summarize what the guide is saying, I'll be using this picture. So I would like you to imagine that all the human knowledge is between the gray outer circle. So that's where the bounds of the human knowledge lies. And what Matt said was okay, in the first years of education, when you go to primary school and then to high school, the way our education is expanding our knowledge is in a circular way. And that is because we've been taught various subjects from literature to science and even gymnastics, right? But once you start your undergraduate level degree, you start deviating towards a very specific direction. And this little first arrow is just me choosing to do my undergraduate in business analytics and information systems. So then I managed to push just a bit further when I did my master's degree. And then it was a big question, how do I get from there all the way to the bounds? And that is what a PhD is. And the big difference with the PhD and the rest years of education is the first time you don't have someone there to teach you. And how do you show how do we do that? How do we do this leap of faith? And that's actually, you do that by doing literature review. So literature review is going back, reading what other researchers have done in your field, understanding the historical concepts of your field, and maybe just maybe at the end of the day, you will manage to boost that bound just a bit further. But how do we conduct a literature review? So we read articles. And though I'm all, I'm so you're all familiar with the process, please let me go through it again. A researcher, a researcher comes and he, he or she develops writes a paper and the paper will go through the journal. The journal is not only in charge for reviewing the article, but also publishing it. The review process is a back and forward process between the journal and the researcher. And once a journal gets a thumbs up by the reviewers, then he publishes the paper. So once a paper has been published, how do we get to find it? Well, in 2017, and there are many journals have the websites online and the databases and so are preprint, uh, uh, Greenpeace service as well. So just a demonstration, let's just say that I've been looking for articles that have to do with sustainable software. I'm going to demonstrate what I would do. I would, I would open my browser. I would go to archive, which is a print pitch event is just an example. I would go at the search here. I would type what I want. I want sustainable software. I would present her and I would have some answers back and you can see I can get various articles. One of them is visit experiences with sustainable software. Let's say I want to keep track of what I'm doing. I want to keep track of the articles. I would open up a date and editor and I will start copy pasting the information that I want. And let's just say that I want the metadata for now. And the metadata is the title is the name of the authors is the year is the journal. So I want all this information. Let's just say I want to do that for a hundred articles. So it would take me approximately zero point five minutes to open my web browser and do my search. But I don't take me one point five minutes to copy for each article all the information that I want into an editor. And that would be one point five times 10, uh, times a hundred. And then just imagine that the journal has pages, right? So you have to click to the next page and keep it reloads together next article. So it would take me two hours and three and a half and 35 minutes to get the information that I wanted. Is there a way to do this faster? And yes, there is. There's something that we call API. So an API is just a way from the user to communicate with the database of the journal straight away. And you do that by writing a query. And so you talk to the API. So back in my example again, I want to talk to the archive API. Another archive API has a standard URL that you have to answer. I have to type there. And then all I'm just saying is, okay, I want in the title, the sustainable software to exist. 100% is just the space character here. So the API will say, okay, this is your results. And we can see visit experiences with sustainable software. The article is still there. So this is an XML file. And the problem with that is that it will take me some time to actually take the information that I want from the XML file to parse it. So let's say I'm just going to write a script and it's going to do it for me. So it will take me 15 minutes to learn to speak to the API. It will take me one minute to write the query. But it will take me 15 minutes approximately to understand what the answer looks like was the information I want and write my scripts. So we're down to one hour and six minutes, which is way better than before. But there's another question now. What if I'm not only looking for articles from archive, but I want to go to various other journals or reprint service as well? Well, I'm going to have to learn to speak to the other ones. So that's the query for the archive. And this one is for plus. So we can see the difference is small, right? Instead of the standard URL still exists. And then instead of TI, now write title. But there are other APIs and the way to speak to them is even more complicated. But it's not just three APIs, right? It's more than six. And it's not just the way you speak to them that is different, but it's also the answer you get back that is different. So that's the problem that I came across when I was contacting my literature review. And I thought, oh, what am I going to do now? And I decided to build a library for this. So Archis is a library that works as a translator between the user and the APIs. You can pip install Archis and all you do is I can import it. I'll say again, in our example, create an object that knows how to speak to archive. Then I'm going to talk to our cause like I'm a human being. So I'm just going to say title is this, I want one record and I want to be the first record in line. Our case going to create the URL in the way he knows our archive speaks, is going to get the request is going to make the request going to get the answer and is going to translate it again in the way he knows our kind of speech. And I'm saving to a JSON file. JSON file just because I think it's more readable and it works good with large data. So here's the answer I'm getting. And once again, we can see the visit experiences with some state of the software is there. But what happens now if you want to speak to more, we said Archis is a translator, right? So it's quite easy to speak to the other APIs. All I have to do is write a list with the APIs I want to create objects from. And the only piece of code that changes is this one because now I'm writing to a different file and the only thing, there's a try and expect, except, because if the API doesn't have software with the articles with this title, it's going to bring a Boolean variable so the type is going to be wrong. So it's going to pass. So now it will take me 15 minutes to read about Archis, write my script, it will take me five minutes to run it. So I'm down to 20 minutes to collect the articles that I wanted. So I actually did that. I did run a script and I did look for articles that have sustainable software in the title because one of the questions you can get is, okay, you're not actually collecting the article itself. I'm just collecting metadata. So what's the value of these metadata? So here we can see that I collected 87 articles in total and I created a time plot. Why? Because well, thing is interesting. Now, you can see that the first article that I collected was in 2004, which is quite recent. So the articles that contain sustainable software are quite recent. Also we can look at the provenance. And here we can see that I tried four different journals, but I only had answers for my triple E, archive and plus. And another thing we can also do, which I find quite interesting, we can look at the core authors' connections. So this plot doesn't really look really good, but it's actually really interesting because the cycles are the authors that are in the data set that I collected. And if the two circles have a line between, it means that these two people wrote together. Why is quite interesting here that we can see that some clusters exist, like this one and this one here. And also what's really interesting, we can see that this person is connecting to two clusters. It would be nice to see who influenced who and how. And that person happens to be Beatrice. I'm not familiar with the lady, unfortunately. And going back to the workshop that we attended these days, things like NetworkX, all these, we measure the degree centrality and all these things or machine learning algorithms to categorize the article, all the abstract, all these methods could be applied to the data that you are collecting. Just to finish off, I would like to tell you a few things about how ARCAS work. So each API is implemented in a folder itself and it has the methods that let ARCAS know how to speak to each API. If there are similar methods, they will be written in tools.py. And what is actually nice is really easy to contribute. So if you wanted just to implement an extra API, you would create a folder here and you would make sure you test it. And just following best practice, of course, ARCAS has a documentation and you can find our line at ARCAS read the docs.io. Just finish off, ARCAS also works as a command line too. So here again in our example, I'm just saying take the API archive with the title to be sustainable software and I want you to return me one record. All the information about ARCAS and myself, you can find here and thank you very much. Thanks. Thank you for the talk. What about Google Scholar? If I want to implement Google Scholar, comparing Google. The API that you can include in the project? So yes, there are various APIs. There is an issue on ARCAS with all the listing of the APIs. Google Scholar is one of them. Last time, which was nearly a year ago, I tried there were some issues with Google Scholar that I couldn't get through. But it isn't the issue. So I'll be glad to take a look if I could implement. Yeah, it looks like it's really cool. I've also been having a lot of issues doing better literature research. Do you have any means of looking at citation metrics or something like that? Like could I say, okay, show me the most cited articles containing these keywords or something like that. There's many services where you can do that from your browser, but they're all super slow and super annoying. It would be so much cooler if you could do this from your terminal. So the only issue with this, because I was trying to score my articles, it's not all APIs return the number of citations. PLOS actually returns a score that they conduct themselves. But it's not easy to get the number because the API itself doesn't return it. And that would be something else that I would have to do. But it doesn't come back as a result if I answer your questions. Is a question up. For recording, it would help if you use the mic. Thanks. Hi. Thank you very much for your nice talk. Just a quick question. I've been using, through you at my degree program, Mendeley. So I do not really see the difference because there I can basically do exactly what you do, but with a nice GUI. With a nice GUI instead of using Python and this command line interface. So I use Mendeley to get my citations. But I've never, I didn't see how I could be wrong. Mendeley does not return me the data and the format that I like, that I want to have this big data set to collect to then analyze my data sets. And again, Mendeley, I have to find the article and then say, okay, Mendeley, he's an article, take it. With our because I can just randomly look for a specific title and it just saves everything for me. You can do that. You can search in Mendeley for sustainable software and just add everything to your library and export it to Leitach, to your Bibtech file or whatever. So a good argument. Can I just go software sustainable, save me 100 articles in my library? Some of these articles are obviously behind a paywall, but at least you got the meta information and then can have a look into you. That would be something to look at. And that's, sorry. Thanks.
using Python to access Open Research Literature. things about myself. I am a first year PhD game theory, which is what this logo stands for. tries to make sure that all researchers such as myself use reproducible and sustainable software towards a very specific direction. And this have done in your field, understanding the historical concepts of your field, and maybe just through it again. A researcher, a researcher and the paper will go through the journal. The process is a back and forward process between the websites online and the databases and so are preprint, uh, uh, Greenpeace service as well. So keep track of the articles. I would open up a API. So an API is just a way from the user to And the problem with that is that it will take me also the answer you get back that is different. a library that works as a translator between the see that the first article that I collected was are the authors that are in the data set that I see who influenced who and how. And that person centrality and all these things or machine learning algorithms to categorize the article, implemented in a folder itself and it has the folder here and you would make sure you test it. API archive with the title to be sustainable APIs return the number of citations. PLOS themselves. But it's not easy to get the number