Leveraging Open Source Designs
Mönch, Lasse
Incorporating new components into PCBs is a difficult task that often requires reading multiple datasheets and creating prototypes to get it right. The funny thing is: every engineer needs to re-read reference designs! Even though there are tens of thousands of designs with new components documented and available on Github. The reason: it is almost impossible to find a relevant project. The solution? Instead of using Github search, which only retrieves files by filename, our approach creates a local database that takes the search results from Github, and then parses the used components inside the PCB designs to index them. That way, you can easily search a component and get the most relevant designs as a reference. This talk will give an overview of the software that was created, discusses the difficulties that were overcome and the potential for improvement in future work. We have currently indexed 8,000 KiCad-projects on GitHub containing 37,000 electronic components as a proof-of-concept.
 All right. Hi everyone. Thank you for your interest in my talk. I'm Lasse. I'm a student at Avatar Aachen University and I'm going to tell you something about a project that I did during the course of a lab course at that university. So I am mainly a software developer and an electronics hobbyist, which means that I've only tinkered with Arduino compatible microcontrollers and I haven't been really into hardware design all that much. But a little over a year ago, I wanted to create my own PCB, just a simple microcontroller with some sensors and I wanted it to be better operated. So I thought, okay, creating PCBs is not that hard, right? It's just like four steps. You have to choose the components that you want to put on your PCB, use some EDA tool to create schematics, design layout, upload the files to a manufacturer and wait for the mail to arrive. All right. So I started with the first step and I went to choose my components. So as I said, I had no idea on how to do that. So I went to Reddit and I asked what components should I use to control my battery circuit. And I got a couple of suggestions. So I went off to the internet and I searched for different data sheets. I read through them. I understood half of them. And finally, I had decided for a chip that I wanted to use. And then I realized that I had no idea on how to actually implement it on the board. So I went off to GitHub and I tried to find other people's projects that incorporated the same chips. And as it turns out, it was quite difficult because the chip I was looking for wasn't the main component on the PCB. So it wasn't mentioned anywhere in the description or in the readme. And I had to actually search through the files, which isn't that simple, on GitHub. And by the time I had found the first project, it was 1 AM and I had to go to bed because I had university the next day. And then I kind of forgot about the whole project. All right. So this semester, I had the opportunity to work on an electronic component search engine that basically allows you to enter the name of an electronic component that you want to incorporate into your PCB. And it will return repositories on GitHub that have projects that include those components. All right. So let's just have a look at the result. This is what it looks like when you search for a component. On the top, you just enter the component name. And as you can see, we found about 100 repositories, including the component. This is one of them. On the top, we mentioned the repository name and the description. If the project has a license, the license is shown right there, on the top right, you have these stars and forks to give you some kind of an idea whether the project is liked and therefore possibly useful. And on the bottom, we show the relevant files, which exact component was found in the file. And we give you a quick link to the datasheet if we find one. And if you want to, you can directly go to a distributor and buy the part or even manufacture the project's PCB. All right. So let's get into the technical details a little bit. We tried to keep the software modular. We split it into four components. We have a crawler, the crawler GitHub for PCB design files. Then we have a parser that actually parses those files and checks for the components that are included. The component names are then validated. I'll tell you more on that later. And finally, we have a database that can be searched and returns the repositories that are relevant to your part search. All right. So more details and difficulties that we met. For the crawler, our idea was very simple. Just search the whole of GitHub and return all PCB files that we can find and for later indexing. However, we found out that it's not quite that simple to search all of GitHub. There are rate limits on GitHub and there's not only visible rate limits, which is like 30 searches per minute, I think. But there's also an invisible counter that somehow calculates how computationally intensive your search queries are. And if you exceed those, you'll get an abuse note and your shutdown for a while. All right. So because code search or searching inside the files turns out to be very computationally intensive for GitHub, we had to just search for repositories and readme's that mentioned certain words in the description like PCB or key cad or stuff like that. And then we searched both repos whether or not they actually contain relevant files. All right. So the parser. So the parser downloads key cat PCB files and extracts the used components from them. That was the idea, but we had to find out that the component names are actually free text fields and some people just wrote down whatever they deemed useful. So that was why we had to incorporate a validation process to find out whether the components that were entered in the PCB files are actually existing components or if it's just something like 12 to 24 volts DC-DC converter. All right. And for the validation, we also had quite a simple idea. So our idea was to use a component search, for example, one offered by a distributor and see if it does return any results and if it does return results, how many. So we just said, okay, if it returns zero results, it's probably non-existent or just some gibberish in the component text. And if it returned more than 10 results, we assumed it would be something very generic like resistor or something. However, we could not find a component search API that offered rate limits that were high enough for us to find anything. So we looked at DigiKey and they offer about a thousand requests per day for free. We also looked at OctoPart. They offer even less. They offer 500 for educational users after you register with them. And after that, it gets really expensive really quickly. Yeah, but we found a workaround by actually using Eisler's component search and a piggy backing onto a cache that they built inside their API. And we can use that without triggering any rate limits at all. All right. Here is some, well, future work or if you're already hyped, good first issues if you want to participate. I don't think do I have time for those? All right. Yeah. So if you're heating your question time, you have to do it first. Okay. Great. Yeah. So we want to move the, we developed on a private GitLab instance and we want to move that to gitlab.com, get the CI CD, which is already running on the private instance, running again. And also it's a huge monolithic mess, which we want to split up into modular repos. We want to improve the search a little bit, support other formats than Kikat. Finally I already met the maintainer of fritzing and so that might be a good next project to work on. And we also want to support other platforms than GitLab. All right. That's about it. Thank you for your attention. Just to be clear, is the intention of other reference designs using the component? So the idea is just to find example projects that have used the same components so that you can get an idea on how to use the component in your PCB. Okay. And then second part is, is there some kind of way to rate the found designs because every round again, if I look at these kinds of designs, I find something which I can see is probably not going to work in real life. So any kind of rate system? Not yet. There was one future work issue here, improve the ranking. But right now we are ranking using GitHub, Stars and Forks. So that might already give you an idea whether a project is useful or not. Yeah. So another improvement would be actually using reference designs from the manufacturer because they actually provide, I don't think they are available on one. Can you repeat to the microphone? Oh, sorry. There was a suggestion to use reference designs provided by the manufacturers of the parts. Yeah. But I would have to look into how to actually find them. As I said, I'm not a hardware specialist, but I only know from reference designs inside the PDFs. So those are usually only graphics, right? It depends about how complicated the part is. For a microcontroller, which needs, for example, an antenna, you might have a reference design. Okay. So that's something maybe worth looking into. Yeah. Have you maybe thought about offering some sort of snippet? One could just add to this repo, so it's easier to find it? Okay. The question was whether one could add some certain snippet of code or something to the repo to make it easily findable? No. I haven't thought about it. For now, it would just be enough to include PCB or something, some words like that in your repo description, and then we'll find it. But yeah, maybe some more specific keywords could be provided for that. Because I mean, all the developers on GitHub, I mean, there's some on Catlab, for example, or in other vehicles. All right, I think the question time is over. There's my, oh yeah. Sorry. You want to? Yeah, no, go ahead. One more. I saw that. So you said you were using the cache data to actually extract this. How do you actually make that work? Because both Digi-Key and Octopart explicitly forbid caching data. Well, we're not using the Digi-Key cache, right? So Isler has a component search that they use for their customers. So I didn't actually know that because we don't really interface directly with Digi-Key. So, okay, thank you so much. Thank you.
 All right.  Hi everyone.  Thank you for your interest in my talk.  I'm Lasse.  I'm a student at Avatar Aachen University and I'm going to tell you something about a project  that I did during the course of a lab course at that university.  So I am mainly a software developer and an electronics hobbyist, which means that I've  only tinkered with Arduino compatible microcontrollers and I haven't been really into hardware design  all that much.  But a little over a year ago, I wanted to create my own PCB, just a simple microcontroller  with some sensors and I wanted it to be better operated.  So I thought, okay, creating PCBs is not that hard, right?  It's just like four steps.  You have to choose the components that you want to put on your PCB, use some EDA tool  to create schematics, design layout, upload the files to a manufacturer and wait for the  mail to arrive.  All right.  So I started with the first step and I went to choose my components.  So as I said, I had no idea on how to do that.  So I went to Reddit and I asked what components should I use to control my battery circuit.  And I got a couple of suggestions.  So I went off to the internet and I searched for different data sheets.  I read through them.  I understood half of them.  And finally, I had decided for a chip that I wanted to use.  And then I realized that I had no idea on how to actually implement it on the board.  So I went off to GitHub and I tried to find other people's projects that incorporated  the same chips.  And as it turns out, it was quite difficult because the chip I was looking for wasn't  the main component on the PCB.  So it wasn't mentioned anywhere in the description or in the readme.  And I had to actually search through the files, which isn't that simple, on GitHub.  And by the time I had found the first project, it was 1 AM and I had to go to bed because  I had university the next day.  And then I kind of forgot about the whole project.  All right.  So this semester, I had the opportunity to work on an electronic component search engine  that basically allows you to enter the name of an electronic component that you want to  incorporate into your PCB.  And it will return repositories on GitHub that have projects that include those components.  All right.  So let's just have a look at the result.  This is what it looks like when you search for a component.  On the top, you just enter the component name.  And as you can see, we found about 100 repositories, including the component.  This is one of them.  On the top, we mentioned the repository name and the description.  If the project has a license, the license is shown right there, on the top right, you  have these stars and forks to give you some kind of an idea whether the project is liked  and therefore possibly useful.  And on the bottom, we show the relevant files, which exact component was found in the file.  And we give you a quick link to the datasheet if we find one.  And if you want to, you can directly go to a distributor and buy the part or even manufacture  the project's PCB.  All right.  So let's get into the technical details a little bit.  We tried to keep the software modular.  We split it into four components.  We have a crawler, the crawler GitHub for PCB design files.  Then we have a parser that actually parses those files and checks for the components  that are included.  The component names are then validated.  I'll tell you more on that later.  And finally, we have a database that can be searched and returns the repositories that  are relevant to your part search.  All right.  So more details and difficulties that we met.  For the crawler, our idea was very simple.  Just search the whole of GitHub and return all PCB files that we can find and for later  indexing.  However, we found out that it's not quite that simple to search all of GitHub.  There are rate limits on GitHub and there's not only visible rate limits, which is like  30 searches per minute, I think.  But there's also an invisible counter that somehow calculates how computationally intensive  your search queries are.  And if you exceed those, you'll get an abuse note and your shutdown for a while.  All right.  So because code search or searching inside the files turns out to be very computationally  intensive for GitHub, we had to just search for repositories and readme's that mentioned  certain words in the description like PCB or key cad or stuff like that.  And then we searched both repos whether or not they actually contain relevant files.  All right.  So the parser.  So the parser downloads key cat PCB files and extracts the used components from them.  That was the idea, but we had to find out that the component names are actually free  text fields and some people just wrote down whatever they deemed useful.  So that was why we had to incorporate a validation process to find out whether the components  that were entered in the PCB files are actually existing components or if it's just something  like 12 to 24 volts DC-DC converter.  All right.  And for the validation, we also had quite a simple idea.  So our idea was to use a component search, for example, one offered by a distributor  and see if it does return any results and if it does return results, how many.  So we just said, okay, if it returns zero results, it's probably non-existent or just  some gibberish in the component text.  And if it returned more than 10 results, we assumed it would be something very generic  like resistor or something.  However, we could not find a component search API that offered rate limits that were high  enough for us to find anything.  So we looked at DigiKey and they offer about a thousand requests per day for free.  We also looked at OctoPart.  They offer even less.  They offer 500 for educational users after you register with them.  And after that, it gets really expensive really quickly.  Yeah, but we found a workaround by actually using Eisler's component search and a piggy  backing onto a cache that they built inside their API.  And we can use that without triggering any rate limits at all.  All right.  Here is some, well, future work or if you're already hyped, good first issues if you want  to participate.  I don't think do I have time for those?  All right.  Yeah.  So if you're heating your question time, you have to do it first.  Okay.  Great.  Yeah.  So we want to move the, we developed on a private GitLab instance and we want to move  that to gitlab.com, get the CI CD, which is already running on the private instance,  running again.  And also it's a huge monolithic mess, which we want to split up into modular repos.  We want to improve the search a little bit, support other formats than Kikat.  Finally I already met the maintainer of fritzing and so that might be a good next project to  work on.  And we also want to support other platforms than GitLab.  All right.  That's about it.  Thank you for your attention.  Just to be clear, is the intention of other reference designs using the component?  So the idea is just to find example projects that have used the same components so that  you can get an idea on how to use the component in your PCB.  Okay.  And then second part is, is there some kind of way to rate the found designs because every  round again, if I look at these kinds of designs, I find something which I can see is probably  not going to work in real life.  So any kind of rate system?  Not yet.  There was one future work issue here, improve the ranking.  But right now we are ranking using GitHub, Stars and Forks.  So that might already give you an idea whether a project is useful or not.  Yeah.  So another improvement would be actually using reference designs from the manufacturer because  they actually provide, I don't think they are available on one.  Can you repeat to the microphone?  Oh, sorry.  There was a suggestion to use reference designs provided by the manufacturers of the parts.  Yeah.  But I would have to look into how to actually find them.  As I said, I'm not a hardware specialist, but I only know from reference designs inside  the PDFs.  So those are usually only graphics, right?  It depends about how complicated the part is.  For a microcontroller, which needs, for example, an antenna, you might have a reference design.  Okay.  So that's something maybe worth looking into.  Yeah.  Have you maybe thought about offering some sort of snippet?  One could just add to this repo, so it's easier to find it?  Okay.  The question was whether one could add some certain snippet of code or something to the  repo to make it easily findable?  No.  I haven't thought about it.  For now, it would just be enough to include PCB or something, some words like that in  your repo description, and then we'll find it.  But yeah, maybe some more specific keywords could be provided for that.  Because I mean, all the developers on GitHub, I mean, there's some on Catlab, for example,  or in other vehicles.  All right, I think the question time is over.  There's my, oh yeah.  Sorry.  You want to?  Yeah, no, go ahead.  One more.  I saw that.  So you said you were using the cache data to actually extract this.  How do you actually make that work?  Because both Digi-Key and Octopart explicitly forbid caching data.  Well, we're not using the Digi-Key cache, right?  So Isler has a component search that they use for their customers.  So I didn't actually know that because we don't really interface directly with Digi-Key.  So, okay, thank you so much.  Thank you.
kicadsearch_0_167.png kicadsearch_168_227.png kicadsearch_228_275.png kicadsearch_276_1067.png kicadsearch_1068_1595.png kicadsearch_1596_3527.png kicadsearch_3528_4775.png kicadsearch_4776_5579.png kicadsearch_5580_7031.png kicadsearch_7032_8063.png kicadsearch_8064_9851.png kicadsearch_9852_11099.png kicadsearch_11100_12239.png kicadsearch_12240_15155.png kicadsearch_15156_16307.png kicadsearch_16308_16431.png
All right. Hi everyone. Thank you for your interest in my talk. I'm Lasse. I'm a student at Avatar Aachen University and I'm going to tell you something about a project that I did during the course of a lab course at that university. So I am mainly a software developer and an electronics hobbyist, which means that I've only tinkered with Arduino compatible microcontrollers and I haven't been really into hardware design all that much. But a little over a year ago, I wanted to create my own PCB, just a simple microcontroller with some sensors and I wanted it to be better operated. So I thought, okay, creating PCBs is not that hard, right? It's just like four steps. You have to choose the components that you want to put on your PCB, use some EDA tool to create schematics, design layout, upload the files to a manufacturer and wait for the mail to arrive. All right. So I started with the first step and I went to choose my components. So as I said, I had no idea on how to do that. So I went to Reddit and I asked what components should I use to control my battery circuit. And I got a couple of suggestions. So I went off to the internet and I searched for different data sheets. I read through them. I understood half of them. And finally, I had decided for a chip that I wanted to use. And then I realized that I had no idea on how to actually implement it on the board. So I went off to GitHub and I tried to find other people's projects that incorporated the same chips. And as it turns out, it was quite difficult because the chip I was looking for wasn't the main component on the PCB. So it wasn't mentioned anywhere in the description or in the readme. And I had to actually search through the files, which isn't that simple, on GitHub. And by the time I had found the first project, it was 1 AM and I had to go to bed because I had university the next day. And then I kind of forgot about the whole project. All right. So this semester, I had the opportunity to work on an electronic component search engine that basically allows you to enter the name of an electronic component that you want to incorporate into your PCB. And it will return repositories on GitHub that have projects that include those components. All right. So let's just have a look at the result. This is what it looks like when you search for a component. On the top, you just enter the component name. And as you can see, we found about 100 repositories, including the component. This is one of them. On the top, we mentioned the repository name and the description. If the project has a license, the license is shown right there, on the top right, you have these stars and forks to give you some kind of an idea whether the project is liked and therefore possibly useful. And on the bottom, we show the relevant files, which exact component was found in the file. And we give you a quick link to the datasheet if we find one. And if you want to, you can directly go to a distributor and buy the part or even manufacture the project's PCB. All right. So let's get into the technical details a little bit. We tried to keep the software modular. We split it into four components. We have a crawler, the crawler GitHub for PCB design files. Then we have a parser that actually parses those files and checks for the components that are included. The component names are then validated. I'll tell you more on that later. And finally, we have a database that can be searched and returns the repositories that are relevant to your part search. All right. So more details and difficulties that we met. For the crawler, our idea was very simple. Just search the whole of GitHub and return all PCB files that we can find and for later indexing. However, we found out that it's not quite that simple to search all of GitHub. There are rate limits on GitHub and there's not only visible rate limits, which is like 30 searches per minute, I think. But there's also an invisible counter that somehow calculates how computationally intensive your search queries are. And if you exceed those, you'll get an abuse note and your shutdown for a while. All right. So because code search or searching inside the files turns out to be very computationally intensive for GitHub, we had to just search for repositories and readme's that mentioned certain words in the description like PCB or key cad or stuff like that. And then we searched both repos whether or not they actually contain relevant files. All right. So the parser. So the parser downloads key cat PCB files and extracts the used components from them. That was the idea, but we had to find out that the component names are actually free text fields and some people just wrote down whatever they deemed useful. So that was why we had to incorporate a validation process to find out whether the components that were entered in the PCB files are actually existing components or if it's just something like 12 to 24 volts DC-DC converter. All right. And for the validation, we also had quite a simple idea. So our idea was to use a component search, for example, one offered by a distributor and see if it does return any results and if it does return results, how many. So we just said, okay, if it returns zero results, it's probably non-existent or just some gibberish in the component text. And if it returned more than 10 results, we assumed it would be something very generic like resistor or something. However, we could not find a component search API that offered rate limits that were high enough for us to find anything. So we looked at DigiKey and they offer about a thousand requests per day for free. We also looked at OctoPart. They offer even less. They offer 500 for educational users after you register with them. And after that, it gets really expensive really quickly. Yeah, but we found a workaround by actually using Eisler's component search and a piggy backing onto a cache that they built inside their API. And we can use that without triggering any rate limits at all. All right. Here is some, well, future work or if you're already hyped, good first issues if you want to participate. I don't think do I have time for those? All right. Yeah. So if you're heating your question time, you have to do it first. Okay. Great. Yeah. So we want to move the, we developed on a private GitLab instance and we want to move that to gitlab.com, get the CI CD, which is already running on the private instance, running again. And also it's a huge monolithic mess, which we want to split up into modular repos. We want to improve the search a little bit, support other formats than Kikat. Finally I already met the maintainer of fritzing and so that might be a good next project to work on. And we also want to support other platforms than GitLab. All right. That's about it. Thank you for your attention. Just to be clear, is the intention of other reference designs using the component? So the idea is just to find example projects that have used the same components so that you can get an idea on how to use the component in your PCB. Okay. And then second part is, is there some kind of way to rate the found designs because every round again, if I look at these kinds of designs, I find something which I can see is probably not going to work in real life. So any kind of rate system? Not yet. There was one future work issue here, improve the ranking. But right now we are ranking using GitHub, Stars and Forks. So that might already give you an idea whether a project is useful or not. Yeah. So another improvement would be actually using reference designs from the manufacturer because they actually provide, I don't think they are available on one. Can you repeat to the microphone? Oh, sorry. There was a suggestion to use reference designs provided by the manufacturers of the parts. Yeah. But I would have to look into how to actually find them. As I said, I'm not a hardware specialist, but I only know from reference designs inside the PDFs. So those are usually only graphics, right? It depends about how complicated the part is. For a microcontroller, which needs, for example, an antenna, you might have a reference design. Okay. So that's something maybe worth looking into. Yeah. Have you maybe thought about offering some sort of snippet? One could just add to this repo, so it's easier to find it? Okay. The question was whether one could add some certain snippet of code or something to the repo to make it easily findable? No. I haven't thought about it. For now, it would just be enough to include PCB or something, some words like that in your repo description, and then we'll find it. But yeah, maybe some more specific keywords could be provided for that. Because I mean, all the developers on GitHub, I mean, there's some on Catlab, for example, or in other vehicles. All right, I think the question time is over. There's my, oh yeah. Sorry. You want to? Yeah, no, go ahead. One more. I saw that. So you said you were using the cache data to actually extract this. How do you actually make that work? Because both Digi-Key and Octopart explicitly forbid caching data. Well, we're not using the Digi-Key cache, right? So Isler has a component search that they use for their customers. So I didn't actually know that because we don't really interface directly with Digi-Key. So, okay, thank you so much. Thank you.
creating PCBs is not that hard, right? It's just the name of an electronic component that you want software modular. We split it into four components. We have a crawler, the crawler GitHub components that are included. The component names later. And finally, we have a database that can details and difficulties that we met. For the extracts the used components from them. That was components that were entered in the PCB files are component search, for example, one offered by a could not find a component search API that repos. We want to improve the search a little component? So the idea is just to find example that you can get an idea on how to use the is, is there some kind of way to rate the found was one future work issue here, improve the give you an idea whether a project is useful or actually using reference designs from the don't think they are available on one. Can you know from reference designs inside the PDFs. So microcontroller, which needs, for example, an description, and then we'll find it. But yeah,