Machine Learning PhD Survival Guide 2021 | Advice on Topic Selection, Papers, Conferences & more!
machine learning phd how to do a phd in machine learning phd advice machine learning phd thesis topics machine learning phd topics how to machine learning phd how to select a thesis topic how to machine learning conferences how to write a machine learning paper advice for phd students advice for new phd students how to survive a phd what to do in a machine learning phd deep learning phd advice machine learning phd thesis machine learning phd thesis topic
#machinelearning #phd #howto

This video is advice for new PhD students in the field of Machine Learning in 2021 and after. The field has shifted dramatically in the last few years and navigating grad school can be very hard, especially when you're as clueless as I was when I started. The video is a personal recount of my mistakes and what I've learned from them. If you already have several published papers and know what to do, this video is not for you. However, if you are not even sure where to start, how to select a topic, or what goes in a paper, you might benefit from this video, because that's exactly how I felt.

Main Takeaways:
- Select niche topics rather than hype topics
- Write papers that can't be rejected
- Don't be discouraged by bad reviews
- Take reviewing & teaching seriously
- Keep up your focus
- Conferences are for networking
- Internships are great opportunities
- Team up with complementary skills
- Don't work too hard

OUTLINE:
0:00 - Intro & Overview
1:25 - Thesis Topic Selection
4:25 - How To Publish Papers
5:35 - Dealing With Reviewers
6:30 - How To Be A Reviewer
7:40 - Take Teaching Seriously
8:30 - Maintain Focus
10:20 - Navigating Conferences
12:40 - Internships
13:40 - Collaborations
14:55 - Don't Forget To Enjoy

Transcript: https://www.notion.so/Yannic-Kilcher-s-PhD-Survival-Guide-Transcript-c507ab8e963e496fbb185cdfdb8d65ae

Credits to Lanz for editing

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 on how to do a PhD. So mainly that you don't repeat my mistakes. Train. So you've made it into a PhD program. Congratulations, you made it. So today we're going to have a look at what to do during a PhD, how to succeed at publishing papers, how to deal with reviews, what to do at conferences and many other things. So I hope you enjoy this little guide of how to survive a machine learning PhD in 2021. So first of all, let me say, I'm not good at this. I'm not an expert. I'm at the end of my PhD and I've done many things wrong and by no means am I a successful academic. However, if you're like myself, and at the beginning of your PhD, you don't really have a clue what to do, you don't know how to select topics, you don't know how to write papers, or even what a paper is really, then there might be something in here that could help you. I'm not super successful myself. But what I can tell you is that I've seen many people who are good at it. So I can tell you what those people did right, what I did wrong, and generally what I think you should do. Alright, that being said, let's dive right in. When it comes down to choosing a topic, make sure you look for something that your advisor or the senior people around you have lots of experience in. They can help you much better like this. You also want to choose something that matches your particular interests, because you're going to be stuck with it for a while. Lastly, you want to choose something that fits your expertise, where you're already reasonably good at or can get good at very quickly. At the intersection of those three things, you're going to find something that is unique to you, and is going to be a very good topic for your PhD. But there are a few more things to consider when selecting a topic. First of all, resources, how much access to resources you have will determine what kind of topics are even accessible to you as a researcher. So I'm going to assume that you do not have a giant compute cluster or heaps of money around. And therefore, my recommendations are going to be for, let's say the rather average PhD student who is not a giant tech company. However, if you do happen to have 1000s of TPUs in your backyard, ignore my advice and just train big language models. Alright, there are two fundamental ways how you can choose a topic. Way one is to choose the biggest most hype topic in the area right now. Now that is not necessarily a bad strategy, but it has some drawbacks. And the reason is that in a hype topic, there are many papers, but there is also a giant amount of competition, not only from other researchers, but from large corporations with lots and lots of resources behind them. And the bigger reason why it's a bad idea is the fact that they wane. If you pick transformers to research today, it's very likely that three, four years down the road, you'll still be stuck with transformers, the field has moved on. And now all of these people that have made the same choice, namely to invest in the biggest topic right now, are trying to finish their PhD, are trying to get papers published in that topic that is no longer of such a big interest at that particular point in time, and therefore already be on the declining side of the hype cycle. So what's the alternative to hype topics? The alternative is niche topics. And that's what I would recommend for most people. The advantages of finding niches is there isn't as much competition around and you can actually become an expert and the best at whatever you do. Some examples of niche topics are things like bandits, optimization, biologically plausible neural network, text based games, I'm not suggesting you go into these topics, but look for smaller communities that nevertheless publish year after year after year. Alright, so now the important stuff, how do you get papers published? Now if I had to summarize the style of writing papers that get published in one sentence is that write papers that cannot be rejected. And that is not as obvious as it sounds. The review process in machine learning is heavily incentivized to reject your paper as quickly and easily as possible. Do not give reviewers any reason to reject your paper. And the easiest way to learn how to write papers is to literally read papers. Go into your niche, gather the papers that are there, read them, try to emulate their writing style, try to emulate the type and way they do and present experiments, try to emulate the way they write up theoretical foundations for their ideas. Your goal is going to be to write a paper where there is no obvious criticism to be had by reviewers. Reviews are the single biggest obstacle to achieving your goals. And let me tell you right now, getting reviews is one of the most cruel experiences you're going to have in your PhD. Reviewers are nasty, they don't have time, they don't read the paper correctly, they misunderstand, they criticize that you didn't evaluate on some obscure data set. And in general, you're going to feel quite misunderstood by reviewers. This happens to all of us. What I can tell you is don't get discouraged by bad reviews. Don't take individual reviews too seriously, and just resubmit the paper to the next conference. So keep your sanity, don't take it personally. There are many famous papers that have been rejected at first try. And not because the paper was bad, but just because the reviewers were crappy. Now there are going to be things during your PhD that you'll have to do that are not writing papers. And one of those things is, especially as you get more senior, you're going to be asked to review yourself. Now it is an easy option to take all that frustration that you have against reviewing, and you see all these other people doing such a crappy job that you just think, whatever, I'm going to do a crappy job myself. And it's tempting. It's very tempting, especially because you gain nothing from doing good reviews. But other than a you, hey, thanks for the review. You'll get nothing. And it is really, really hard to write a good review. Do it. Nevertheless, please, not only are you helping the field by being not one of the crappy reviewers, but writing a good review also helps you really dig into a paper, really see the weaknesses in other papers. And it makes you a better author, researcher, and community member. So for your own sake, and for the community, take the review seriously, even though you don't have time, even though other people do a crappy job. Another thing that you're going to be asked to do very probably is teaching. Now again, you're going to have very little incentive to do a good job at teaching. After all, students are nuisances, the faster you can get it over with, the better the earlier you can go back to writing papers. However, I urge you to take teaching seriously, not only because the world relies on the next generation of researchers being competent, but also think about the fact that the people you teach will be probably some of them working with you in the future. They might be researchers in other labs you collaborate with, they might even be joining your own lab, and you will profit from them being more competent. So take teaching seriously for your benefit and for the benefit of your students. So besides the things you have to do, like reviewing and teaching, what should you work on all day? And here's my answer. Start working on your thing, go pee, and then continue working on your thing. A PhD is first and foremost an exercise in long term focus, you're going to be tempted to do all kinds of things during your PhD, you're going to look and here's a reading group and here's a seminar and here's a lecture. Now unless it is on your specific thing on your specific niche, it's probably going to be not a productive use of your time. I'm not saying you shouldn't go there. What I'm saying is that be aware that what ultimately gets you to get your papers is a long term laser focus on your topic and other topics will creep up on you. It's going to be so interesting because you're stuck here with your thing that you know and that is boring and there's going to be this other cool topic. Wow. Here we are, this is the NURBS 2019 poster session, one of the poster sessions. There are about 250 posters in this room and there are so many people. It is crazy, every single poster has a ball of people around it, presenters trying to explain to the bystanders their work. And you're going to be tempted, oh this is interesting, this is interesting, this is interesting and my topic is so lame. I'm going to just look into this and that's also cool. Yeah, you know who did that? Me. It did not turn out well. Focus, focus, focus, focus your research on your thing and you'll be successful. So now you've written your paper, you've submitted it to peer review and with a little bit of luck you've actually managed to get it published and you get to go to a conference. Now the conference itself and the conference website and everyone on Twitter might give you the impression that conferences are there for people giving talks about their research and you listening and learning. That's crap. Conferences, especially the talking part of conferences, have become more and more irrelevant with the years. Specifically now that everything is recorded and streamed, just look at that stuff from the comfort of your couch at 2x speed. You're missing nothing. These talks are often very short, very rehearsed and most importantly they are about research that is at least six months old. The interesting part about conferences are the people there. The interesting talking happens in workshops, in panels, in tutorials, try to find places where current research is discussed. Workshops are a great place to go for this because the research is often much more recent and not done yet. Go to conferences to interact with people. This whole oh we come together for research, that's a charade. The best researchers I know do nothing else but meet and talk to people all day at conferences. And I don't mean this in a mean way. I don't mean go out and deliberately engineer contact with people for your own benefit. No, a conference is a place where you can find other people that are interested in the same things as you are and you can talk to them, get to know things that you could never get to know through a writing or in a paper. A lot of paper authors will tell you things face to face that they would never write down. A paper such as which experiments that don't work, problems in research, weaknesses of papers. You'll get a lot of knowledge by being there and talking to people. But you have to go out of your way and do it actively. I know this is hard for a lot of us but it pays off and it's going to make your life a lot more enjoyable. All right the next thing I want to talk about is internships. Should you go to an internship at a company at a different university and this depends entirely on your preference. Now I myself have had pretty good experiences with internships and people I know have done so as well. Generally if you do an internship it gives you a bit of a different perspective because you do it at a different place. And if you do an internship with a large company it can be quite a switch of environment. You'll have access to many more resources and you can do maybe a little bit of a different type of research and most importantly you'll meet people that are not academics or not academics anymore. And that is very very valuable. Once you've been stuck in academia for a while meeting someone who just cares to build a cool product is so refreshing and gets you a bit down to earth with what's really important. Lastly I want to talk about the topic of collaborations. Now academia is a bit tricky in that the system tries to alienate and isolate you as a person. You need those first author papers, you need to provide a personal contribution to the knowledge of humankind. Look for people who have the same interests in terms of topic but who have a little bit different skills or experiences such that your papers and your research can become more well rounded. That could be a difference in theoretical versus experimental knowledge, that could be a difference in your academic background. So if you can find someone that has complementary skills to yours and is interested in the same niche it definitely pays off to work together and produce research together. However only do this if they really work in the same field. It is very tempting to start all kinds of collaborations with people all over the place. If you can handle that good for you but again it pays to have a little bit of focus on your particular field and really view collaborations as a joint effort to get research done more quickly and with more rigor. Right so the way I discussed it right now it seems like doing a PhD is gruesome and lots of work and you never get to do anything fun and while there is an aspect to that and it definitely can happen to people especially if they want to finish real quickly. I urge you to also make some time to enjoy this time. A PhD is a cool time. You'll get to meet so many interesting people, get to learn so many interesting topics and ideas and you'll hopefully get to go to many interesting places and that is an invaluable experience. So my advice is if you can take it a bit easier, enjoy your time, take as much out of it as you can and don't work all the time. Maybe you'll have half a year longer, who cares? You only get to do a PhD once and enjoy the time at university while you still can. You can get a job any day. So I hope you've gained at least something from this video and you should be on a path to a successful machine learning PhD. Cheers!
 on how to do a PhD. So mainly that you don't repeat my mistakes.
 Train.
 So you've made it into a PhD program. Congratulations, you made it. So today we're
 going to have a look at what to do during a PhD, how to succeed at publishing papers,
 how to deal with reviews, what to do at conferences and many other things. So I hope
 you enjoy this little guide of how to survive a machine learning PhD in 2021.
 So first of all, let me say, I'm not good at this. I'm not an expert. I'm at the end of my PhD and
 I've done many things wrong and by no means am I a successful academic. However, if you're like
 myself, and at the beginning of your PhD, you don't really have a clue what to do, you don't know how
 to select topics, you don't know how to write papers, or even what a paper is really, then there
 might be something in here that could help you. I'm not super successful myself. But what I can
 tell you is that I've seen many people who are good at it. So I can tell you what those people
 did right, what I did wrong, and generally what I think you should do. Alright, that being said,
 let's dive right in. When it comes down to choosing a topic, make sure you look for something that
 your advisor or the senior people around you have lots of experience in. They can help you much
 better like this. You also want to choose something that matches your particular interests, because
 you're going to be stuck with it for a while. Lastly, you want to choose something that fits
 your expertise, where you're already reasonably good at or can get good at very quickly. At the
 intersection of those three things, you're going to find something that is unique to you, and is
 going to be a very good topic for your PhD. But there are a few more things to consider when
 selecting a topic. First of all, resources, how much access to resources you have will determine
 what kind of topics are even accessible to you as a researcher. So I'm going to assume that you do
 not have a giant compute cluster or heaps of money around. And therefore, my recommendations are going
 to be for, let's say the rather average PhD student who is not a giant tech company. However, if you
 do happen to have 1000s of TPUs in your backyard, ignore my advice and just train big language
 models. Alright, there are two fundamental ways how you can choose a topic. Way one is to choose
 the biggest most hype topic in the area right now. Now that is not necessarily a bad strategy,
 but it has some drawbacks. And the reason is that in a hype topic, there are many papers,
 but there is also a giant amount of competition, not only from other researchers, but from large
 corporations with lots and lots of resources behind them. And the bigger reason why it's a bad
 idea is the fact that they wane. If you pick transformers to research today, it's very likely
 that three, four years down the road, you'll still be stuck with transformers, the field has moved on.
 And now all of these people that have made the same choice, namely to invest in the biggest topic
 right now, are trying to finish their PhD, are trying to get papers published in that topic that
 is no longer of such a big interest at that particular point in time, and therefore already
 be on the declining side of the hype cycle. So what's the alternative to hype topics? The
 alternative is niche topics. And that's what I would recommend for most people. The advantages
 of finding niches is there isn't as much competition around and you can actually become
 an expert and the best at whatever you do. Some examples of niche topics are things like bandits,
 optimization, biologically plausible neural network, text based games, I'm not suggesting
 you go into these topics, but look for smaller communities that nevertheless publish year after
 year after year. Alright, so now the important stuff, how do you get papers published? Now if
 I had to summarize the style of writing papers that get published in one sentence is that
 write papers that cannot be rejected. And that is not as obvious as it sounds. The review process
 in machine learning is heavily incentivized to reject your paper as quickly and easily as possible.
 Do not give reviewers any reason to reject your paper. And the easiest way to learn how to write
 papers is to literally read papers. Go into your niche, gather the papers that are there,
 read them, try to emulate their writing style, try to emulate the type and way they do and present
 experiments, try to emulate the way they write up theoretical foundations for their ideas. Your goal
 is going to be to write a paper where there is no obvious criticism to be had by reviewers. Reviews
 are the single biggest obstacle to achieving your goals. And let me tell you right now,
 getting reviews is one of the most cruel experiences you're going to have in your PhD.
 Reviewers are nasty, they don't have time, they don't read the paper correctly, they misunderstand,
 they criticize that you didn't evaluate on some obscure data set. And in general, you're going to
 feel quite misunderstood by reviewers. This happens to all of us. What I can tell you is
 don't get discouraged by bad reviews. Don't take individual reviews too seriously, and just
 resubmit the paper to the next conference. So keep your sanity, don't take it personally.
 There are many famous papers that have been rejected at first try. And not because the paper
 was bad, but just because the reviewers were crappy. Now there are going to be things during
 your PhD that you'll have to do that are not writing papers. And one of those things is,
 especially as you get more senior, you're going to be asked to review yourself. Now it is an easy
 option to take all that frustration that you have against reviewing, and you see all these other
 people doing such a crappy job that you just think, whatever, I'm going to do a crappy job myself.
 And it's tempting. It's very tempting, especially because you gain nothing from doing good reviews.
 But other than a you, hey, thanks for the review. You'll get nothing. And it is really,
 really hard to write a good review. Do it. Nevertheless, please, not only are you helping
 the field by being not one of the crappy reviewers, but writing a good review also helps you really
 dig into a paper, really see the weaknesses in other papers. And it makes you a better author,
 researcher, and community member. So for your own sake, and for the community,
 take the review seriously, even though you don't have time, even though other people do a crappy
 job. Another thing that you're going to be asked to do very probably is teaching. Now again,
 you're going to have very little incentive to do a good job at teaching. After all, students are
 nuisances, the faster you can get it over with, the better the earlier you can go back to writing
 papers. However, I urge you to take teaching seriously, not only because the world relies
 on the next generation of researchers being competent, but also think about the fact that
 the people you teach will be probably some of them working with you in the future. They might be
 researchers in other labs you collaborate with, they might even be joining your own lab, and you
 will profit from them being more competent. So take teaching seriously for your benefit and for
 the benefit of your students. So besides the things you have to do, like reviewing and teaching,
 what should you work on all day? And here's my answer. Start working on your thing, go pee,
 and then continue working on your thing. A PhD is first and foremost an exercise in long term
 focus, you're going to be tempted to do all kinds of things during your PhD, you're going to look
 and here's a reading group and here's a seminar and here's a lecture. Now unless it is on your
 specific thing on your specific niche, it's probably going to be not a productive use of
 your time. I'm not saying you shouldn't go there. What I'm saying is that be aware that what
 ultimately gets you to get your papers is a long term laser focus on your topic and other topics
 will creep up on you. It's going to be so interesting because you're stuck here with your
 thing that you know and that is boring and there's going to be this other cool topic. Wow.
 Here we are, this is the NURBS 2019 poster session, one of the poster sessions. There are
 about 250 posters in this room and there are so many people. It is crazy, every single poster
 has a ball of people around it, presenters trying to explain to the bystanders their work.
 And you're going to be tempted, oh this is interesting, this is interesting, this is
 interesting and my topic is so lame. I'm going to just look into this and that's also cool.
 Yeah, you know who did that? Me. It did not turn out well. Focus, focus, focus, focus your research
 on your thing and you'll be successful. So now you've written your paper, you've submitted it
 to peer review and with a little bit of luck you've actually managed to get it published
 and you get to go to a conference. Now the conference itself and the conference website
 and everyone on Twitter might give you the impression that conferences are there for
 people giving talks about their research and you listening and learning. That's crap. Conferences,
 especially the talking part of conferences, have become more and more irrelevant with the years.
 Specifically now that everything is recorded and streamed, just look at that stuff from the comfort
 of your couch at 2x speed. You're missing nothing. These talks are often very short, very rehearsed
 and most importantly they are about research that is at least six months old. The interesting part
 about conferences are the people there. The interesting talking happens in workshops, in panels,
 in tutorials, try to find places where current research is discussed. Workshops are a great
 place to go for this because the research is often much more recent and not done yet. Go to
 conferences to interact with people. This whole oh we come together for research, that's a charade.
 The best researchers I know do nothing else but meet and talk to people all day at conferences.
 And I don't mean this in a mean way. I don't mean go out and deliberately engineer contact with people
 for your own benefit. No, a conference is a place where you can find other people that are interested
 in the same things as you are and you can talk to them, get to know things that you could never get
 to know through a writing or in a paper. A lot of paper authors will tell you things face to face
 that they would never write down. A paper such as which experiments that don't work, problems in
 research, weaknesses of papers. You'll get a lot of knowledge by being there and talking to people.
 But you have to go out of your way and do it actively. I know this is hard for a lot of us
 but it pays off and it's going to make your life a lot more enjoyable. All right the next thing I
 want to talk about is internships. Should you go to an internship at a company at a different university
 and this depends entirely on your preference. Now I myself have had pretty good experiences with
 internships and people I know have done so as well. Generally if you do an internship it gives you a
 bit of a different perspective because you do it at a different place. And if you do an internship
 with a large company it can be quite a switch of environment. You'll have access to many more
 resources and you can do maybe a little bit of a different type of research and most importantly
 you'll meet people that are not academics or not academics anymore. And that is very very valuable.
 Once you've been stuck in academia for a while meeting someone who just cares to build a cool
 product is so refreshing and gets you a bit down to earth with what's really important. Lastly I
 want to talk about the topic of collaborations. Now academia is a bit tricky in that the system
 tries to alienate and isolate you as a person. You need those first author papers, you need to provide
 a personal contribution to the knowledge of humankind. Look for people who have the same
 interests in terms of topic but who have a little bit different skills or experiences such that your
 papers and your research can become more well rounded. That could be a difference in theoretical
 versus experimental knowledge, that could be a difference in your academic background. So if
 you can find someone that has complementary skills to yours and is interested in the same niche it
 definitely pays off to work together and produce research together. However only do this if they
 really work in the same field. It is very tempting to start all kinds of collaborations with people
 all over the place. If you can handle that good for you but again it pays to have a little bit
 of focus on your particular field and really view collaborations as a joint effort to get
 research done more quickly and with more rigor. Right so the way I discussed it right now it
 seems like doing a PhD is gruesome and lots of work and you never get to do anything fun and
 while there is an aspect to that and it definitely can happen to people especially if they want to
 finish real quickly. I urge you to also make some time to enjoy this time. A PhD is a cool time.
 You'll get to meet so many interesting people, get to learn so many interesting topics and ideas
 and you'll hopefully get to go to many interesting places and that is an invaluable experience. So my
 advice is if you can take it a bit easier, enjoy your time, take as much out of it as you can and
 don't work all the time. Maybe you'll have half a year longer, who cares? You only get to do a PhD
 once and enjoy the time at university while you still can. You can get a job any day. So I hope
 you've gained at least something from this video and you should be on a path to a successful
 machine learning PhD. Cheers!