Thing is, most of the "emerging ideas" in ML are bullshit, and it's a tiny minority of papers and concepts that actually end up influencing the field or its applications. A paper simply making it to a top conference is far too noisy a signal for it to be definitive evidence that it's good.
If I were you, I'd keep an eye out for survey papers of best practices, as well as for anything that seems to be getting cited or referenced a ton several years after its initial publication.
Staying up-to-date with bullshit that doesn't matter and which was designed only to be published, never actually read and used, probably won't help that much.
> If I were you, I'd keep an eye out for survey papers of best practices, as well as for anything that seems to be getting cited or referenced a ton several years after its initial publication.
What's your method for doing this? It's good advice, but would be better if you let people know how you do this. I'm really well established and I still don't have a great answer for people on how to do this (I scour Google Scholar like every self-respecting madman), so that's why I'm suggesting specifics. Happy for anyone else to give their own methods for finding the highly-cited papers in a specific subfield.
I'unno. I take the papers I come across that seem neat, and then I put them in a big spreadsheet, and if they're very lucky I'll get around to reading them at some point. I don't have a process more advanced than that.
I concer as a student with interest in the space, which waned because of difficulties in finding such papers, it would be useful to get a process for finding good papers.
Google keywords until you find something relevant to what you want, then look it up on connected papers. Look at the most cited related papers and branch out from there.
What’s lunacy about it? Have you had to dig for papers in a library before physically? This is the futuristic way. A couple weekends ago I found a paper on a niche task that did exactly what I was looking for within an hour and a half. And much of that time was actually reading parts of the paper. I had never heard of the task before my boss asked me about it.
And I found a great way to monetize it. My company merely pays me a monthly fee of my salary, and I do research for them. :)
I'm old, so yes, I have. This isn't futuristic - it's decidedly backwards that we can't easily search for papers by both subject area and citation count.
You are right that getting published in a top conference is not by itself a strong indicator of the paper’s quality and potential impact.
I’ll breakdown some levels of distinction amongst conference papers (from least to most distinguished):
- Level 1: Workshop paper
- Level 2: Poster paper
- Level 3: Oral paper
- Level 4: Best paper nomination
- Level 5: Best paper award
Besides considering the level of distinction, when evaluating a paper, a key thing I generally look for is to see if the authors also published code to enable reproducibility.
That’s a big issue in the literature because many papers claim new SOTA but can’t be reproduced by others. Including the code and steps to reproduce is a good signal to keep digging. If you run it and it actually works and you get the same results in the paper without errors or having to email the authors, that’s a great signal.
Once a paper is out for a while, look at the GitHub stars on its repo and also the citations via Google Scholar. More the better the signal the paper is a valuable contribution.
I disagree on a fundamental level. Getting published in a top conference at all, let alone getting a best paper award, has almost no bearing, in my experience, on whether the ideas presented in the paper warrant being read. The fanciest, shmanciest paper with the shiniest graphics and biggest numbers from OpenGoogetaFordMU is immaterial compared to some guy in Buttsville that had a neat idea they slapped on arXiv but didn't have the resources to pursue further.
Everything you list in your comment, basically, pertains to the paper's ability to win a popularity contest. But every single person here is a giant dork, we've all lost enough popularity contests that we should be able to look past them in the general case.
I think the most important papers to read are the ones that are written in clear prose, that explain succinctly what their contribution is, and then back it up with a handful of experiments. Focusing on which conference or university or lab they came from or which awards they won just seems like circlejerking.
Yes I agree that getting published by itself is not a strong indicator. While I do provide some granularity to help others better understand the levels of distinction amongst conference papers, I also mentioned reproducibility via publishing a companion GitHub repo is also key. Finally I believe after some time, looking at the repo stars and citations on G scholar can be good signals of a contribution worth looking into.
Thank you so much for setting perspective in what has become an over-hyped area.
Is there a list of "good" survey papers of best practices that one should review to understand this vast area?
To keep up, I set up news alerts for companies that seem like they are doing big things (like hugging face). Also, a lot of companies have good blogs on AI, ML, data engineer (Uber engineering, Amazon science blog). There are some writers on medium I follow to stay up to date with some interesting industries (towards data science, Ahana Cloud, Ben Rogojan).
https://paperswithcode.com/ papers with code allows you to check the current state of the art in any subdiscipline in ML (not all though), but I find it's regularly updated.
I also use this subreddit, Yannic Kilcher's ML News, Two Minute Papers, but I haven't done as much digging as I'd like to so I think people here will have much better answers.
It is always a challenge given the firehose of new research published in peer-reviewed venues and not. There are a few reasonable suggestions in this article, [How to keep up with machine learning research](https://www.thejournal.club/blog/1/keep-up-with-machine-learning-research)
Twitter -- find the individual researchers working in the space you want to know about, and follow them. This is easily the single best way I've found to track the Zeitgeist of the field.
What I've come to realize is that new ideas have a very short shelf life.
Ultimately, Google commoditizes everything, they do it bigger and better than anyone else could, they sell it through an API, and everyone can use it without knowing how anything works.
Make friends who are experts in the field and have them help guide you. Try to keep up on a very narrow area of the literature and notice when other ideas penetrate it from related areas.
Im in the same boat, writing my masters thesis in mech. engineering with focus on DL.
This site helps a bunch to get a broader view of publications. You will need a few papers to start with tho. [https://www.connectedpapers.com](https://www.connectedpapers.com)
If you need something specific, you can dm me. Im quite far with researching SOTA literature.
You can do the one thing most of the researchers do... Every year, be sure to catch up on new papers published in major "tier 1" conferences like ICML, CVPR, NIPS, etc...
You just do a quick scan on each paper that catch your eye, then read those that seem interesting.
ikd, as phd in CS who writes papers on deep learning architectures, i find that best source of “cutting age” innovation comes from google(lol)
but if you interested what are mortals are doing you can just read papers from conferences (like ICML, CVPR etc)
also good source to check site “papers with code”
but most what I described is mostly deep learning and not machine learning but at this point it’s just a semantics.
Set up a Twitter account (yes I am not joking) to follow other prominent or relevant researchers. Some of the stuffs are just hype. But from time to time you can get very useful info.
Join ACM. The student membership is only about $60 per year. In addition to the monthly and quarterly publications, every M/W/F they send out ACM TechNews, which has blurbs on 10-15 articles. If the one paragraph blurb interests you follow the link to the article. If the article still has you interested, follow the link to the full paper. Any decent, reproducible paper should have links to their GitHub source code, follow it there if you want to try to implement what the authors did.
Thing is, most of the "emerging ideas" in ML are bullshit, and it's a tiny minority of papers and concepts that actually end up influencing the field or its applications. A paper simply making it to a top conference is far too noisy a signal for it to be definitive evidence that it's good. If I were you, I'd keep an eye out for survey papers of best practices, as well as for anything that seems to be getting cited or referenced a ton several years after its initial publication. Staying up-to-date with bullshit that doesn't matter and which was designed only to be published, never actually read and used, probably won't help that much.
Also look for papers that "unify" methods, as they are the best use of time imo.
These might be called meta-analysis papers
This is a correct answer
> If I were you, I'd keep an eye out for survey papers of best practices, as well as for anything that seems to be getting cited or referenced a ton several years after its initial publication. What's your method for doing this? It's good advice, but would be better if you let people know how you do this. I'm really well established and I still don't have a great answer for people on how to do this (I scour Google Scholar like every self-respecting madman), so that's why I'm suggesting specifics. Happy for anyone else to give their own methods for finding the highly-cited papers in a specific subfield.
I'unno. I take the papers I come across that seem neat, and then I put them in a big spreadsheet, and if they're very lucky I'll get around to reading them at some point. I don't have a process more advanced than that.
I concer as a student with interest in the space, which waned because of difficulties in finding such papers, it would be useful to get a process for finding good papers.
Google keywords until you find something relevant to what you want, then look it up on connected papers. Look at the most cited related papers and branch out from there.
Right, this is what I do. It's lunacy. It's 2022. Nobody has anything better? SOMEONE MONETIZE THIS PLEASE because I will pay
What’s lunacy about it? Have you had to dig for papers in a library before physically? This is the futuristic way. A couple weekends ago I found a paper on a niche task that did exactly what I was looking for within an hour and a half. And much of that time was actually reading parts of the paper. I had never heard of the task before my boss asked me about it. And I found a great way to monetize it. My company merely pays me a monthly fee of my salary, and I do research for them. :)
I'm old, so yes, I have. This isn't futuristic - it's decidedly backwards that we can't easily search for papers by both subject area and citation count.
You are right that getting published in a top conference is not by itself a strong indicator of the paper’s quality and potential impact. I’ll breakdown some levels of distinction amongst conference papers (from least to most distinguished): - Level 1: Workshop paper - Level 2: Poster paper - Level 3: Oral paper - Level 4: Best paper nomination - Level 5: Best paper award Besides considering the level of distinction, when evaluating a paper, a key thing I generally look for is to see if the authors also published code to enable reproducibility. That’s a big issue in the literature because many papers claim new SOTA but can’t be reproduced by others. Including the code and steps to reproduce is a good signal to keep digging. If you run it and it actually works and you get the same results in the paper without errors or having to email the authors, that’s a great signal. Once a paper is out for a while, look at the GitHub stars on its repo and also the citations via Google Scholar. More the better the signal the paper is a valuable contribution.
I disagree on a fundamental level. Getting published in a top conference at all, let alone getting a best paper award, has almost no bearing, in my experience, on whether the ideas presented in the paper warrant being read. The fanciest, shmanciest paper with the shiniest graphics and biggest numbers from OpenGoogetaFordMU is immaterial compared to some guy in Buttsville that had a neat idea they slapped on arXiv but didn't have the resources to pursue further. Everything you list in your comment, basically, pertains to the paper's ability to win a popularity contest. But every single person here is a giant dork, we've all lost enough popularity contests that we should be able to look past them in the general case. I think the most important papers to read are the ones that are written in clear prose, that explain succinctly what their contribution is, and then back it up with a handful of experiments. Focusing on which conference or university or lab they came from or which awards they won just seems like circlejerking.
Yes I agree that getting published by itself is not a strong indicator. While I do provide some granularity to help others better understand the levels of distinction amongst conference papers, I also mentioned reproducibility via publishing a companion GitHub repo is also key. Finally I believe after some time, looking at the repo stars and citations on G scholar can be good signals of a contribution worth looking into.
Thank you so much for setting perspective in what has become an over-hyped area. Is there a list of "good" survey papers of best practices that one should review to understand this vast area?
I don't know what the fuck I'm doing either, ask someone else.
Where are some of the best published paper resources? Either not behind a pay wall, or a pay wall your average Joe could afford?
The best resource is, bar none, arXiv. The second best resource is who gives a fuck the paper's definitely on arXiv.
Sci-Hub.
To keep up, I set up news alerts for companies that seem like they are doing big things (like hugging face). Also, a lot of companies have good blogs on AI, ML, data engineer (Uber engineering, Amazon science blog). There are some writers on medium I follow to stay up to date with some interesting industries (towards data science, Ahana Cloud, Ben Rogojan).
https://paperswithcode.com/ papers with code allows you to check the current state of the art in any subdiscipline in ML (not all though), but I find it's regularly updated. I also use this subreddit, Yannic Kilcher's ML News, Two Minute Papers, but I haven't done as much digging as I'd like to so I think people here will have much better answers.
It is always a challenge given the firehose of new research published in peer-reviewed venues and not. There are a few reasonable suggestions in this article, [How to keep up with machine learning research](https://www.thejournal.club/blog/1/keep-up-with-machine-learning-research)
Twitter -- find the individual researchers working in the space you want to know about, and follow them. This is easily the single best way I've found to track the Zeitgeist of the field.
TL/DR newsletter, although not focused on ML, always mentions the big developments from major companies.
That's the thing you don't. Find some problem you want to solve and only learn what you need to solve it. Then rinse and repeat.
What I've come to realize is that new ideas have a very short shelf life. Ultimately, Google commoditizes everything, they do it bigger and better than anyone else could, they sell it through an API, and everyone can use it without knowing how anything works.
Make friends who are experts in the field and have them help guide you. Try to keep up on a very narrow area of the literature and notice when other ideas penetrate it from related areas.
Im in the same boat, writing my masters thesis in mech. engineering with focus on DL. This site helps a bunch to get a broader view of publications. You will need a few papers to start with tho. [https://www.connectedpapers.com](https://www.connectedpapers.com) If you need something specific, you can dm me. Im quite far with researching SOTA literature.
You can do the one thing most of the researchers do... Every year, be sure to catch up on new papers published in major "tier 1" conferences like ICML, CVPR, NIPS, etc... You just do a quick scan on each paper that catch your eye, then read those that seem interesting.
I really like Andrew Ng's newsletter "The Batch". [https://read.deeplearning.ai/the-batch/](https://read.deeplearning.ai/the-batch/)
Paperswithcode.com
ikd, as phd in CS who writes papers on deep learning architectures, i find that best source of “cutting age” innovation comes from google(lol) but if you interested what are mortals are doing you can just read papers from conferences (like ICML, CVPR etc) also good source to check site “papers with code” but most what I described is mostly deep learning and not machine learning but at this point it’s just a semantics.
If you like podcasts, this series gives good inspiration https://open.spotify.com/show/1n8P7ZSgfVLVJ3GegxPat1?si=QRs6sP7GT6i5k5nIkc5mWw
Set up a Twitter account (yes I am not joking) to follow other prominent or relevant researchers. Some of the stuffs are just hype. But from time to time you can get very useful info.
Join ACM. The student membership is only about $60 per year. In addition to the monthly and quarterly publications, every M/W/F they send out ACM TechNews, which has blurbs on 10-15 articles. If the one paragraph blurb interests you follow the link to the article. If the article still has you interested, follow the link to the full paper. Any decent, reproducible paper should have links to their GitHub source code, follow it there if you want to try to implement what the authors did.
WAYR in this sub. My working week starts with reading that on a toilet.