Overfitting is a situation in which a model accurately describes some data but not the phenomenon that generates that data. Overfitting was a huge problem in the good old times, where each data point was expensive, and researchers operated on datasets that could fit a single A4 sheet of paper. Today, with mega- giga- and tera-bytes datasets, overfitting is … still a problem. A very painful one. Following is a short reading list on overfitting.
I would like to start with Mehmet Suzen mllib.wordpress.com who treats overfitting as “inaccurate meme in supervised learning”
cross-validation does not prevent your model to overfit and good out-of-sample performance does not guarantee not-overfitted model.
Another blogger, whose name I couldn’t find, has two very detailed posts on overfitting:
Understanding overfitting from bias-variance trade-off and Understanding overfitting from Haussler 1988 theorem
Finally, Adrian from the “morning paper” (please don’t tell me you don’t follow that blog) has a summary of another paper, titled “Understanding deep learning requires re-thinking generalization” (I only read Adrian’s summary).
Conclusion
No conclusions here. It’s a reading list.
Featured image credit: https://en.wikipedia.org/wiki/Overfitting#/media/File:Overfitting.svg
Boris Gorelik 