Statistical physics of learning from examples - a brief introduction

Abstract

The problem of how one can learn from examples is illustrated on the case of a student perceptron trained by the Hebb rule on examples generated by a teacher perceptron. Two basic quantities are calculated : the training error and the generalization error. The obtained results are found to be typical. Other training rules are discussed. For the case of an Ising student with an Ising teacher, the existence of a first order phase transition is shown. Special effects such as dilution, queries, rejection, etc. are discussed and some results for multilayer networks are reviewed. In particular, the properties of a selfsimilar committee machine are derived. Finally, we discuss the statistics of generalization, with a review of the Hoeffding inequality, the Dvoretzky Kiefer Wolfowitz theorem and the Vapnik Chervonenkis theorem.