Statistical Mechanics of Nonlinear On-line Learning for Ensemble Teachers

DOI Web Site Web Site BIBCODE 5 Citations 17 References
  • Utsumi Hideto
    Department of Electronic Engineering, Kobe City College of Technology Department of Electrical and Electronic Engineering, Faculty of Engineering, Kobe University
  • Miyoshi Seiji
    Department of Electronic Engineering, Kobe City College of Technology
  • Okada Masato
    Division of Transdisciplinary Sciences, Graduate School of Frontier Sciences, The University of Tokyo RIKEN Brain Science Institute

Search this article

Abstract

We analyze the generalization performance of a student in a model composed of nonlinear perceptrons: a true teacher, ensemble teachers, and the student. We calculate the generalization error of the student analytically or numerically using statistical mechanics in the framework of on-line learning. We treat two well-known learning rules: Hebbian learning and perceptron learning. As a result, it is proven that the nonlinear model shows qualitatively different behaviors from the linear model. Moreover, it is clarified that Hebbian learning and perceptron learning show qualitatively different behaviors from each other. In Hebbian learning, we can analytically obtain the solutions. In this case, the generalization error monotonically decreases. The steady value of the generalization error is independent of the learning rate. The larger the number of teachers is and the more variety the ensemble teachers have, the smaller the generalization error is. In perceptron learning, we have to numerically obtain the solutions. In this case, the dynamical behaviors of the generalization error are nonmonotonic. The smaller the learning rate is, the larger the number of teachers is; and the more variety the ensemble teachers have, the smaller the minimum value of the generalization error is.

Journal

Citations (5)*help

See more

References(17)*help

See more

Related Projects

See more

Keywords

Details 詳細情報について

Report a problem

Back to top