The purpose of this study is to investigate to what extent learning English at an early stage can influence junior high school, high school, and university students' motivation and attitudes towards learning English. The questionnaire was given to 1610 Japanese students, and the data were analyzed using factor analysis. Subsequently, the average factor scores between students who had learned English in early childhood and those who had not were compared. The results showed that language instruction at an early stage has some influence on junior high school students' motivation towards learning English. Especially, there was a remarkable impact on instrumental motivation. However, the motivation of high school and university student was not influenced very much. The results indicate that it is doubtful that English instruction in public elementary schools can be as influential on affective aspects of Japanese students as the Ministry of Education, Culture, Sports, and Technology expects.