Finding the Correspondence of Audio-Visual Events by Object Manipulation

DOI Web Site Web Site 2 Citations 24 References
  • Nishibori Kento
    Graduate School of Information Science, Nagoya University
  • Takeuchi Yoshinori
    Information and Communications Planning Office Information and Commuications Headquarters, Nagoya University Bio-Mimetic Control Research Center, RIKEN
  • Matsumoto Tetsuya
    Graduate School of Information Science, Nagoya University
  • Kudo Hiroaki
    Graduate School of Information Science, Nagoya University
  • Ohnishi Noboru
    Graduate School of Information Science, Nagoya University

Bibliographic Information

Other Title
  • 物体操作による視聴覚事象の対応付け
  • ブッタイ ソウサ ニ ヨル シチョウカク ジショウ ノ タイオウ ズケ

Search this article

Abstract

A human being understands the objects in the environment by integrating information obtained by the senses of sight, hearing and touch. In this integration, active manipulation of objects plays an important role. We propose a method for finding the correspondence of audio-visual events by manipulating an object. The method uses the general grouping rules in Gestalt psychology, i.e. “simultaneity” and “similarity” among motion command, sound onsets and motion of the object in images. In experiments, we used a microphone, a camera, and a robot which has a hand manipulator. The robot grasps an object like a bell and shakes it or grasps an object like a stick and beat a drum in a periodic, or non-periodic motion. Then the object emits periodical/non-periodical events. To create more realistic scenario, we put other event source (a metronome) in the environment. As a result, we had a success rate of 73.8 percent in finding the correspondence between audio-visual events (afferent signal) which are relating to robot motion (efferent signal).

Journal

Citations (2)*help

See more

References(24)*help

See more

Keywords

Details 詳細情報について

Report a problem

Back to top