History of Robotics Research and Development of Japan1991Integration, Intelligence, etc.Teaching by Showing : Generating Robot Command Sequences Based on Real Time Visual Recognition of Human Pick and Place Actions

Yasuo KUNIYOSHIGraduate School of Information Science and Technology
Hirochika INOUEProfessor Emeritus at the University of Tokyo
Masayuki INABAGraduate School of Information Science and Technology
“Teaching by Showing” is a straightforward teaching method in which a human instructor shows an assembly task to a robot by simply performing it with his own hand, and the robot automatically generates a program by watching the example task.  For such a method to be realized, visual recognition of human action sequences becomes a crucial issue.  In this paper, an experimental system that demonstrates the teaching method is presented.  The system observes example tasks by stereo video cameras and generates symbolic descriptions of action sequences, which are translated into manipulator command sequences.  Actions are modeled as partial state changes over segmented time intervals.  And the partial state descriptions constitute the environment model.  Segmentation events are defined for actions involved in hand assembly tasks, which are detected in terms of time differences of visual features. Integrating these, an algorithm of action recognition is fully described.  It automatically segments and classifies every action in a robust and efficient manner.  An experiment revealed that the system can recognize each action correctly in real time, and it can carry out the instructed task successfully.
図1 実験概念図
Fig. 1 Concept of Experiment
図2 行為認識モデル
Fig. 2 Action Recognition Model
図4 作業実行の様子
Fig. 4 Work Execution


Correspondence papers

Y. Kuniyoshi, H. Inoue and M. Inaba:Teaching by Showing : Generating Robot Command Sequences Based on Real Time Visual Recognition of Human Pick and Place Actions

Journal of the Robotics Society of Japan, Vol.9, No.3,pp.295-303, 1991. (in Japanese)

Related papers

[1] Y. Kuniyoshi, M. Inaba and H. Inoue: Learning by Watching: Extracting Reusable Task Knowledge from Visual Observation of Human Performance, IEEE Transactions on Robotics and Automation, vol.10, no. 6, pp.799-822, Dec., 1994.

[2] Y. Kuniyoshi and H. Inoue: Qualitative Recognition of Ongoing Human Action Sequences, Proceedings of International Joint Conference on Artificial Intelligence, pp.1600--1609, 1993. (Outstanding Paper Award).

[3] Y. Kuniyoshi: Behavior Matching by Observation for Multi-Robot Cooperation, Proceedings of. 7th International Symposium on Robotics Research (ISRR), (In G. Giralt and G. Hirzinger (eds.), Robotics Research -- The Seventh International Symposium, Springer, ISBN 3-540-76043-1, pp. 343-352), 1995.

Related Article