SpeechPen

Speech Pen: Predictive Handwriting based on Ambient Multimodal Recognition

Kazutaka Kurihara
Masataka Goto
Jun Ogata
Takeo Igarashi

Links

It is tedious to handwrite long passages of text by hand. To make this process more efficient, we propose predictive handwriting that provides input predictions when the user writes by hand. A predictive handwriting system presents possible next words as a list and allows the user to select one to skip manual writing. Since it is not clear if people are willing to use prediction, we first run a user study to compare handwriting and selecting from the list. The result shows that, in Japanese, people prefer to select, especially when the expected performance gain from using selection is large. Based on these observations, we designed a multimodal input system, called speech-pen, that assists digital writing during lectures or presentations with background speech and handwriting recognition. The system recognizes speech and handwriting in the background and provides the instructor with predictions for further writing. The speech-pen system also allows the sharing of context information for predictions among the instructor and the audience; the result of the instructor’s speech recognition is sent to the audience to support their own note-taking. Our preliminary study shows the effectiveness of this system and the implications for further improvements.

Publications

Kazutaka Kurihara, "A Study on Software Tools for Flexible Presentations," Presented at The ACM Symposium on User Interface Software and Technology Doctoral Symposium (UIST 2006). PDF
Kazutaka Kurihara, Masataka Goto, Jun Ogata and Takeo Igarashi, "Speech Pen: Predictive Handwriting based on Ambient Multimodal Recognition," Proc. of ACM SIGCHI Conference on Human Factors in Computing Systems(CHI'06), pp.851-860, 2006. PDF