Naturalistic Pen-Based Data Interaction

By: John Cook, Dr. Hari Narayanan and Dr. Jeff Overbey

The proliferation of affordable multi-touch devices over the last five years has brought the power of data visualization and interaction to the average
consumer. However, development and integration of pen-based input via new “smart stylus” tools has not resulted in data visualization applications that respond intelligently to markup and that enhance the naturalistic pen-and-paper interaction these tools are designed to replicate. The aim of this research is to create a system that provides natural pen-based data visualization and manipulation techniques and to evaluate it for feasibility and effectiveness against its touch-based “Windows, Icons, Menus, Pointer” (WIMP) counterparts. I have developed a spreadsheet application for the Apple iPad Pro that recognizes the pen gestures users would naturally use on a paper spreadsheet (circling, crossing out, highlighting, etc.; Figure 1) and maps them to their respective operations. The application also provides features typically offered by electronic spreadsheets (sorting, computation, etc.). 

For this project, I designed a user study in which fourteen undergraduate computing and liberal arts students first completed a number of gestures in isloation, then carried out a sequence of data manipulation tasks using both my application and its leading competitor, Microsoft Excel™. The former task provided data regarding the intuitiveness and usability of each naturalistic gesture, while the latter provided the same insights regarding the application and interaction scheme as a whole. An automatic time-stamping function collected timing data for the users’ interactions with each application, and an administered questionnaire provided qualitative feedback.

The results showed that the pen-based interactions yielded significant time savings over traditional WIMP interactions. On average, completing a set of data interaction tasks was two times faster using pen-based interactions (~5.5 versus 2.7 minutes). Unsurprisingly, previous experience with Excel™ was a predictor of
increased task-completion speed using pen-based techniques (27% faster). Interestingly, liberal arts students completed the tasks 17% faster than computing students. While all of the individual pen gestures were more efficient than their WIMP-based counterparts, some gestures stood out as particularly efficient. Basic arithmetic operations require multiple steps to complete in Excel™, but only two gestures using the pen-based system. As such, users completed these operations 88% faster using the pen-based techniques. Similarly, deletion and summation require multiple steps to complete using Excel™, and users completed these tasks 70% and 66% faster (respectively) using the pen-based techniques. The learning curve for the multiple-selection interaction was somewhat high using the WIMP system, and users completed this interaction 62 times faster using the pen-based system when performing the gesture in isolation.

Overall, the feedback from the questionnaire indicated that users found the pen-based interaction techniques more enjoyable, more intuitive (with the exception of a gesture to circle and paste), and easier to use (though users with a large amount of spreadsheet experience were understandably more comfortable using Excel™). This positive feedback, combined with the significant increase in user efficiency, indicates that a spreadsheet interface that implements pen-based gestural interaction techniques is quantitatively and qualitatively superior to a traditional WIMP-based interface. While no spreadsheet or other data manipulation applications that provide a pen-based gestural interface exist today, this study proves that such a system is both feasible and preferred by the end users.

Figure 1. Screenshot from the application displaying an example of a ‘cross out’ gesture to delete (left) and a line gesture to select (right).

Statement of Research Advisor

John Cook recognized a deficiency in spreadsheet and other data manipulation applications in the marketplace in that while end users increasingly work with such applications on devices like tablets and smartphones with no keyboard or mouse, the interfaces of such applications are still based on keyboard and mouse interaction. He proposed a research project to develop and test a pen-based gestural interface against the traditional interface on a spreadsheet, succeeded in getting the project funded through an Auburn University Undergraduate Research Fellowship, and carried the project to a successful conclusion. His work provides a compelling argument for new interface designs for the “smart” devices of the present and future.

Hari Narayanan, Computer Science and Software Engineering.

Last Modified: Jul 24, 2023 @ 12:09 pm