Interactive input systems that allow users to inject input (i.e., digital ink, mouse events, etc.) into an application program using an active pointer (e.g., a pointer that emits light, sound or other signal), a passive pointer (e.g., a finger, cylinder or other suitable object) or other suitable input device such as for example, a mouse or trackball, are known. These interactive input systems include but are not limited to: touch systems comprising touch panels employing analog resistive or machine vision technology to register pointer input such as those disclosed in U.S. Pat. Nos. 5,448,263; 6,141,000; 6,337,681; 6,747,636; 6,803,906; 7,232,986; 7,236,162; and 7,274,356 assigned to SMART Technologies ULC of Calgary, Alberta, Canada, assignee of the subject application, the contents of which are incorporated by reference; touch systems comprising touch panels employing electromagnetic, capacitive, acoustic or other technologies to register pointer input; tablet personal computers (PCs); laptop PCs; personal digital assistants (PDAs); and other similar devices.
Multi-touch interactive input systems that receive and process input from multiple pointers using machine vision are also known. One such type of multi-touch interactive input system exploits the well-known optical phenomenon of frustrated total internal reflection (FTIR). According to the general principles of FTIR, the total internal reflection (TIR) of light traveling through an optical waveguide is frustrated when an object such as a pointer touches the waveguide surface, due to a change in the index of refraction of the waveguide, causing some light to escape from the touch point. In a multi-touch interactive input system, the machine vision system captures images including the point(s) of escaped light, and processes the images to identify the position of the pointers on the waveguide surface based on the point(s) of escaped light for use as input to application programs. One example of an FTIR multi-touch interactive input system is disclosed in United States Patent Application Publication No. 2008/0029691 to Han.
In multiple or single-touch interactive input systems, graphic objects, such as the background or “canvas”, and “widgets” overlying the canvas including windows, icons, menus, pictures, text, lines, curves and shapes, are displayed on the display surface. Depending upon the application, there may be a number of graphic widgets displayed at different positions on the canvas, one or more of which may overlap with another.
In prior art interactive input systems, manipulating a graphic widget generally comprises two steps. First, a user selects a graphic widget by contacting the touch surface with a pointer at a location exactly corresponding to the location at which the graphic widget is displayed. With the widget having been selected, the user then manipulates the selected graphic widget using the pointer, for example, by moving the pointer across the display surface thereby moving the selected graphic widget. One drawback with systems requiring such touch precision on the part of the user is that the user may find it difficult to select a small widget. This may occur if the pointer occludes the small widget, if the viewing angle is extreme, or when calibration of the system renders the touch point offset somewhat from the display. Furthermore, interactive input systems of this nature do not typically employ useful feedback subsystems employing, for example, haptics.
This so-called “target acquisition” problem has previously been studied. Proposed solutions to the target acquisition problem generally fall into one of two categories of input techniques: (1) those that improve target acquisition by optimizing Fitts Law parameters; and (2) those that improve target acquisition by leveraging crossing targets.
Fitts Law is commonly used to model target acquisition, as shown by MacKenzie in the 1989 publication entitled “A note on the information theoretic basis for Fitts' Law”; Journal of Motor Behavior, 21:323-330, the content of which is incorporated entirely herein.
The Shannon formulation of Fitts Law, as shown by MacKenzie in “Movement time prediction in human-computer interfaces” in Readings in Human-Computer Interaction; Kaufmann; second edition; R. M. Baecker, W. A. S. Buxton, J. Grudin, and S. Greenberg, editors, the content of which is incorporated entirely herein, states that the movement time (MT) that it takes to acquire a target of width W and distance (or amplitude) D is predicted according to Equation 1, below:MT=a+b log2(D/W+1)  (1)where:
a and b are empirically determined constants; and
the logarithmic term is the index of difficulty (ID).
Equation 1 predicts that smaller target widths and larger distances (from the current location) will increase selection time. Accordingly, target selection can be improved by decreasing target distance D, by increasing target width W, or by modifying both parameters accordingly.
Baudisch, et al., in the publication entitled “Drag-and-Pop and drag-and-pick: Techniques for accessing remote screen control on touch and pen operated systems”; Proc. Interact, 57-64, the content of which is incorporated herein in its entirety, propose reducing target distance by bringing distant targets closer to the user. This Drag-and-Pop method analyzes the directional movements of the cursor, and then brings virtual proxies of the potential targets towards the cursor (e.g., a folder or application). Studies of Drag-and-Pop showed selection to be faster for large target distances. However, the method is unable to determine whether the user intends to select a distant target versus one nearby. Thus the presence of distant objects can make selection difficult for a nearby target.
Bezerianos, et al., in the publication entitled “The Vacuum: Facilitating the manipulation of distant objects”; Proc. CHI 2005, ACM Press, 361-370, the content of which is incorporated entirely herein, propose a Vacuum method that is similar to Baudisch, et al. Drag-and-Pop method, but in addition allows the user to control the approach angle of distant targets in which they are interested. Multiple object selection is also supported. Selection time was found to be similar for single targets but significantly faster for multiple target selection.
Directly increasing the target width W by advocating a very large target area, e.g., a large button, decreases the index of difficulty. However, this requires a significant amount of screen real estate and limits the amount of content that can be placed on a smaller display.
Kabbash, et al., in the publication entitled “The ‘Prince’ technique: Fitts' law and selection using area cursors”; Proc. ACM CHI '95, 273-279, the content of which is incorporated entirely herein, propose increasing the target width, W, effectively by increasing the cursor size. Instead of having a single pixel hotspot as seen in standard cursors, area cursors have a larger active region for selection. By setting target width, W, to be the width of the area cursor, it was shown that selection of a single pixel target could be accurately modeled using Fitts Law. Thus, very small targets would be easier to acquire. However, area cursors are problematic in dense target spaces where multiple targets could be contained in a single area cursor.
McGuffin, et al., in the publication entitled “Fitts' law and expanding targets: Experimental studies and designs for user interfaces”; ACM TOCHI, 12(4), ACM Press, 388-422, the content of which is incorporated entirely herein, propose increasing the target size dynamically as the cursor approaches. It was found that users were able to benefit from the larger target width even when expansion occurred after 90% of the distance to the target was traveled. It was also shown that overall performance could be measured with Fitts Law by setting the target width to the size of the expanding target.
Different approaches that modify target width W and distance D dynamically adjust the control-display gain (C:D). By increasing the gain (cursor speed) when approaching a target and decreasing the gain while inside a target the motor space distance and target width are decreased and increased, respectively. Blanch, et al., in the publication entitled “Semantic pointing: improving target acquisition with control-display ratio adaptation”; Proc. ACM CHI '04, 519-525, the content of which is incorporated entirely herein, showed that performance could be modeled using Fitts Law, based on the resulting larger target W and smaller distance D in motor space. However, problems could arise when there are multiple targets, as each would slow down the cursor as it approached.
Grossman, et al., in the publication entitled “The Bubble Cursor: Enhancing target acquisition by dynamic resizing of the cursor's activation area”; Proc. CHI '05, 281-290, the content of which is incorporated entirely herein, disclosed the development of the Bubble Cursor to ease target acquisition in a sparse display. The Bubble Cursor is surrounded by a dynamically resizing bubble so that only the closest target is enveloped by the bubble. The bubble around the cursor expands until it just touches the nearest target. Although this effectively increases target width (since the bubble gets bigger), and decreases target distance (because less distance needs to be traveled to reach the target), if other targets, or distracters are nearby and within close proximity to the chosen target the size of the bubble is limited and can be much smaller. In other words, the width of the target is dependent on the distance of the closest distracters adjacent to it, as it expands so that only the closest target is selected at any time. This new target size is called the Effective Width (EW). Their study shows that Bubble Cursor's performance can be modeled using Fitts Law by setting W=EW.
U.S. Pat. No. 5,347,295 to Agulnick, et al., the content of which is incorporated entirely herein, discloses a method that, when a stylus moves into the proximity of graphic widgets, display events are triggered to provide the user a preview of what graphic widgets are targeted. For example, the appearance of a button may be expanded or altered in anticipation of its selection.
As set out above, another proposed solution category to the target acquisition problem involves leveraging crossing targets. One such technique is embodied in a crossing based drawing application called “Cross Y” for simplifying pointing tasks on a tablet computer, developed by Apitz, et al., and described in the publication entitled “CrossY: a crossing-based drawing application”, Proceedings of the 17th Annual ACM Symposium on User interface Software and Technology (Santa Fe, N. Mex., USA, Oct. 24-27, 2004); UIST '04; ACM, New York, N.Y., 3-12; http://doi.acm.org/10.1145/1029632.1029635, the content of which is incorporated entirely herein.
The CrossY application enables a user to cross the target area to make a selection from a menu or a list. FIG. 1 is an exemplary diagram shown by Apitz, et al. illustrating some examples of using the CrossY technique. In each example, the dot 4 represents the position where the stylus touches the touch screen, and the arrow 2 represents the direction the stylus then moves. For example, in the example 8 of selecting the radio item “Black” from a list, the user touches the stylus over the radio item “Black”, and then moves the stylus to cross the radio item 6.
While the CrossY technique is effective for object selection such as for example, clicking a button, and selecting a menu option, separate operations to move, rotate, or otherwise manipulate graphic widgets are required.
As will be appreciated, although the above-described techniques improve the user experience of selecting and manipulating graphic widgets, the possibilities of user interaction with interactive input systems have not been fully exploited. It is therefore an object to provide a novel method for selecting and manipulating a graphic object in an interactive input system, and a novel interactive input system executing the method.
USPTO Backgrounds
USPTO Backgrounds