Sketching as a Method of Knowledge Acquisition and Refinement of Spatial Concepts
Sketches are a ubiquitous method of human communication. Sketches, in context, convey ideas. Sketches provide a shared interaction medium where participants achieve a mutual understanding by creating, modifying, and agreeing upon the meaning of sketches. We believe that sketching can be an equally powerful tool for human-computer interaction in knowledge acquisition and joint problem solving. Domain experts could use sketches as a complementary modality for defining and refining concepts with spatial aspects, such as the layers of the atmosphere. An advisory program could use sketched overlays on maps to communicate its assessment of a situation, which could in turn be annotated by all participants, human and software, in deciding what to do. We propose to develop techniques for using sketching as an interaction modality for knowledge acquisition.
Our approach is to combine our Metric Diagram/Place Vocabulary (MD/PV) model of spatial reasoning with methods drawn from computer vision, notably Visual Routines, to support a rich two-way flow between a sketch and the software's understanding of what is depicted. Visual routines provide the geometric processing required to convert low-level information about marks into mixed symbolic/numeric assertions (i.e., a metric diagram). These assertions are interpreted in turn by programs that use them plus the domain semantics to both perform some operations directly and to construct higher-level spatial entities that make explicit important properties of the domain (i.e., a place vocabulary).
We have some promising preliminary results in this area. Our Juxta model focuses on a particular kind of diagram, juxtaposition diagrams, that is important in science instructional materials. Juxta to date has only tested with a handful of examples, but the results make us very optimistic. Specifically, we have demonstrated that the kind of two-way flow outlined above between the perception of simple, schematic diagrams and domain semantics is feasible. Juxta processes diagrams drawn using a graphical editor, interprets them using a combination of geometric and conceptual information, and can depict its understanding graphically.
A powerful feature of our approach is a model of symmetry, MAGI, that handles conceptual and mathematical descriptions as well as perceptual representations. We treat symmetry and grouping operations as a variation of the kinds of comparisons involved in similarity. The psychological plausibility of symmetry algorithms in mixed-initiative interactions with people is especially important, because if our programs see the world too differently from people, communication will break down. The evidence to date is running in favor of MAGI.
Our goal is to turn these preliminary results into a usable technology. This requires (1) creating models of mixed-modality, mixed-initiative interactions for sketching in knowledge acquisition tasks., and (2) incorporating sketches as an everyday medium available in knowledge representation systems. We outline each in turn.
Turning sketches into an everyday medium for knowledge representation: We conjecture that capturing the full range of human flexibility in common sense reasoning in software will require the incorporation of case libraries that include interpreted sketches. This conjecture is based on two observations about human cognition. First, one purpose of sketches for people appears to be as a semi-formal representation; the geometric and visual structure of a sketch may be remembered, even though the intended explanation of it in terms of domain semantics is not. This remembered sketch may be subsequently reinterpreted as the learner's understanding grows. We believe such processes of revisiting previously understood examples and testing new ideas against them is one of the key mechanisms people use to achieve robust conceptual change. Second, the visual structure of sketches themselves can be an important memory cue. Given the structured descriptions of sketches that our software produces, our MAC/FAC retrieval model will be sensitive to overlapping visual structure as well as purely conceptual overlap. This overlapping visual structure could provide valuable suggestions about, for example, how to depict the results of an advisory program's reasoning in spatial terms. We plan to integrate the sketch understanding software into our domain theory development environment, including facilities for storing sketches as a media in case and example libraries, and interfaces that support the rest of the operations described here.
Models of mixed-modality, mixed-initiative dialogs to support knowledge acquisition: Finding natural ways for domain experts to interact with software via sketches is a hard problem. We will focus on specific knowledge acquisition tasks that are both essential to the HPKB effort and narrow enough that we can make progress. The two tasks are (1) interactive creation of spatial recognizers and (2) interactive explanation and validation of domain concepts.
Spatial recognizers create propositional representations from diagrammatic input. Giving domain experts the ability to define new higher-order spatial concepts and signs will make sketching a more habitable medium. Suppose a domain expert draws a new glyph in interpreting a map. Using the spatial characteristics of the glyph (such as its length and relative position to other glyphs), an initial propositional representation can be created from the digital ink. Successive sketches of the same glyph lead to additional representations, which can be compared to extract the commonalties and refine the definition of the new glyph. Ambiguities can be refined either by waiting for more examples or by deliberate variations in the concept when the system is drawing it as part of an interaction. As the definition is refined, a new recognition procedure is added to the encoding process so that the new glyph is recognized as quickly as anything else in the system.
The space of possible descriptions for spatial concepts is enormous, particularly with higher-level spatial concepts. We believe that mixed-initiative, multi-modal dialogs could provide a critical source of constraint in acquiring new concepts with spatial aspects. The ability to extract a symbolic spatial representation provides the possibility of generating and understanding natural language descriptions that can be used to cross-check the system's hypotheses. For instance, progress has been made on describing spatial terms like "near" and "above" in computational terms. The ability of the system to articulate concepts both diagrammatically and propositionally should greatly facilitate communication with the domain expert.
The interactive explanation and validation of domain concepts involves similar refinement processes to that of defining new recognizers. In explaining the hydrological cycle, for example, a domain expert might sketch the path of clouds over typical terrain. As part of checking its understanding, the system should try to sketch the path of clouds over similar terrain and check its results with the domain expert. The domain expert points out what is wrong, editing both the diagrammatic marks created by the system and the conceptual annotations of them. These changes are used as evidence by the system to modify its concepts, and try again. The process continues, moving to terrain that is less similar to the initial examples, in order to provide solid bounds on the applicability of the new concepts. The dialog continues until the domain expert is satisfied.
To make dialogs like these possible requires the combination of our sketch processing techniques, our analogical processing techniques, and a set of interaction strategies that govern how the system responds to its human partner's actions. We believe that these strategies will be similar to the strategies used in natural language discourse processing. We will develop an analogous set of strategies for using sketches in knowledge acquisition tasks, with the goal of making interaction with software via sketches as natural as interacting with a person.