NSF Project: Information integration and human interaction for indoor and outdoor spaces

NSF Project: Information integration and human interaction for indoor and outdoor spaces (Grant No. IIS-0916219)

Updated June 2011

Department of Spatial Information Science and Engineering

University of Maine

Co-PI: Dr. Nicholas Giudice, director: Virtual Environment and Multimodal Interaction lab

Graduate students:

Liping Yang (Ph.D. student, Fall 2010-present)
Hengshan Li (Ph.D. student, Fall 2010-present)

This material is based upon work supported by the National Science Foundation under Grant No. IIS-0916219.
Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the University of Maine or the National Science Foundation (NSF).

If you have any questions about the project, or wish to contact the PIs, please email Mike at: worboys@spatial.maine.edu or Nick at: giudice@spatial.maine.edu.

Several aspects of the project are described on this website, including a Brief project description, its intellectual merit and theoretical motivation, research tasks and results, broader impacts, collaborations, and related resources.

Brief Project Description

The goal of this research project is to provide a unified framework model that integrates existing models of outdoor (O) and indoor (I) space, and to use this model to develop an interactive platform that supports human navigation in mixed outdoor/indoor (OI) environments. The user should feel the transition between inside and outside to be seamless, in terms of the navigational support provided. The approach consists of integration of outdoors and indoors on several levels: conceptual models (ontologies), formal system design, data models, and human interaction. At the conceptual level, the project draws on existing ontologies as well as examining the “affordances” that the space provides. For example, an outside pedestrian walkway affords the same function as an inside corridor. Formal models of place and connection are also used to precisely specify the design of the navigational support system. Behavioral experiments with human participants using both real and virtual environments assess the validity of our framework for supporting human spatial learning and navigation in integrated outdoor and indoor spaces. These experiments also enable the identification and extraction of the salient features of OI environments for incorporation into the formal framework and interface design.

As an example, imagine emergency response workers equipped with hand-held devices that would allow them to navigate outdoors to the scene of a city disaster and then navigate inside buildings, bypassing dangerous or blocked routes to locate survivors. Unfortunately, the ability to integrate and reason using these differing types of data has lagged behind data acquisition technologies. We believe that providing a unified informatic framework, implemented on a portable, context-aware system, as is done in this project, is the best way to address this vexing OI navigation challenge.

Intellectual merit and Theoretical Motivation

The intellectual merit of this work lies in framework models of outdoor-indoor (OI) space that advance knowledge in the unification of OI-spaces at the levels of ontology, formal design, data models, and human interaction. The framework’s efficacy will be determined by the functional integration of OI models in a series of human subject experiments in virtual and real settings. Both topics are novel areas of research in the context of indoor-outdoor human navigation informatics.

An ontology for OI-space will be developed based on individual ontologies of outdoor and indoor spaces, combined with feedback about critical OI-environment features based on the empirical results from the human navigation studies. From the OI-ontology, a formal design of OI-space will be created using a refinement of Milner’s bigraphs, which simultaneously model spatial containment hierarchies and agent connectivity relationships and will be extended to include more complex topological and geometrical relationships. Because layout topology is most important for spatial learning, our translation from the framework model into a data model for the experiments will be effected using Baumgardt’s winged-edge data structures to capture static structures in OI-environments. Dynamic behavior can be represented by changes in geometric and topological relationships in the structural entities. A series of behavioral experiments will be conducted that (1) propose a minimal set of environmental features used to refine the ontology and (2) demonstrate that our framework supports human spatial learning and seamless navigation of OI-environments.

Research Tasks and Results

The project can be broken into several interrelated topics:

Topic 1: Ontology Development (Worboys)

Topic 2: Ontology integration (Worboys and Giudice)

Topic 3: Formal design (Worboys)

Topic 4: Data Model Integration (Worboys and Giudice)

Topic 5: Phase I testing (Giudice)

Topic 6: Phase II studies (Giudice)

Year 1 work will include graduate student hiring and dissertation identification, as well as work on the ontology development, migration to the data model and some initial work on formal design. Human studies (Phase 1) will also begin in year 1. The initial Phase I emphasis will concentrate on development of behavioral protocols for information presentation and subject running, isolating several indoor and outdoor experimental layouts on campus, making 3D virtual renderings of these layouts for use in the Phase I studies, and creating the requisite experimental software for logging subject trajectories, capturing behavioral responses, and data analysis. Pilot subjects will be run to guide the development of perceptually valid virtual models for use in the Phase I studies and fine tuning the experimental software simulating system implementation on a portable platform for use in the Phase II studies. The results from this testbed work will help inform the ontological and formal design activities. Our advisory board will also meet in year 1 in order to provide guidance and feedback.

Year 2 will continue the ontology activity to integration into an OI-space ontology, as well as migration of the integrated space into the indoor data model. Phase 1 testing will continue with formal behavioral experiments to define the best information content to be displayed and to optimize the interface design. Empirical results will ensure usability and interoperability of information deployed in the experiments with the data represented in the formal and data model levels of the proposal.

Year 3 will conclude the modeling and design phase, and focus on Phase II studies, using the results of Topics 1-4. Special attention will be paid to dissemination, and to technology transfer. Students will begin to prepare their dissertations, based on the results of the project.

Models Developed:

Research questions:
1. What are the similarities and differences between the ontologies, geometries, scales, and dimensions of O-space and I-space?

2. How to construct a unified conceptual model of O-space and I-space? What are the key concepts in a unified conceptual model of O- and I-space?

3. What theories and technologies employed in O-space can be applied for navigation in I-space? What new theories and technologies need to be developed for implementing the seamless navigation within the two spaces?

To build a unified model of O-space and I-space for navigation, investigating these two spaces separately is a good starting point. Exploring the similarities and differences between O-space and I-space from the perspective of navigation can lay the ontological and formal foundations for a unified model of the two spaces (For more information please see here), in which with applications to seamless navigation. Many geometric and topological constructs can be used to represent the structure of both O-space and I-space. For example, networks can often usefully represent both spaces. In O-space, roads are represented by edges and road junctions by nodes, and in I-space, nodes represent rooms which are connected by edges representing corridors. I-spaces generally have more “regular" geometries, for example, the boundary of a room compared with a coastline. In O-space, distances, angles and coordinates play essential roles (e.g., latitude and longitude), so Euclidean concepts are important. In I-space, topology plays a more prominent role, for example, connectivity becomes more important than direction. The scale of O-space is generally larger than I-space. Structurally, O-space is usually represented in two dimensions, whereas I-space uses three dimensions (e.g., levels of buildings). In past decades, GIS is traditionally focused on O-space. We have investigated what theories and technologies applied in O-space can be used or extended for I-space by comparing the two spaces and how to use them in I-space. For navigation, network and topology play essential roles. So in this research we will investigate and extend the existing network and topology theories that have already been applied in O-space in order to fit the practical needs of the navigation applications in I-space.

Ontologies developed

Building several microworlds (proposed by Sowa) is more beneficial than a large comprehensive ontology, because microworlds make it easier to share and reuse knowledge. We have built different levels of microworlds, based on given upper ontologies:

Domain ontologies (Structure) related to the fabric of the I, O, and OI spaces. For example rooms, corridors, walls for indoor space, and roads, parks, bridges for outdoor space.
Task ontologies (Navigation) related to navigation in I,O, and OI spaces.
Application ontologies, applying the navigation tasks in different types of spaces (indoor navigation for pedestrian, outdoor navigation for vehicle, outdoor navigation for pedestrian).

For more information please see here (after Guarino, 1998). We are seeking and constructing the intra-relationships between the concepts within each ontology and inter-relationships between the concepts in different ontologies.

Data models developed

Data models of O-space, I-space, and OI- space were built using UML. UML helps us to keep our data models consistent with the ontologies developed already.

Following the development of the ontologies and data models, we are in the process of building the geospatial database of Boardman Hall, the office building where our department is located. By this means, we can test our ontologies and data models.

Human Studies:

Research questions:

1. How does environmental learning and route finding ability differ as a function of simulation fidelity? Specifically, is a high fidelity simulation model necessary for promoting efficient indoor navigation or is a "sparse" model sufficient? To address this question, we are testing four levels of models: each manipulating the amount of environmental information that is available to the user. These include: high fidelity simulation models (HM), low fidelity simulation models (LM), wireframe models (WM) and sparse models (SM). The four types of models represent a clear progression of decreasing visual granularity, what we call “simulation fidelity”.

2. Which viewing perspective should be used on a real-time display to best support navigation of complex indoor buildings? We are investigating the pros and cons of both First Person and Third Person (bird’s-eye) viewing perspectives.

3. Should the optimal display adopt a heading-up (track up) or a north-up (fixed) model? Heading-up means that the information displayed on the PDAs will synchronize with your orientation during navigation, whereas North-up means that the information on the PDA always remains in a fixed north-up orientation irrespective of your facing direction.

Experiment design:

These types of questions motivate the current line of behavioral experiments. The first experiment manipulates the visual granularity of information displayed on a PDA to assess how environmental learning and route finding differs as a function of simulation fidelity. This study is being done in virtual reality, as it allows us to make multiple building environments of significant complexity, easily vary the information content rendered on the simulated PDA,and simplifies logging of user spatial behavior and collection of test data during experimental sessions. Significant time has been spent in making models of four levels of visual granularity, ranging from a high fidelity simulation, which richly renders the details of the environment, to a sparse rendering, which only shows the floor plan topology. Our goal is to find the minimal information display which supports the highest level of spatial learning and navigation performance. These human behavioral data will be important in subsequent experiments for knowing what information should be rendered in a spatial display supporting indoor navigation and will also feed back to the ontology development and modeling work. For more information please see here (Poster), here (video), and here (Talk).

Research Videos:

Informatics of Indoor and Outdoor Spaces, presented by Liping Yang

Exploring the Minimal Information Set for Human Navigation in Indoor and Outdoor Environments, presented by Hengshan Li

Tools Used:

Ontology tools used:

1. Protege is a free, open-source platform that provides a growing user community with a suite of tools to construct domain models and knowledge-based applications with ontologies. Protégé is being developed at Stanford University in collaboration with the University of Manchester.

2. OpenCyc is the open source version of the Cyc technology, the world's largest and most complete general knowledge base and commonsense reasoning engine. OpenCyc can be used as the basis of a wide variety of intelligent applications such as:rapid development of an ontology in a vertical area; email prioritizing, routing, summarization, and annotating; expert systems; games.

3. WordNet Search is a large lexical database of English, developed under the direction of George A. Miller (Emeritus) at Princeton University. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations. WordNet's structure makes it a useful tool for computational linguistics and natural language processing.

4. Merriam-Webster Online Dictionary

5. Conceptnet aims to give computers access to common-sense knowledge, the kind of information that ordinary people know but usually leave unstated.ConceptNet is an open source project, with a Python implementation and a REST API that anyone can use to add computational common sense to their own project.

Interface tools used:

1. 3DS MAX software providing integrated 3D modeling, rendering, and compositing tools.

2. Vizard is a versatile virtual reality development interface designed for building and rendering virtual worlds.

Publications and Talks:

Grant-related publications, Presentations, keynotes, and invited lectures

Yang, L. & Worboys, M. (2011). Similarities and differences between outdoor and indoor space from the perspective of navigation. Extended abstract (Poster) accepted for the Conference on Spatial Information Theory (COSIT 2011), September, Belfast, ME.

Yang, L. (2011). A unified informatic framework supporting seamless navigation within built indoor and outdoor spaces. Doctoral colloquium talk to be given at the Conference on Spatial Information Theory (COSIT 2011), September, Belfast, ME.

Li, H. & Giudice, N.A. (2011). Finding the minimal information set for assisting navigation in multilevel indoor spaces. Poster presentation at the UMaine Graduate Expo, April 2011, University of Maine.

Li, H. & Giudice, N.A. (2011). Finding the optimal visual interface for assisting navigation. Doctoral colloquium talk to be given at the Conference on Spatial Information Theory (COSIT 2011), September, Belfast, ME.

Giudice, N.A., Walton, L.A., & Worboys, M. (2010).The informatics of indoor and outdoor space: A research agenda. Second ACM SIGSPATIAL International Workshop on Indoor Spatial Awareness (ISA 2010), November, San Jose, CA.

Raja, M.K. (2010). Indoor positioning and indoor location based services. Poster presentation at the UMaine Graduate Expo, April 2010, University of Maine. (N.A. Giudice: advisor).

Worboys, M. The spatio-temporal mosaic: local views to global visions. Invited keynote address, 13th AGILE International Conference on Geographic Information Science, Guimarães, Portugal, 11-14 May 2010. (This presentation included a component that showed the main features that distinguished indoor spaces from outdoor spaces).

Worboys, M. Current research in geospatial science, Invited talk to the Centre for Geospatial Science, University of Nottingham, UK, June 2010. (This presentation included a component that presented the research agenda for indoor-outdoor spaces).

Bringing geographic information indoors, some of the slides from the 2010 UCSB Dangermond Lecture, presented by Mike Worboys.

Related research awards by Worboys and Giudice (active/completed projects)

Worboys, M.; Government of South Korea, Indoor Spatial Awareness, ($270,000), 2008-2011.
Giudice, N.A. (UMaine PI), with Daniilidis, K., UPenn (PI); Roumeliotis, S., UMN; and Manduchi, R., UCSC; NSF grant CDI-0835689, entitled CDI-Type II: Cyber Enhancement of Spatial Cognition for the Visually Impaired, ($276,000 to Giudice), 2008-2012.
Giudice, N.A. (UMaine PI), with Koronis Biomedical Technologies (PI); NIH Phase II SBIR EY017228-02A2, entitled Indoor Personal Navigation System for the Blind using Augmented GPS, ($31,078 to Giudice), 2009-2011.

For more information about our projects on Indoor/Outdoor Spaces, please visit our projects wiki.

Broader Impacts

In addition to this website, research results from this project will be disseminated in peer-reviewed journals and conferences. At least two graduate students will benefit from mentoring from the PIs, as well as interdisciplinary work in fields such as computer science, mathematics, cognitive science, and Experimental Psychology. Students working on the project will have the opportunity to participate in the IGERT summer research symposia at UMaine and the yearly UMaine graduate Expo. Our research topics will also be incorporated into several existing and new graduate courses. A workshop is planned on the use of VR for research, which has applications across many departments on campus. Research topics will also be shared with public school teachers participating in the University of Maine’s NSF-funded RET (Research Experiences for Teachers) program.

Collaborations

This project will benefit from a strong Collaboration with the University of Melbourne, where PI Worboys holds an honorary professorial fellowship. We will also be collaborating with several industry and academic partners who are working on similar issues. These partners will serve on an advisory board to provide feedback on grant-related work. The advisory board consists of: Penobscot Bay Media, Global Relief Technologies, the Spatial Information Science Research Group at the University of Melbourne, the Centre for Geospatial Science at the University of Nottingham, UK, and the National Center for Geographic Information and Analysis, University of California at Santa Barbara. The industrial members of the board will be particularly involved in system specification, specific data models, and technology transfer.

Worboys, M. Meeting with experts at the Centre for Geospatial Science, University of Nottingham, UK, June 2010, on indoor positioning technologies.

More grant related information about O/I spaces projects: http://nav.spatial.maine.edu/oispacewiki/index.php/

oispace