Objectives.
The project shall act as a melting pot for mixing novel technology, that
allows real-time visualisation for a moving observer of recorded REAL PLACES,
with ideas of researchers from diverse fields to develop new tools for empirical
and theoretical studies of presence based on the concept of the observer’s
embodiment in the computationally created virtual environment. As real places
(possibly known to the observer) with man-made and/or organic objects (like
trees, foliage etc.) are otherwise hard to represent in a virtual environment,
the aim is to bring about new insight into presence, when also comparison
to real presence is possible.
Approach
Tools for Presence.
The project will develop and explore new recording and visualisation
technologies enabling people to experience being present at REAL and possibly
known places - without actually having to GO to those places. The experience
will be based on true-to-life visual and auditorial sensory information presented
to the observer in real-time. As opposed to passively watching a film of
such places, the project's technology will be designed to support the observer’s
active exploration of the visual and auditory space, thus adding an intuitive
physical dimension to the experience. This physical dimension is paramount
to achieving "embodiment", i.e. the observer’s sensation of being bodily
grounded in an environment.
The
new technology of image based rendering (IBR) does not require a reconstructed
geometrical model of the scene, and hence it bypasses an important technological
problem and presents a break-through for the state-of-the-art. However, a
rather large amount of image data needs to be stored, and the fast real-time
visualisation is not perfectly faithful to the perspective transformation
governing human visual perception. Also, the scene is assumed static, and
the (central) exploration region, within which an arbitrary viewpoint can
be chosen for real-time visualisation, is constrained by the particular databank
of images captured from the real scene.
Through
augmentation, visually and auditorily, a sense of life can be added to the
static scene, objects for interaction can be included, as well as avatar representations
of companions. Projection technologies to be used will range from HMD to
large screens incl. 6-sided CAVE.
Presence Research
. The special point of the
project’s empirical research will be to exploit the possibilities to investigate
the experience of “being there” when we refer to REAL places and possibly
a place which the observer knows from previous real presence. The theoretical
framework will be based on the concept of embodiment in conjunction with
presence and investigate how it arises from e.g. fidelity of experience and
presentation, domain specific elements, the sense of place, and physiological
and neurological aspects like consistency of sensory-motor co-ordination.
The theory will be extended to consider the feeling of presence of others
and hence social contexts. The framework will be developed in close interaction
with, and as a guide for, technical development by focusing on the particular
aspects that the technology offers as well as on its weak points. Feed-back
from empirical studies will form an essential part of the project.
Work Schedule
The Workplan Schedule
will be organised around a sequence of implementations and tests of application
scenarios of increasing technical and perceptual complexity. Each of them
is characterised by specific functional issues representing challenges to
the core technological components of the project, and how these issues impact
on presence:
1. Single
observer in a panoramic visualisation of a real scene, - like a tourist site,
a forest scene, or grandfather’s living room. Various conditions for image
capture reflecting different regions for allowed observer movements. Importance
of ego-motion and of fidelity will be investigated using a range of visualisation
systems.
2. Augmenting
1) by adding modelled objects centrally as well as sound and acoustics.
(2a? If the scene was Acropolis then augment as it appeared in 400 B.C.,
500 A.D. etc.)
3. Single
observer looking around while following a restricted path along staircases
and the like, in a garden or in a house. Investigate required "observer mobility"
along the path, to maintain presence, and the need for interaction facilities
in various visualisation systems etc.
4. Augmenting
3) by adding modelled objects along path as well as sound and acoustics.
5. Two
distant observers in a meeting scenario, at grandfather’s living room or
at a picnic in the forest. Investigate need and detail of avatar representations
to obtain a feeling of presence of others etc.
6. Augmenting
5) by adding modelled objects centrally as well as sound and acoustics.
7. more
users, - etc.
Workplan Issues
Presence Tools (HUJI, CTUP, AAU, DIKU)
Scene
registration for model-free real-time visualisation, - optimised for a min.
number of images, while extended to a larger (central) exploration region.
Loosen constraints regarding static world and perspective correctness.
Selected
3D-reconstructions (or ordinal depth) off-line and possibly for salient points
in real-time, to support graceful augmentation. For historical scenes extensive
off-line reconstruction is required for more comprehensive augmentation.
Visual
augmentation with near-field objects (in exploration region) to fill-in and
to allow interaction. Alternative solutions to avatar representation. Challenge
to achieve visually seemless integration of rendered 3D models into IBR "background"
(geometrical and spectral issues)
Auditorial
augmentation: i) “3D sound backdrop” as part of the visualised scene, ii)
acoustic response related to the scenario if observer speaks out, and iii)
acoustics for multi user scenarios. Use of loudspeakers or headphones as appropriate
and relevant for targeted level of presence.
Presence Virtual Environment Platform (AAU)
A VE-software
platform will be selected and extended to facilitate integration and efficient
execution of the various Presence Tools. The platform will support stereo
visualisation in a variety of display systems, and provide for relevant user
interaction and registration facilities. A protocol for model sharing and
update for multiple VE-systems will be chosen and/or developed and implemented.
Presence Research (NUSC, UBBR, DIKU)
A theoretical
framework for the concept of presence will be developed to cover the particular
aspects of presence that the project’s technology will make available for
empirical studies. It will incorporate the state-of the-art of cognitive neuroscience
and phenomenology and be based on the concept of embodiment and hence the
factors that are considered to contribute thereto.
Empirical
research will be planned as a close integration of technical development
and related investigations, the result of which will feed back to development.
The above framework will guide priorities among the possibilities offered
from technology, and the proposal will include a projection of a series of
development-test cycles that iteratively will converge to a contribution
to a theory of presence. The plan will be re-evaluated after each cycle.
Benefit and expected results
The particular
benefit is the possibility of virtual environments presenting real
places and places known to the
observers. This is technologically hard or impossible with a reasonable “fidelity”
using current techniques, and the approach of the project opens and explores
new additional avenues of Presence Research, - technologically as well as
empirically - regarding the experience of “being there”. Hence we expect results
that contribute new insight into important aspects of presence, that otherwise
would not be available for investigation at this point in time.
Consortium, 6 partners (Est. Budget: 2.3 M Euro (30 Person Years))
AAU:
Aalborg University (DK), Computer Vision and Media Technology Laboratory.
Prof. Erik Granum. Virtual Environments, Visual Augmentation, Collaborative
VE’s. Co-ordinator.
HUJI:
Hebrew University of Jerusalem (IL), School of Computer Science and Engineering.
Prof. Shmuel Peleg. Camera Technology, Scene Registration for Real-Time Visualisation.
NUSC:
Napier University, Edinburgh (UK), School of Computing. Prof. David Benyon.
Human-Computer Systems, Presence Research, Phenomenology.
UBBR:
University of Bremen, Institute of Brain Research (D). Prof. Manfred Fahle.
Brain Research, Human Vision.
CTUP:
Czech Technical University, Prague, Centre for Machine Perception (CZ). Ass.
Prof. Tomas Pajdla. Camera technology, 3D scene reconstruction.
DIKU:
University of Copenhagen, Inst. of Computer science (DK). Asc. Prof. Jens
Arnspang. Multi Modal Multi Media, Psychophysics of Vision.