Augmented Reality as Public Art

A proposal to conduct a research initiative at UCSD’s Center for Research and Computing in the Arts (CRCA)

Contact name & information:

John Craig Freeman
Associate Professor
Emerson College
Department of Visual and Media Arts

120 Boylston Street
Boston, MA 02116-4624
(617) 824-8862
john_craig_freeman at emerson dot edu

1 page describing the project:
For over two decades, I have used emergent technology to explore new forms of interventionist public art. As a founding member of ManifestAR, an international artists’ collective, I have produced site specific public art worldwide using augmented reality technology.

I intend to spend the months of January and February, 2012, in California, including this proposed residency at CRCA between January 27 and February 21, in order to conduct the exploratory phase of a research initiative titled “Augmented Reality as Public Art,” which will seek to answer the following questions.

  • What possibilities exist for the use of augmented reality as a public art form?
  • Can augmented reality be used to invent new forms of monuments and memorials?

What follows is a partial list of 4 projects which I am currently pursuing that relate directly to this research. Much of this work requires interdisciplinary collaboration.

During my time in California, I hope to identify and meet with potential collaborators at UCSD and across the UC system. In particular, I am looking for collaborators in neuroscience and computer science. I will be assessing the associated cost of developing these collaborations as I transition into the grant writing phase of the research upon my return to Boston in March.

EEG AR: Things We Have Lost

The goal of “EEG AR: Things We Have Lost?,” is to develop a user interface that will allow participants to conger up augments by simply thinking of them existing at a specific location using biometric sensor technology. A database of augments will be generated base around the broad theme of “Things We Have Lost,” things such as pensions, empires, dodo birds. Participants, or test subject, would be outfitted with EEG brainwave sensors and asked to think of a specific object or idea from the database. If I can detect and record a measurable and consistent pattern in the participants brainwaves, I should be able to use image recognition technology to assign a PHP database call which would instantiate an augmentation just in front of the participants GPS location. The person would then be taken out into the city to see if it is possible to create and place augments just by thinking them into existence. If successful these augments would remain at the location they were produced and be visible on any iPhone or Android device.

ManifestAR was recently awarded the ARtSENSE Commission at FACT, in Liverpool.

FACT (Foundation for Art and Creative Technology) has been leading the UK video, film and new media arts scene for 20 years with groundbreaking exhibitions, education and research projects. The organization aims to pioneer new forms of artistic and social interaction with individuals and communities.

ARtSENSE is a European research project in collaboration with 2 other cultural organizations and 5 technical and research organizations to develop wearable technology that aims to improve and augment the gallery and museum visitor experience. The technology will be discreetly embedded into eye glasses, with information beamed directly onto the retina of the wearer. The planned research will look at monitoring the physiological activity of the user, included eye-tracking, heart rate and skin conductivity to ascertain the “state” of the user (bored, excited, interested) and to then adapt the information sent to the user (textual, audio, video) accordingly.

This commission was awarded in order to create a significant new artwork that aligns with one of its current research interests looking at augmented visitor experience.

Our project, entitled "Invisible ARtaffects" will explore linking augmented virtual objects with audience response translated through compact wearable bio, audio, eye and brainwave sensing devices. We will create individual and collective works based on the theme; I think, I see, I speak, I feel- therefore it is. Our research will involve a year-long collaboration between ManifestAR artists, FACT and the ARtSENSE Consortium. The project will culminate in an exhibition at FACT in 2013. ManifestAR will also explore and create work in the city of Liverpool, beyond the galleries at FACT.

“EEG AR: Things We Have Lost,” will be my contribution to the project.

Mysterium Cosmographicum Cumulatus

In his first major astronomical work, Mysterium Cosmographicum (The Cosmographic Mystery), published in 1596, Johannes Kepler attempted to demonstrate the periodic conjunction of Saturn and Jupiter in the zodiac. He began experimenting with the five Platonic solids, which he thought could be nested, within one another to produce six concentric spheres, corresponding to orbits of the six known planets—Mercury, Venus, Earth, Mars, Jupiter, and Saturn. By ordering the solids correctly—octahedron, icosahedron, dodecahedron, tetrahedron, cube—Kepler found that the spheres could be placed at intervals corresponding (within the accuracy limits of available astronomical observations) to the relative sizes of each planet’s path, assuming the planets circle the Sun.

Kepler attempted to explain the proportions of the natural world and the cosmos in terms of music based on a central set of “harmonies” or the musica universalis, “music of the spheres,” which is an ancient philosophical concept, studied by Pythagoras, Ptolemy and others, that regards proportions in the movements of celestial bodies as a form of inaudible mathematical music.

Kepler toiled over the idea of nested Platonic solids for most of his life and could never corroborate the theory with the improved observational evidence collected by his contemporary Tycho Brahe, because it was simply wrong. Despite his fondness of the idea, it was Kepler’s tenacious insistence on fact that led to his first law of planetary motion, The orbit of every planet is an ellipse with the Sun at one of the two foci, thus setting Western culture on the path to reason and, indeed, modern science.

Project Description
Mysterium Cosmographicum Cumulatus will place observers at the center of Kepler’s nested Platonic solids with augmented reality technology. Located at the precise GPS coordinates 42.273417,-71.808017, atop the Earle Bridge on the campus of the Worcester Polytechnic Institute, the project will allow people to view the solids slowly rotating around them using any late model Android or iPhone.

Visitors will stand at the center of the bridge and launch a mobile application which will activate one of the five solids. The object will appear in the camera of the user’s phone as a immense translucent 3 dimensional shape viewed from its center point. Prompted by the user, the solid will begin to slowly rotate as sound derived from the radiation and seismic data collected from each of the five innermost planets and transposed into audible scaled resonance, will emanate from a hidden 5-channel outdoor speakers system on location. Please see an example from project collaborator, Jeff Talman.

The speaker system will use parabolic dish technology to isolate the sound to the bridge so that it will not be heard elsewhere on campus.

The Platonic solids will be created by mapping semitransparent video animation based on the five classical elements, Earth, Water, Air, Fire, and Aether, onto polygonal models of an octahedron, an icosahedron, a dodecahedron, a tetrahedron, and a cube.

HD Mobile Device Proposal

In June, 2011, ManifestAR created the Venice Biennial 2011 AR Intervention in the main pavilion complex of the Biennial in Giardini and in the Piazza San Marco.

Goddess of Democracy, by 4Gentlemen, Piazza San Marco, June 5th, 2011.

In a city with so much vying for people’s attention, what is needed is something with more spectacle appeal. What follows is a proposal for the development of two different types of HD augmented reality mobile devices.

The first is a remote control cart equipped with an HD display monitor, which could be deployed as an art performance at dusk in public squares across Europe. People could push the device around the square to look at various AR projects, or we could take it for a spin remotely and interact with the crowd. Of course the work would still be viewable on smart phones.

The second is a proposal specifically for Venice which would include HD display monitors attached to the windows of the Actv Venice Public Transit Boats, specifically aboard the boats on the #1 line, which includes all stops on the Grand Canal.

Remote Control HD Augmented Reality Mobile Device

Visualization of remote control HD augmented reality mobile device.

View a video of a remote control golf bag cart. The cart could be retrofitted with a medium sized HD flatscreen monitor attached to a smart phone via DNLA technology.

Visualization of remote control HD augmented reality mobile device in Piazza San Marcos, Venice, with the Crystal Coffin augment by Lily & Honglei.

Venice Public Transportation Boat Augmented Reality Mobile Device

Actv Venice Public Transit Boat with proposed HD augmented reality device attached to window exterior.

Visualization of view from inside the #1 boat near Rialto Mercado on the Grand Canal with the Goddess of Democracy augmented into the campo.

Border Memorial: Frontera de los Muertos

The Border Memorial: Frontera de los Muertos, is an augmented reality public art project and memorial, dedicated to the thousands of migrant workers who have died along the U.S./Mexico border in recent years trying to cross the desert southwest in search of work and a better life.

Visualization of the Border Memorial project.

Built for smart phone mobile devices, this project allows people to visualize the scope of the loss of life by marking each location where human remains have been recovered with a virtual object or augmentation. The public can simply download and launch a mobile application and aim their device’s cameras at the landscape along the border and the surrounding desert. The application uses geolocation software to superimpose individual augments at the precise GPS coordinates of each recorded death, enabling public to see the objects integrated into the physical location as if they existed in the real world.

This map indicates the GPS data points where remains have been recovered.

Based on a traditional form of wood-carving from Oaxaca, The virtual augmentation objects consist of life sized, three dimensional geometric models of a skeleton effigy or calaca. Calacas are used in commemoration of lost loved ones during the Mexican Día de los Muertos, or Day of the Dead festivals. Tracing their origins from Aztec imagery and ushered into the modern era at the turn of the twentieth century by Mexican artist José Guadalupe Posada, calacas are generally depicted as joyous rather than mournful. According to Aztec belief, despite the tragedy, death should always be a joyous occasion. In the tradition of Día de los Muertos, the Border Memorial project is designed to honor, celebrate and remember those who have died and to vault this issue into public consciousness and American political debate. The project will provide a kind of lasting conceptual presence in an otherwise ephemeral physical environment and cultural discourse.

Border Memorial Screenshot

1 page describing the project’s relevance to CRCA’s mission:
CRCA’s mission to investigate what constitutes the potent cultural acts of our time and the viable mechanisms that should be engaged to create them is consistent with my working methodology over the past two decades. This is what inspired me to make this proposal.

For the past eight years, I have worked on the corner of Tremont and Boylston Streets overlooking the historic Boston Common, the first public park in the United States. I walk across the park every morning. As I do, I often contemplate the role that the town square plays in shaping of political discourse and national identity formation. As the location of the public sphere, the town square is where we air grievances, display solidarity, express our difference, celebrate our similarities, remember and mourn. Public hangings took place on the Common until 1817, an example of the public reinforcement of the shared values of right and wrong. The Common still maintains a tradition of soapbox oratory and we even have a town crier, who exchanges weather forecast and sport scores for spare change.

This is why monuments and memorials are located in town squares. As Greg Ulmer points out in his book Electronic Monuments, it is an expression and acknowledgment of sacrifice on behalf of shared values. The public square is a geographical anchor for the public sphere. As Benedict Anderson argued in Imagined Communities, the nation state was made possible, in part, by the printing press, including the invention of associated forms and practices such as the novel, contributing to the creation of national languages. Newspapers and the rise of a mass reading public within industrialization are part of this history.

In the early 1990s we witnessed the migration of the public sphere from the physical realm, the town square and its print augmentation, to the virtual realm, the Internet. In effect, the location of public discourse and the site of national identity formation has been extended into the virtual world. As Bernard Stiegler, among others, has argued, this virtual dimension, with its industrialization of collective memory, is again transforming the “We,” away from the nation state to a new collectivity that he fears will be an ersatz global “America.”

This threat/promise is a context for experiments in augmented reality which allows us to overlay this virtual public sphere onto our experience of the physical, cultural world. It is important to keep in mind that the practices of the virtual public sphere have to be invented, just as the equipment is invented. What is the future of “We” in electracy? It is open to invention. “Augmented Reality as Public Art” will explore the potential of augmented reality as public art, mobile location based monuments and virtual memorials.

List of specific labs & equipment being requesting:
I have the equipment I need. Mostly, I need a place to work. An office would be ideal, but use of a desk, power and internet would do. I want to be able to work at the center and use it as a base for operations while I am there.

List of all collaborators who need access & to which specific systems they require access:

John Craig Freeman is a public artist with over twenty years of experience using emergent technologies to produce large-scale public work at sites where the forces of globalization are impacting the lives of individuals in local communities. His work seeks to expand the notion of public by exploring how digital networked technology is transforming our sense of place. He has produced work and exhibited around the world including in Xi’an, Belfast, Los Angeles, Beijing, Zurich, New York City, Taipei, São Paulo, Warsaw, Kaliningrad, Miami, Bilbao, Havana, Atlanta, Calgary, Buffalo, Boston, Mexico City, London and San Francisco. In 1992 he was awarded an Individual Artist Fellowship from the National Endowment for the Arts. He has had work commissioned by both and His work has been reviewed in The New York Times, El Pais, Liberation, Wired News, Artforum, Ten-8, Z Magazine, Afterimage, Photo Metro, New Art Examiner, Time, Harper’s and Der Spiegel. Christiane Paul cites Freeman’s work in her book Digital Art, Second Addition, as does Lucy Lippard in the Lure of the Local, and Margot Lovejoy in Digital Currents: Art in the Electronic Age. His writing has been published in Rhizomes, Leonardo, the Journal of Visual Culture, and Exposure. Freeman received a Bachelor of Art degree from the University of California, San Diego in 1986 and a Master of Fine Arts degree from the University of Colorado, Boulder in 1990. He is currently an Associate Professor of New Media at Emerson College in Boston.

Estimate of hours required in each space & on particular systems:
I would hope to spend 20 – 40 hours per week from January 27 through February 24, 2012 at the Center.

Brief description of project funding sources:
I have been awarded a sabbatical leave from January 1 through August 31, 2012 by the Department of Visual and Media Arts at Emerson College. This equates to $42,000 worth of support for the project.

Emerson has agreed to pay the administrative overhead expenses if this application is successful.

Last year I was awarded a $2,000 in Faculty Development Funds Grant in support of this work by Emerson College for the 2011-2012 academic year.

This year I have a travel budget of $1,500 which I will use for travel to, from and around California and the border region.

I hope to submit a proposal to the curatorial staff of the Museum of Contemporary Art, San Diego in hope of gaining institutional endorsement and logistical support for publicizing the deployment of the Border Memorial.

Timeline of project including goals & milestones:

January, 22 – 26, 2012 – ”ManifestAR: an augmented reality manifesto”, SPIE 
Engineering Reality of Virtual Reality, Panelist/Paper, San Francisco, CA.

January, 22 – 26, 2012 – Center for Research and Computing in the Arts, Research Initiative, University of California San Diego, CA.

February, 2012, Deployment and documentation of the Border Memorial project, AZ.

February 22-25, 2012 – “Mobile Art: The Aesthetics of Mobile Network Culture in Place Making,” College Art Association 2012 national conference, Panelist, Los Angeles, CA.

February 22-25, 2012 – Public Art in the Virtual Sphere: College Art Association 2012 national conference, Round Table Chair, Los Angeles.

February 22-29, 2012 – “LA Re.Play,” Citywide mobile art exhibition, Digital Media Art Gallery, University of California, Los Angeles, CA.


One Response to Augmented Reality as Public Art

  1. Pingback: Research on AR environment – SM4602

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s