First published at COSIGN-2004,
14 – 16 September 2004, University of Split (Croatia),
Keywords
Interactive computer art, keyboard interaction, interaction design
INTRODUCTION
This work is a reflection upon two different aspects of communication and personal expression through digital media. The first is text-based communication through digital media, and how the extensive use of such forms of communication may affect our culture and social patterns. The second aspect is the dynamic media itself as a form of expression, and how the interactive properties of a system are being used as a way of expressing oneself artistically.
This paper will try and bring these two aspects of the dynamic media together, by describing a series of interactive artworks that have been developed around this theme.
We start by giving a brief description of the two exhibited versions in the series, and thereafter we discuss the design process leading to these pieces.
In contrast to many other works in the area of interactive computer art, this work has not aimed for developing new technology, instead focus have been to explore expressive properties of digital material that already exist.
This is done by describing the development of four experimental artworks in a series called Pixel Express. At the beginning of the process, our focus was on how digital media affects our culture, and how the meaning of a digital conversation could be captured and illuminated in a visual and interactive form. However, as the project went on, focus has shifted towards the act of typing, the meaning of the keyboard and the computer-mediated form of expression and interaction itself. We discuss this shift of focus and how it has emerged through our creative process.
DYNAMIC SYSTEMS AS ART
The use of new technologies of course has an impact on artwork. Artists use available tools to express themselves, to experiment and get ideas from, artists turns everything around and develop new ways of using and understanding the world and the society. Artists make new tools.
Walter Benjamin (1937), opened a discourse on the authenticity of the artwork, the question of originality and the roles of creator and audience in the time of mechanical reproduction. He discussed the use of new technologies effects on the art works aura. The camera had made it possible to depict reality and that made the painters free to work with pure painting matters such as materiality, colours etcetera. In this sense, the camera opened the way for the modernity in fine art. Soon artists also made the camera a tool for expression and experimentation, however, it took almost fifty years after it’s invention in 1840 for photography to be accepted as a real art form. Similar to what Walter Benjamin (ref) described as the influence of photography on the art object and on art itself, is the question of how the new digital technologies, such as the use of mobile phones, video games and the Internet have and will change the way we express ourselves, our language, our way of thinking and communicating.
In 1977, Alan Kay and Adele Goldberg published the article “Personal Dynamic Media”, in which they presented their then quite pioneering work and visions on computer software and hardware. Among other things, they proposed that for a technology to become really successful, it should be usable even for children, placing high demands on robustness, bandwidth and ease of use. The article is mostly recognized for providing the first sketch of the laptop computer. However, it also describes how the object oriented-programming language, or “communication system”, Smalltalk was used by children to create their own pictures, animations, interactive applicatios and pieces of music. By describing the outcome of children’s activities as “personal dynamic media”, they also stressed construction of computer programs as a way of expressing and sharing ideas, as opposed to a more traditional view of computers as tools for automating calculations or other work-related tasks. (Kay and Goldberg, 1977)
Today, many of the visions discussed by Kay and Goldberg in the seventies have become commonplace, people do to large extents use computers for creative purposes and there are many powerful and easy-to-use software for creating and communicating pictures, music, interactive homepages and animations.
However, the area of designing interactive systems for personal expression is still suffering from the fact that production of artistically appealing computer systems are very hard to produce for artists without the sufficient technological expertise. Most artists who express themselves through computational media still have their backgrounds in technical areas such as computer science and engineering. While this is a relethe vant background for being able to realise the artistic ideas, art critics have been concerned that the lack of a background in fine art often results in interactive installations that are technologically adthe vanced but “artistically naïve” (Beryl graham, p 21).
A common approach to this problem is for artists to work in close collaboration with technicians. Such collaborations are commonly argued to hold some particular adthe vantages, both from a technical development perspective as well as from a perspective of theoretic and artistic development (steven wilson). The artist may provide imaginative ideas that pushes the technological adthe vances forward into new directions, and the technicians provide the artists with the set of skills required to actually realise the artistic ambitions. For researchers in human machine interaction, the interactive installation may give rise to new modes and forms of interaction which would not have been explored otherwise.
This work has been a collaborative project where the first author is a PhD student in human machine interaction, with a particular interest in the design of interactive systems as a means of communication and personal expression. The second author is an exhibiting artist in the fields of fine arts: fiber art, new media and sculpture. The initial phases of this work also included three other participants, all with artistic backgrounds, serving as discussants and partners when generating the initial ideas for the project.
THE DESIGN PROCESS
The design process has involved an initial period of brainstorming and development of the basic ideas for the piece, several periods of sketching, implementation, testing the hardware and software components of the piece, as well as public exhibitions of two different installations at Royal Academy of Fine Art in Stockholm, Sweden and at the Switch Media Festival 2004 at Chiang Mai University Art Museum, Thailand.
The work process was initiated by a collaborative coursework in art and new media at the Royal Academy of Fine Art in Stockholm, but has continued since with several new versions variations of the piece.
During the actual development of the piece, the design process was divided so that the first author developed the interactive and dynamic properties in the programming environment, while the second author focused on making and selecting the photographs that should appear in the piece as well as designing the physical parts with which the user should interact. Both authors were also continuously discussing the various prototypes and provided possible directions for future development.
We would not like to characterise this development as a collaboration between an artist and a technician, but rather as a full collaboration between two designers with complementary expertise. Both partners were actively involved in shaping the appearance of the system, and clearly the piece world not have been developed to its current shape and functionality if either of the partners had been absent in the design process.
As a piece designed for collaborative interaction, it was also essential for the design process to involve at least one partner during the phases of development, so that the interactive properties could be fully tested internally before putting them on show.
Initial sources of inspiration
Our work has been based around two themes:
Text-based communication through digital media
Protection coded digital material, and the aesthetics embedded in the way such appear.
This section summarises these influences.
Text-based communication
Beginning with a given theme ”remote experience”, we decided to focus on distance in the context of communication with digital media and to explore ideas around social presence in computer mediated settings. Our focus was on text-based communication, such as email, SMS and online chatting, and issues related to the distance and disturbance often experienced when engaged in such forms of communication. We were particularly concerned with how this could be represented and designed to invite for playful interaction and at the same time result in an aesthetic experience. These decisions were made after several meetings based on lectures, literature reviews and our respective backgrounds and areas of interest.
We also realised that a three party conversation has some fundamental and particularly interesting differences from a normal two-party dialogue. A conversation between three persons often involves two people that are active, while a third relates to the conversation as witness or observer. These role patterns usually shift several times through the conversation, but could also be completely dominated by one person, independently of how many participants there are.
We decided to build a system consisting of three physical chat-rooms located in three cities. As you type in any of these locations, a picture is appearing on all the screens, but as you remain silent and no longer take part in the conversation your picture disappears. Apart from expressing yourself through what you write, you participate by making yourself, i.e. or your picture, visible for yourself and for others.
This idea was based around our experience that in a natural conversation, one person may, without noticing it, dominate the discussion completely. Others may feel as though active in the conversation, without actually saying anything. In an online chat session or in an email conversation however, such silences become more evident. Our piece intends to in a visual way bring light to and make this observable, and at the same time raise questions regarding how the technical development affects our language, our means of communication and our social patterns.
Pixels and protection coded images
Another source of inspiration was the aesthetic appearance of the interactive Installation “Out of Sight Out of Mind” [West, 2001 #110], which was an interactive video installation produced in 2001 by Ulla West (the second author) and Anita Malmqvist, both exhibiting artists in the field of fine arts and new media.Figure SEQ Figure \* ARABIC 1. Out of sight out of mind installation.
This project worked with the protection code for commercial dvd-productions. A coded pixel protected film is moving between two monitors placed in opposite directions of the room. The film is jumping between the monitors depending on how people move in the room. The monitor where the film is shown initially is pointing at a wall reflecting the images on the screen (see REF _Ref78444792 \h Figure 1).
The sound from the original film made people curious, but when they were getting closer a light sensor reacted and the film “jumped” over to another monitor facing the center of the room in a distance. To be able to see the film the visitors had to collaborate. And if and when they succeeded the film was pixel-coded. It was important during the work with this project to make the coded film in itself joyful to watch, like a non-figurative film animation. The sound played an important role in this installation.
Out of Sight Out of Mind has also been showed at Zone-IP Fluid Image New Media, in Grenoble France 2003. This time it was presented as a projected code-film (see REF _Ref78445916 \h Figure 2), not as an interactive art installation. Figure SEQ Figure \* ARABIC 2. Two stills from the “Out of Sight Out of Mind” video
While working on the project, we also heard that the pixel, i.e. the small components that build up a digital image, would be fifty years in 2004, which brought up a whole new series of ideas.
Each pixel on a colour computer screen has a red, a green and a blue component which are controlled by the additive colour system RGB. In this system, colours are mixed by setting different strength of the three light sources. If all components are lit on full strength, the eye will read it as white, and the same amount of all components will result in a neutral grey. If all three are off then the result is black. If all three light sources are of different strength, new colours are made. In this way, most colours can be created that the human eye can perceive.
Development of the first prototypes
As the practical process of actually implementing the installation began, it became clear that the concrete work leads forward. We realised that the real artistic process began only as it was taken into practice. This process included much practical work in the form of programming, sketching and searching for images as well as hands on testing of different variations of the prototypes.
The first prototypes were built around the ideas of faces blending with in with one another as you type. After developing the first prototype based on faces we realised that this would concentrate the focus completely on the shifting faces instead of a mixed occupation on the activity of writing and the uncertainty of what that could lead to.
The first version was also developed using only two discussing partners, and implementing the same system for three partners would require a significantly more adthe vanced system of implementation.
Technical difficulties in blending three different pictures.
An important element of the process became to discuss and test the prototypes over the internet, apart from each other. This was both in the sense of the theme that we worked with, but also because of the practical situations with a software that needed to be run over a network. Around thirty slightly different prototypes were developed and tested locally by the first author, and eleven of these were tested collaboratively.
We decided to combine our ideas around the RGB and pixels with the three party conversation and our result was the working name “Pixel Express”.
This development was particularly inline with the second authors previous work with pixelated imagery. We decided to combine the ideas of the three party conversation, the aesthetics of “the out of site out of mind installation” and the knowledge about the RGB-system.
The first exhibited version
In our first experiment, we focused on a three-part conversation that took place between users through keyboards painted in three colours (red, green and blue). Your presence, i.e. your keystrokes, resulted in the exposure of a shared picture, pixel by pixel. If you were passive, the pixels in your colour would disappear, one by one.
In the first exhibited version of the piece, we divided the conversants into three, a red a green and a blue, represented as keyboards painted in the three colours. When participating in the piece you take on the role of either of these colours. If you type on the red keyboard, you will control the red components in the picture, i.e. the red component is added to a random set of pixels in the picture.
If you are passive, the picture will eventually loose your colour component. If nobody types on the red keyboard, the red components in the pixels will be turned off, one by one.
Picture: pixel express projected on the wall
We did not intent to control what the topic of the conversations should be. Instead of focusing on the content, we choose to visualise the activity in the conversation.
The black and white pictures that appeared, were taken during a study trip and where exchanged every other minute. This meant that the conversation needed to be kept alive in order to see all pictures.
The actual shape and appearance of the interactive properties of the piece were however designed in an iterative process of many short loops when working hands on with the programming of the piece, resulting in 32 slightly different prototypes, of which only a few where tested by the whole group.
A central part of the project was also that a shared picture should be revealed as the participants took part in the “trialogue”. Therefore an important part in the design process was to select a picture, or a set of pictures to play this role. The piece were tested with several different pictures during this development process, and we found the most interesting effect to appear when using black and white pictures, since the red, blue and green components of the participants still would result in a bright and colorful projection.
Independently of what the topic of the conversation is, your activity controls how the picture will end up like.
Figure SEQ Figure \* ARABIC 3. Three keyboards in different colours.
# 2 & #3
After having done the first exhibit some further experimentation lead to testing of the piece with a moving, rather than a static image to be displayed. It then became important to select an episode of film to serve this role. We had discussions regarding using live video in the installation, but were concerned that a lot of work in new media art concerning live video relates to surveillance, and we would prefer to keep the simplicity of the piece without bringing in two many different sources of inspiration.
We also developed a version of the piece to be demonstrated online, but we found that a central aspect of the piece was its physical components with the three colored keyboards, and that the piece when brought online would loose this aspect.
Instead, further focus was drawn to the hardware components of the installation.
The second exhibited version
In our latest experiments, we have chosen to completely exclude the written words typed into the machine. Instead, two computers with keyboards with no signs or characters are connected and by typing, the pictures on the computer screens become mixed.
Entering a dark room while visiting a brightly lit museum is not always the best environment for interactions.
This was of several reasons.
Firstly, as we observed when looking at the way people interacted with the first prototype, the meaning of the text typed into the machine became only a peripheral part of the interaction and understanding of the piece. Independently the meaning of the sequence of characters that people typed, the same expression would appear in the visualization. Instead of viewing this as restricting participation and interaction with the system, this turned out to allow for a playful and creative experience, and users became more active when they realised that they did not have to concern about what they were supposed to understand and do.
Secondly, restricting the communication to the expressions available in a specific language also hinders people from different cultures to meet in artworks like this. When bringing this experiment to an international conference, particularly to a country with another set of characters than was available on our keyboards, a natural consequence was that the keys on the keyboards had to be blank.
Thirdly, we believe that the simplicity made the experience richer and more direct. Through the act of writing, or typing, you make ”your perspective” visible to others, independently of what you say.
In #4, the version we showed in Chiang Mai, Thailand, we included pictures from worlds news as they were presented in our part of the world. The pictures were taken the day before leaving Sweden, on our way to Chiang Mai and the festival. The new spictures were from the two international news channels dominating in our homes. With a private parabol antenna it’s of course possible to watch any news channel in the world. None of these two channels, CNN and BBC, represents Asia, or an asian perspective, even if they show news from and about Asia.
Figure SEQ Figure \* ARABIC 4. The installation.
Picture: the installation in Chiang Mai
When communicating through pixel express #4, it is only through the act of writing, or typing, that you make yourself and your perspective visible to others. The keys on the keyboards have no signs or characters. By typing, your picture gets mixed up with someone elses view. This ”disturbance” is here represented as vertical stripes of another view, referring to the effects of an un-tuned TV-antenna.
The reason why we choose pictures from newschannels in this version of Pixel, was also to put a finger on the different views and strategies we have from northern European countries compared to other parts in the world, in our relation to new media and to make the technique our tools.
We were not particularly interested in the questions around media and reporting news. More the differences in perspective in general.
Figure SEQ Figure \* ARABIC 5. Closeup of two news pictures that are temporarily blended with one another.
DISCUSSION
We are concerned that many exhibitions of digital video or interactive demonstrations are experienced as quite demanding to the visitor. It might for instance take more time than expected, or be too intellectually demanding. We were also concerned that the user should not be required to stay at the piece at any particular amount of time. Instead we wanted to create a form of directness equivalent to that of a traditional sculpture or a painting, where the choice to spend time with the piece lies at the observer.
Closely related to this area of work are also the areas of computer game design and the design of interactive exhibits at for instance technically oriented museums.
We felt that unlike a computer game, an interactive art-piece should be perceived mostly on an aesthetic and emotional level, rather than a source of challenge or as an intellectual or physical activity. And unlike an educational exhibit at a museum, the purpose should not be to teach, but to express or give rise to ideas.
A central concern was also that interaction with the piece should be tightly coupled to the output generated. The user should sense that there was a direct connection between the two, and not at any case be tempted to believe that effects produced by the computer had no mapping to the actions performed by the user. Moreover, the interaction should have a meaning – for the system as well as for the user – and not only serve a purpose of its own.
We also wanted users to interact with each other through the piece, and not only with the piece itself, and that several participants should be able to interact with it at once. We were also concerned that not everyone should have to interact with the piece, some qualities of the piece should be achieved also through passively watching your friends engagements with the piece.
with which users interact using keyboards.
In the book abstracting craft, the author highlights the role of the hands in any craft, even the digital ones. These activities with our hands, is argued to be neglected in most discussions on interaction design. What does it feel like to let ones fingers play over the keyboard, as opposed to letting any other form of activity control the computer.
The physical, as well as the on screen, appearance of the piece is an important aspect of interaction, therefore we put particular effort into the deign of the hardware with which the users interact.
The two versions of the piece that have been exhibited so far, have not been considered by us as proper art pieces, but rather as “prototypes” or experiments leading towards new variations and prototypes and deeper knowledge in the forms of expressions available in the material itself.
As stated by Höök et al (2003), an interactive art piece is to the artist more like a scientific article to be reviewed than a software to be evaluated. Therefore an interactive artwork can not be evaluated with the same measures as other interactive systems, since they can only make statements regarding the interaction and never provide any feedback on the artistic qualities. Like comments on a research paper, feedback on an artwork and acceptance to give a show, serves as forms of acknowledgement of the work performed and it also enable new possibilities and directions of moving forward.
To us, the exhibitions has served more the purpose of getting feedback on our progress, verbally as well as through observation of the interaction between participants and the installation. Verbal feedback consist partly of informal comments from friends and colleagues, but also more formal reviews by art critics and others. Even though our observations have been made informally and our expectations have not always been explicitly stated, we have often been surprised by how the interaction took form. Strengths and weaknesses of the piece have become evident and possible for us to discuss and analyse, and we have been provided with many directions for further work. Even if both variations were exhibited with much pride, an important effect of putting them on show was that it gives rise to many ideas of things that could be improved or done differently. Both these occasions have been very important for our progress, each leading the design forward into directions that otherwise had been impossible to foresee.
The project reported on here has been a fruitful starting point for successful collaboration to state a lot of interesting theories and wonderings around big questions.
Going deeply into reading, searching and discussing how to find answers by testing and actual production of interactive experiments. To try to visualize complicated assumptions. To find out and to point out. But first at the time when the talking and the theories exceeded to experiments and playful explorations of the possibilities with the materials, the expressions took over and we began to play with our ideas and creative solutions as tools. This work led us to “free” playful creativity and maybe something that can be called art was happening. When we decided that it was ok just to make it “beautiful” and playful things happened. Probably or maybe it is this troublesome way, full of resistance that made way for the easiness.
In our positive moments of collaboration, we look at our work and constantly changing results in this way, and we have all necessary hope and confident to go on and are convinced that we will give the work more time and place, certain that it will lead us to something very important and interesting or that it is ok just to have fun.
Apart from the actual production of the two exhibited artworks, our explorations have resulted in some particular insights into the mechanisms involved in the design process of developing interactive art.
Initially, the concept of presence was central, the pictures of the faces of active participants of the dialogue should appear.
The interaction with the piece became fundamentally different when the role of the keyboard became to control the visual output, rather than as a device for insertion of text.
Relation to participatory design.
Working in a collaboration between an artist and a software developer, does in several respects resemble the process of participatory design.
In participatory design processes a group of users are actively involved in the design process, including the phase of brainstorming, sketching and low-fidelity prototyping, as well as testing and evaluation of hi-fidelity prototypes. However, in such design processes, users are normally not involved in the actual implementation of the system, which needs to be done by professional software developers.
Like users taking part in a participatory design project, the artist may not have enough technical skills to actually take an active role in the implementation process, however the artist is essential in the process of developing the artistic ideas and providing feedback on the work as it progresses.
However, as have been noted by for instance (Candy and Edmonds) artists taking part in such collaborations often experience a lack of control in the creative process, and instead the practical and creative skills of the technicians take on an active role in important aspects of the design.
Moreover, artists working with technicians may lack the methods for developing and evaluating user interfaces that are easy to understand and comfortable to interact with.
Exhibits as a source of evaluation
The process has not included any formal evaluation of how users perceive or interact with the piece [Höök, 2003 #106]. Instead our own experience of the different versions as they appeared during the development process, feedback from friends and family as well as our impressions from having users interacting with and making comments on the different versions at exhibitions, has served as more informal methods for informing the process of interaction design. This does not mean that the end user has not played a central role in the work process. However, instead of engaging users in the development process, the design choices have been influenced by an imagined audience. The two exhibitions have played a central role for understanding how the piece will take form in a realistic setting, and have hence served more as user tests for further development than as the display of the final work of art. Both occasions have involved surprises and have given much relethe vant feedback to the ongoing work.
Keyboard interaction
Our focus has been on text-based communication, such as email, SMS and online chatting, and issues related to the distance and disturbance often experienced through such forms of communication. We are particularly concerned with how this could be represented and designed to invite for playful interaction and at the same time result in an aesthetic experience.
The activity that most people associate the keyboard to might be to write texts, for instance to write messages to each other, to write articles and essays, and to take notes. However, in terms of interaction design, the keyboard is much more than a device for inserting text into the computer. Not so long ago, the keyboard was just about the only way for end users to interact with their computers. The keyboard was used not only for programming the computer, but also to navigate its operating system, to locate and open files, and for managing all the tools available in them. Although there are alternative ways of interacting with software today, people still use the keyboard extensively for making shortcut commands, for navigation within files and folders, and for playing games.
A well-designed keyboard user interface plays a key role when you are designing applications. Many users prefer to perform most operations with the keyboard rather than the mouse. Visually-impaired users can navigate software more effectively using the keyboard, because using the mouse depends on visual feedback of the mouse pointer location. And mobility impairments can prevent a user from successfully navigating using the mouse, because of the fine motor control skills required.
What, is it that the keyboard itself has become to represent? What does the act of typing mean when there are no sighs on the keys? Could the keyboards themselves be seen as objects of a particular value for our culture? How will this meaning change as new and other forms of communicating with each other and with our technology appear?
SUMMARY
Our aim was from the beginning to examine how the text-based communication through digital media, and how the extensive use of such forms of communication affects the way we express ourselves: the way we talk and write messages and texts. The second aspect was the dynamic media itself as a form of expression, and how the interactive properties of a system could be used as a way of expressing oneself artistically. A third aspect was protection coded material, and the aesthetics of such “digital disturbances”.
We would not like to characterise this development as a collaboration between an artist and a technician, but rather as a full collaboration between two designers with complementary expertise. Both partners were actively involved in shaping the appearance of the system, and clearly the piece world not have been developed to its current shape and functionality if either of the partners had been absent in the design process
First at the time when the discussions and the theories exceeded to physical experiments with the programming materials testing with different kinds of hardware solutions, the expressions took over and our ideas and creative solutions started to take form. This work led us to “free” playful creativity and maybe something that can be called art was happening.
REFERENCES
M. Hjelm: Krig. artwork och Intelligenta maskiner som ingår i Nomadologin Kairos skriftserie 4
Art, War and Intelligent Machines, Nomadologin Kairos anthologhy seies 4
–Walter Benjamin, ‘The Work of Art in the Age of Mechanical Reproduction’, 1937
Ulla West 