Event: Data for Sound and Image Workshop, by Derek Holzer and Sarah Kolster
Place: Center for Music and Technology, Sibelius Academy
Dates: 11-15 September 2006

.

Event: Composing with Pure Data Workshop, by Koray Tahiroglu
Location: Media Lab, University of Art and Design Helsinki
Dates: 2-6 October 2006

.

Event: Teleradio Workshop, by Adam Hyde
Location: Center for Music and Technology, Sibelius Academy
Dates: 23- 27 October 2006

Images (Flash and Pure Data)

1. Concept and motivation

The concept behind the project is to build a hybrid sound and image composition prototype with a custom, visual appealing and fluid user-interface. Each sound, or component of sound, should have a symbolic on-screen representation. The tools chosen to develop a working prototype were Flash, for the visual side, and Pure Data, for the sound side. The OSC (Open Sound Control) protocol would be used as a “bridge” between the two applications.

Although I have extensive experience in using (and teaching) Flash, and digital sound tools, I had little experience with Pure Data (or similar applications, like Max/MSP) prior to enrolling at Media Lab. Therefore, I attended a series of workshops that would provide the necessary knowledge to build the prototype.

This prototype is directly connected with my research plan, entitled “iAVo – Interactive Audio/Visual Objects: Application for flexible multimedia expression”, since the prototype materializes some of the plan’s key concepts. Therefore, for simplification purposes, I will entitle this prototype “iAVo Gamma” or “iAVo prototype”.

2. Brief description of the project

iAVo Gamma consists of a Flash-based visual side and a Pure Data-based audio side. The visual side is the graphical user interface controlling the audio. Information is sent from Flash to Pure Data through OSC.

There are 3 sound loops that can be controlled by manipulating graphical symbols (icons) in Flash. The manipulations are performed by drag and drop operations in Flash (for position), and pressing the up and down keyboard cursor keys (for size). An “XYZ” logic was implemented, where X is the horizontal position of the icon on the screen; Y the vertical position, and Z size. For this prototype, the following audio/visual connections were established:
- sound 1 (melody loop): X controls volume and Y controls pan;
- sound 2 (bass): Y controls volume;
- sound 3 (rhythm): X controls volume,Y controls filter frequency, and Z controls delay.

Since OSC uses a network connection, several users can interact at once, enabling collaborative composition. Since Flash player is pervasive, any user with Flash player can access the server running Pure Data and use the iAVo Gamma. The audio feedback is provided either by a local speakers system or by audio streaming. A local area network (LAN) connection is preferable, because of audio delay issues over the Internet.

3. Evolution of the project

a) Pure Data for Sound and Image Workshop

The iAVo prototype began being developed in the Pure Data for Sound and Image workshop. There I began exploring Pure Data and OSC. I also began exploring Flosc, an implementation of OSC in Flash. In this workshop, I built a basic Pure Data sound tool that manipulates 3 different audio loops. I also developed the Flash user-interface. Finally, I tried to connect the two (sound tool and user-interface) using OSC, but faced many problems which I could not solve entirely.

b) Composing with Pure Data Workshop

In the Composing with Pure Data workshop, I developed further the Pure Data sound tool and managed to solve the problems concerning OSC. Therefore, by the end of the workshop the prototype was working: the user-interface was controlling and manipulating sound.

c) Teleradio

After having completed a working prototype, I wanted to explore collaborative composition and streaming technologies. Namely, to collaborate using the prototype over the Internet or local area network (LAN). The Teleradio Workshop provided with know-how regarding network communication and streaming media. The prototype was upgraded with collaborative and sound-streaming capabilities. Quick Time Darwin Streaming Server was implemented for this purpose. The delay (audio latency) over an internet connection proved to be too great for a fluid audio streaming feedback. But in a local area network, the delay proved to be very small. A wireless LAN was used, and it would the be ideal type of network for this kind of collaboration.

During the Teleradio workshop, experiments in electronics were also pursued (namely, building a TV video signal transmitter).

4. Possible uses and future developments

iAVo Gamma could be used for audio/visual performances; collaborative composition; interacting with a large screen, or large screens (in a performance or public space context)

To be more operational, however, more content (audio loops and visual icons) should be added.

The audio / visual connection logic (presently based on a visual “XYZ” paradigm) should be reconsidered, since it could be limiting to apply this type of logic to all visual symbols – it could lead to confusing clusters of icons.

The “one sound, one icon” paradigm could also be expanded to a “one sound, one screen, several icons” where each sound would be represented by a screen, and each icon on that screen would represent a given aspect of that sound. For example, the “guitar” screen would have icons representing volume, filter, distortion, delay, etc. Several projections could then be used to convey a “full picture” of the sound manipulation taking place.

There could also be a symbolic representation of the whole sound structure (or “song”): for example, a background animation. Manipulating this symbol would affect the entire song.

The mobile phone should also become a tool for audio/visual composition, with the use of the knowledge I gained from the “Mobile Rapid Prototyping” workshop and “live2screen” project.

5. Technical notes and links

Tools and technologies used for this project were:

Flash:
http://www.adobe.com/products/flash/flashpro/

Pure Data:
http://www.puredata.org/

OSC:
http://www.opensoundcontrol.org/

Flosc:
http://www.benchun.net/flosc/
http://transmote.com/flosc/ (tutorial)
http://www.randominterface.com/forum/ (forum)

Audacity:
http://audacity.sourceforge.net/

QuickTime Darwin Streaming Server:
http://developer.apple.com/opensource/server/streaming/index.html

6. Acknowledgements

Thank you to the teachers: Derek Holzer, Sarah Kolster, Koray Tahiroglu and Adam Hyde.

Thank you to André Carrilho for Flash visuals development