More info: http://www.videojackstudios.com/projects/master-and-margarita/

ENTER PROJECT

More info: http://www.videojackstudios.com/projects/avol/

ENTER PROJECT

INSTALLATIONS/WEB

Role Playing Egas – shown at:

• Encontros de Arte e Comunicacao, CCB, Lisboa (2005)
• Fabrica Electronica, Restart (2005)
• EME Festival, Palmela (2005)
• The Upgrade! Lisbon, Lisboa 20 (2006)
• The Upgrade! International, Oklahoma City, US (2006)

Objectos Transitorios – shown at:

• Festival de Imagem de Oeiras (2004)
• Festival Transatlantico (2004)
• Atmosferas (2005)

INTERACT – MAGAZINE OF ART, CULTURE AND TECHNOLOGY

www.interact.com.pt/11 (2004)
www.interact.com.pt/12 (2005)
www.interact.com.pt/13 (2006)
www.interact.com.pt/14 (2007)

1. Brief description

“live2screen” is a project resulting from the 2006 Autumn term Rapid Mobile Prototyping workshop at Media Lab, University of Art and Design Helsinki. This application provides a means to interact between a mobile phone and a screen. It is a hybrid application with two sides: mobile side and display side. The live2Screen display side has 3 areas: “liveText”, “liveImages” and “liveIcons”. All areas have one thing in common: the user sends information through a mobile phone, which is then used to change animations shown on a large screen.

2. liveText

With liveText, the user writes and sends text messages, and that text information is sent to the server, which adds the message to a pool of past messages. Then Flash retrieves the information, picks a random message from the pool and animates it. When a animation ends, the process starts again. Pressing a key manually changes the message. Random fonts are used for each message, and by pressing a key the font is manually changed.

3. liveImages

liveImages works in a similar way to liveText: the user takes and sends photos, and those images are sent to the server, which adds the image to a pool of past photos. Then Flash retrieves the images, picks random photos from the poll and shows them. The images are shown 2 at a time in a horizontal split screen, and top and bottom halves are refreshed alternatively.

4. liveIcons

liveIcons allows the user to interact through a mobile phone with an animated icon on a screen. His mobile phone shows a representation of the icon on the screen. The user can change properties of the icon on his mobile phone, and the onscreen icon is changed in accordance to that. The properties are: icon shape; size; position on the screen.

5. Purposes and future developments

The application can be used for many purposes, for example: entertainment in public spaces; marketing; instalations; VJing; and performances. In the future, it will be also integrated with audio objects as part of Nuno Correia’s iAVo (interactive audio/visual objects) project.

6. Technical notes

The mobile phones run Python for Nokia S60. The information (text, images, data) is sent to a server, which runs PHP. The software used for the display side is Flash, which retrieves information from a server and creates animations based on that information.

7. Acknowledgements

Thank you to Kalle Mäntsälä for help on PHP programming.
Thank you to André Carrilho for Flash visuals development.
Thank you to Jurgen Scheible for the excelent workshop and Python code.

Images (Flash and Pure Data)

1. Concept and motivation

The concept behind the project is to build a hybrid sound and image composition prototype with a custom, visual appealing and fluid user-interface. Each sound, or component of sound, should have a symbolic on-screen representation. The tools chosen to develop a working prototype were Flash, for the visual side, and Pure Data, for the sound side. The OSC (Open Sound Control) protocol would be used as a “bridge” between the two applications.

Although I have extensive experience in using (and teaching) Flash, and digital sound tools, I had little experience with Pure Data (or similar applications, like Max/MSP) prior to enrolling at Media Lab. Therefore, I attended a series of workshops that would provide the necessary knowledge to build the prototype.

This prototype is directly connected with my research plan, entitled “iAVo – Interactive Audio/Visual Objects: Application for flexible multimedia expression”, since the prototype materializes some of the plan’s key concepts. Therefore, for simplification purposes, I will entitle this prototype “iAVo Gamma” or “iAVo prototype”.

2. Brief description of the project

iAVo Gamma consists of a Flash-based visual side and a Pure Data-based audio side. The visual side is the graphical user interface controlling the audio. Information is sent from Flash to Pure Data through OSC.

There are 3 sound loops that can be controlled by manipulating graphical symbols (icons) in Flash. The manipulations are performed by drag and drop operations in Flash (for position), and pressing the up and down keyboard cursor keys (for size). An “XYZ” logic was implemented, where X is the horizontal position of the icon on the screen; Y the vertical position, and Z size. For this prototype, the following audio/visual connections were established:
- sound 1 (melody loop): X controls volume and Y controls pan;
- sound 2 (bass): Y controls volume;
- sound 3 (rhythm): X controls volume,Y controls filter frequency, and Z controls delay.

Since OSC uses a network connection, several users can interact at once, enabling collaborative composition. Since Flash player is pervasive, any user with Flash player can access the server running Pure Data and use the iAVo Gamma. The audio feedback is provided either by a local speakers system or by audio streaming. A local area network (LAN) connection is preferable, because of audio delay issues over the Internet.

3. Evolution of the project

a) Pure Data for Sound and Image Workshop

The iAVo prototype began being developed in the Pure Data for Sound and Image workshop. There I began exploring Pure Data and OSC. I also began exploring Flosc, an implementation of OSC in Flash. In this workshop, I built a basic Pure Data sound tool that manipulates 3 different audio loops. I also developed the Flash user-interface. Finally, I tried to connect the two (sound tool and user-interface) using OSC, but faced many problems which I could not solve entirely.

b) Composing with Pure Data Workshop

In the Composing with Pure Data workshop, I developed further the Pure Data sound tool and managed to solve the problems concerning OSC. Therefore, by the end of the workshop the prototype was working: the user-interface was controlling and manipulating sound.

c) Teleradio

After having completed a working prototype, I wanted to explore collaborative composition and streaming technologies. Namely, to collaborate using the prototype over the Internet or local area network (LAN). The Teleradio Workshop provided with know-how regarding network communication and streaming media. The prototype was upgraded with collaborative and sound-streaming capabilities. Quick Time Darwin Streaming Server was implemented for this purpose. The delay (audio latency) over an internet connection proved to be too great for a fluid audio streaming feedback. But in a local area network, the delay proved to be very small. A wireless LAN was used, and it would the be ideal type of network for this kind of collaboration.

During the Teleradio workshop, experiments in electronics were also pursued (namely, building a TV video signal transmitter).

4. Possible uses and future developments

iAVo Gamma could be used for audio/visual performances; collaborative composition; interacting with a large screen, or large screens (in a performance or public space context)

To be more operational, however, more content (audio loops and visual icons) should be added.

The audio / visual connection logic (presently based on a visual “XYZ” paradigm) should be reconsidered, since it could be limiting to apply this type of logic to all visual symbols – it could lead to confusing clusters of icons.

The “one sound, one icon” paradigm could also be expanded to a “one sound, one screen, several icons” where each sound would be represented by a screen, and each icon on that screen would represent a given aspect of that sound. For example, the “guitar” screen would have icons representing volume, filter, distortion, delay, etc. Several projections could then be used to convey a “full picture” of the sound manipulation taking place.

There could also be a symbolic representation of the whole sound structure (or “song”): for example, a background animation. Manipulating this symbol would affect the entire song.

The mobile phone should also become a tool for audio/visual composition, with the use of the knowledge I gained from the “Mobile Rapid Prototyping” workshop and “live2screen” project.

5. Technical notes and links

Tools and technologies used for this project were:

Flash:
http://www.adobe.com/products/flash/flashpro/

Pure Data:
http://www.puredata.org/

OSC:
http://www.opensoundcontrol.org/

Flosc:
http://www.benchun.net/flosc/
http://transmote.com/flosc/ (tutorial)
http://www.randominterface.com/forum/ (forum)

Audacity:
http://audacity.sourceforge.net/

QuickTime Darwin Streaming Server:
http://developer.apple.com/opensource/server/streaming/index.html

6. Acknowledgements

Thank you to the teachers: Derek Holzer, Sarah Kolster, Koray Tahiroglu and Adam Hyde.

Thank you to André Carrilho for Flash visuals development

More info: http://www.videojackstudios.com/projects/idiot-prince/

ENTER PROJECT

More info: http://www.videojackstudios.com/projects/heat-seeker/

ENTER PROJECT