Baroque Dreams Live Multimedia Performance Interpretive Culture and Open Source Software

and I'm very through I'm from San Francisco

and what I'm looking for here today is how and to generate ideas with people about some new types of software performance tools compositional tools that people can use for for real-time interactive art and I come here by way of a few organizations was being foundation were

California nonprofit and that uh develops tools and and analyzes their use for music you using the esthetics of the 21st century and I work with some organizations like a common Media labs

which is which is a organization doing spatial media synthesis and technician for this organization and we do a lot of in surround sound and video production 3 surrounded sound with multiple immersive speakers in some more work some commercial things this

is 18 foot multiuser multi-touch wall that's in the hard rock cut hotel was because and it displays a number of rock memorabilia that can be thrown down the wall across the room and displays information you interact with it up to 6 users this whole resolution girl for came back projectors 4 of them so I believe it's largest multitude usual in existence of some military also do

some work with and and their organization and sentences sisters organization called the gray area Foundation for the Arts I work with and and we do some sort of educational and a gallery types of situations and so everything I deal with this technology for digital

arts and so involved with a lot of system design a lot of working with individual artists in museums and to create the ability to show these forward-thinking words and in particular I'm interested in octets immersive and interactive both type works to create certain types of cultural you know and cultural capital and experience economies and words that I engage multiple for performers audience members and those types of things but the question that really happened dealing with and focusing on lately is the the idea of real-time hot real-time performance and the interpretation of existing works by performers and the types of tools that are needed to to accomplish those types of things and as a warning on the talk a lot about music here

the reasons being is that music is kind of that although we're converging and every year almost performance artistic work has some sort of a visual element these days and music was far ahead and that it was the 1st time based medium and and it's always been interactive um yeah in the sense that from time immemorial people gathered together to make music and and Due to its nature it has to be present at the time has passed for music that happen and and so it's always kind of been real time interval this century and summing over here some of the problems with this real

time interactive art with obstacles need to be overcome and talk a little bit about what the myself and some my colleagues are doing the current projects were trying to work on and then hopefully have some time to get into a little bit of the tools discussion and and talk about some of the things that are available some of the things that are available that we needed in the way open source type of things can be involved in and so without further ado will get right into the problems 1 of the problems is that there is a lack of interactivity that's come out in the 20th century in R and interactivity is had this big buzzword in media conferences these days but it's easy to

forget that it's not something that we're gaining something that we lost that we're trying to get back and all kinds of are interactive until we started using technology through its limitations to interactivity away from us and you know this technological innovations in the 20th century a century reduced the the opportunities musical interactions in several ways and it did so by kind of of allowing in a certain sense different interested parties in the production of a work to not have to interact with each other anymore to get the work done for example recording technology and whether we're talking about new video or audio and takes the audience and composer frees them from having to be in the same room with each other which you know fundamentally changes the way does music sound you videos experienced Multi-Tech track tape recording technologies made it so performers didn't have to be in the same room to make attracted to that were in the same time and you have things like performers recording and over older works in some those effects of reverberation and and like user get performances is now even Freeze slide performances from having to do everything in real time so there are a lot of works where you know even with pop music through it let's thinking etc. that so we've got away from a lot of his real-time alive situation I'm not going why this important will later on but you know in the last 25 years we have gotten higher sampling rates you know more distribution methods and more accessible distribution methods and you know the means to use an increasing number of effects in the palace and but there's been very little progress in terms of like performance of the ability to gage enlivens that library interpretation of electronic works is something that I it is vital important culture and through some sort of obsolescence is of technology in their their inability to transfer works to the medium has largely been lost the so I'm going divine kind

3 things that have to be overcome and to freeze apart from these machines and it's kind work noting that while if these are at the core technological problems the been attempts students you know very early on and certainly the compositional world to solve them and traditionally yeah by composing techniques and extended playing techniques of augmented the abilities of traditional orchestra instruments and the use of improvisational elements led to a blurring of the line between composers and performers and even indeterminate saying and you know with composers like age led to a larger role for work for the audience in kind of feeling encompasses compositional structure and so the 1st

problem is can the opening of the palate the filling in of the palate and to kind of any any most of these better for things I talk about going to apply equally well to graphics and the way in here when you guys catch up of and this is largely a matter of this computational power to make to make that happen in a quote by we were so low of the year of the 18th century man could never have endured the discord intensity of certain cost produced by our orchestras whose members have trouble in number since then for years on the other hand this unpleasant sensory hearing has already been educated by modern life so teeming with the irrigated noises years are not satisfied nearly with this and abundance of acoustic emotions so this this something of the the palate really gives us more metaphors with which to compose works and speech or culture and and while digital technologies have done a lot to bridge this gap that also led to culture of kind pre-recorded loops tract-based works and even this empirically looking at you know you go cdga here Bj and there's definitely in a single performer there kind of looking like this which it is certainly not theatrically engaging and role you know much has been gaining in terms of voices and the right of texture accuracy and through composed composition much has also been lost in the realm of human interaction performance and the kind of better answered digital technologies for why these are problems is because and it's difficult to address this or even the 2 other problems on the talk about the 2nd being giving compositional access to performers during real-time performance and in this sense we're talking about really bringing in Proposition back into electronic works through real-time data and metadata were manipulation and giving a performer the ability to and fixed elements of the composition that can be filled in in real time and the 3rd thing is creating standard for storing and these 3 problems and talk about the kind of progressively harder as we go because the reason works and works like music the classic is because a they can be picked up by by any ensemble they're not arbitrary point in the future and that they for example why block pieces classic is that I can pick it up anywhere in the world and play it into a reinterpretation of that existing cultural work and and this is something that we're electronic works which has been the majority of music created last you know 50 60 years and this fairly impossible because it cannot be because of the that the closed systems with which it's created but so we get back to

technologies and search the 21st century and this kind of can all be summed up in the fact that there is a

divide between you know these types of metaphor machines and information processing machines that we have

an obviously this is a loaded example because of the company that certainly has mixed messages in terms of its openness and and people like this is the Kronos Quartet from sentences scope

which is also convoluted example because you know there was the mirror new music performance that string quartets in the world and but also these kind technology In the gimmicky ways a lot more can do throw off he said it the technology is not really integrated with their performance performance structure and again I entitled my talk broke dreams not just because

I'm hilarious but also because but that I am people might feel that they have a certain affinity for the Baroque because new technology creates

new art which is very much what I'm interested in around for a couple reasons 1 is that obviously technology a metaphor for ourselves it's the only way we have for new interacting with the environment and and just as modes of human existence change over time so were modes of perception so when we talk about you know the kind of art that we're making our culture it's the silly at best to think that all types of technology can create the metaphors of so relevant to us and the other course technology that ideas and cultures and in the duration of the choices that are available to you as in you know the creation the instruments and things like median event based sampling certainly have the created an entirely new John and cultural movements and so if you look at the timeline of instrument development as a 1200 you had you know notation people like there was a marked we do here is that of the Guidoni in hand was a huge you know vocabulary and no musical movement the 16 hundred you had a lot of human instruments for the Baroque period and the violin being technologically change to be louder and carry across an entire ensemble took music out of chamber in the chamber setting and brought into can orchestral situations setting hundreds of pianist equal temperament allowed works to be cross transpose across keys and then you have no that letter development instruments with a vengeance Sextons Jan certainly electric top and that's the great thing about you the interesting thing about all this musical development is that they created new new genres but which is something that largely seems to have become lost in the 20th century we have this thing then certainly if not in that they slowing down of styles and John production and if you even little thought experiment and think of something like that you know temp world versus cultural distance of the big band The Beatles and then say nearby to the music that's being made now if you contained to pop music and a lot happened in the previous time period and not a lot has happened genre lies in the laughter and so the question is kind of why is there is not a new generation 2 style anymore which certainly used to be the case there's been sort of Ajanta stagnation and we have a word with problems and you know the the big big cultural events such is you know a giant oil spill or new diseases for you the threat of nuclear war arts pontine artistic music that movements in the same in way that they used to and certainly world war 2 world war 1 the major cultural differences had corresponding a prohibition had corresponding had to be styles that went along with this American Institute has been remixed the which

I would argue the appropriate collage only get you so far now because that's kind of what we've been doing the last

40 years the 1st digital sampler was in 1969 and for sure Pierre Schaeffer not RDF have been doing things the way before that and so again why we get back to society reinterpreted culture of the ability to pick up a work and not a player replayed in real time and 1 of these problems with remakes being that it's largely an offline process in the sense that you know you go and dicha I'm gamut of materials a recompose them into some sort of a satirical or an interpretive work but it's certainly can't be done with current technologies easily in a real time so if you buy my argument which then you know now what we do and so

here's what my and my colleagues encountered during but were trying to get these guys into their when we're trying to do it in

a way that's that's transparent for them so it's largely a question of interface but not novel

interfaces and again we want transparent interfaces and would we have we been at the being Foundation have a and in this sense is a developed a group of technologies to leverage the years of virtuosity of these kind of performers and bring them into the computer so our can 1st technology to do this is called the

cable and that kind founder of the this foundation whom I work with is named Keith McMillen hence the k a and it's a sincere

vote for the string instruments and that takes the performance data from your plane and since it by a

Bluetooth connection to the computer and then it started with a number of sensors of such as 3 three-axis accelerometer there's a grip sensor hair pressures since the distance from the building the bridges since the distance along the bow lengthwise since twist until you get jet gestures mappings articulations of and it's built this stage so it's like this is not like encumbering device there's no way different from the traditional below and its robustness and he can break it again built the stage and his friend entire string quartet and which takes care of our gestural and kind of performance data on on on the on the gesture side and then another protocol the string work

which is a little box that looks like that but which

through a certain type of interface that is the spectral information in the actual audio data from

playing them and so you get a group of information like loudness brightness noisiness even on harmonic balance and the point of all this is

that it's basically metadata about the performance that's being used to go control the composition in real time from the performance but are prototype ensemble to do this is cultural metric and and we use several

performances with light video integrated and you can see everybody's looking at a computer screen so they can see counts the information from the composition of the plane through and all this but this is a guitar violin and AT base ensemble in all of this information is being gathered is network through a central computer

system and the before I mentioned the kind metaphors of our century and I would say that this would be a web metaphor obviously in our age and so I think than that networking together this information is kind of a vital step in the creation of works that can speak a little better to our time we've them things like control robots more novel

funny things like that and dance around and so after this interface is completed everything turns into a software problem which gets me

to about tools and and in the

realm of music there have been several the tools that have been available for the production of sound and from a computer and there are largely programming languages such as C sound supercollider chat with supercolliders a small talk based language charges at that and a set of Princeton it was developed and it's it's a language with them very strict timing and it's useful for music and in their can and some more things in the video realm of like processing and open frameworks processing and developers teaching language and in open frameworks is another that is similar to process and if you're familiar with that and run it it's based on C and it's supposed to each other kind of the problem with some of these tools is that they reinforce this idea of the artist programmer and which is there's nothing wrong with the prosaic except that have the idea that the same person that creates the on that work has to deal with all the technological infrastructure of the work is well can be useful for many artists in a large scale is probably a bit naive and you instead of saving work in dark current age technology can it's just allowed everyone to do their own work and so we have this kind multiplication of labor and division of labor and and so what we want you with our kind of software is to keep this separation at this specialization a little bit so you know uh kind modal software that it can be useful to the programmer when programming the composer when they make the peace in a different set of ways in when the performer place they don't really have to worry about the programming or the compositional process when they played the piece and which gives you a division of labor In this specialization turns in kind of a force multiplier which allows each of these people 2 and through their specialized roles such as performers it's been 20 years building virtuosity on their instruments and leverage that Towards the final construction of the work instead of having to dilute the efforts being a programmer technologists and then of course in this kind of triumph right here eventually and you know even before now we have that audience and so what I'm looking for is to make

platforms for these works as not just tools that can be 1 of works it's not enough just to have a framework that can create a work you have to have a platform that can support multiple works and create kind ecosystem of things and with scoring etc. 2 to to support this type of thing and a tool a lot to to local maxima speedy it's a visual pattern language and there are a lot of variants uh such as a program called touch desire from a chemical derivative Toronto is the apple sports composer there's a kind of a program called the dual from from close which is also in Canada and and the obvious derivative mean PDD which is an open source version of Maximus be that have unfortunately it lacks and features and is in certain ways undeveloped enough that it's not useful for situations where works have to be developed quickly and with limited budgets and teams as so I guess 1 of my please here is that I could talk to people PDE for example it has absolutely no GPU leveraging for its graphics and it's kind of falling behind in that respect and but do you know about the next step where these things are going our going to be the cloud and so it might be useful to jump over P a little bit and go to some of the new emerging technologies 1 of which just kind of came out today and let me show you some of us have

1st right time so that emissions some the

software that the that I work on this is the software that comes with a bow the money's go over real quick to help you can

understand what I'm talking about the the the things for the they settings for your composition is stored in this hierarchy of presets this precept can control the other preset through the entire program and ended for example here is a have standard effects wrap it would happen in

a musical system Parametric EQ and obviously

not have the whole Bowen system here but what you do is kind of opening these mapping screens and choose

a source which there are many of so such as X accelerometer from the bow and then but

destination such as I took the loading on my

Q and when the data came in you can scale

and table and kind of mashup into a form that's artistically useful for you and when the data comes in you can see that you know and the parameters will change real times space and so you can't cure rate a um the space of possibility is that the performance you can create trajectory through and from piping your audience through system like this can create composition that both has features that are the same and features that are played the better and text and

in real time is 1 that doesn't have a smiley face on a

little and so on so unhappy who want to see note

history off because this reminds me that I'm not but

but but I don't have time to get to the web-based patchy language all alternatives come if anybody's interested in that the final show some them to you and if you don't know visual matching languages like Max pressure come and talk and undertaken by 6 you're talking about 1 of my favorite new pieces of open source software the CTO there is here in Brussels but is called NewsFr the which is an

open-source storing program to compete with things like finale instabilities and other ones that are out there and embassy on a final note I would say that I a lot of these programming usage metaphors that I'm trying to get out of the software or what are going to apply to our graphics systems equally well and so a lot of what I would like to talk to everybody about here is ways in which the Scangraphic systems and real-time graphics systems can be developed to bringing the visual part of this puzzle into place and so I guess that's it thank you very much