We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Rusty instruments

00:00

Formal Metadata

Title
Rusty instruments
Subtitle
Rust when it makes sense
Alternative Title
Rusty instruments: Building Digital Musical Instruments with Rust and friends
Title of Series
Number of Parts
490
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
This talk will introduce the Muses project, which applies programming language theory and practice, physical computing, networking, and musicial theory to design and implementation of Digital Musical Instruments. Rust is a key ingredient in the Muses project, providing a robust and performant foundation for cross platform, desktop, and embedded system development. The talk will give a brief introdution to Muses project as a whole and then focus on the use of Rust in developing a selection of very different components in the system and its benefits for these wildy varing use cases. Demos of the Digitial Musical Instruments with Rust at their heart will shown through out the talk. Controller and gesture interaction with audio and/or visual media is today ubiquitous, requiring the development of intuitive software solutions for interaction design. Designing and building these interfaces often require extensive domain expertise in audio and visual media creation, e.g. the musician, but additionally in engineering and software development. In this talk we focus on custom controller-based interactive systems for sound and musical performance, with a focus on an intuitive and simple design process that is accessible to artists. A large part of the software developed for these systems is low-level system code, where direct access to hardware and understandable performance are hard requirements for these systems. Historically, these systems are written in C/C++ and in the case of embedded systems C is still the language of choice. With the emergence of the system programming language Rust an alternative for developing these systems is now with us with its support for high-level features such as traits, type inference, pattern matching, and of course it's affine based type system for pointers. This talk will introduce the Muses project, which applies programming language theory and practice, physical computing, networking, and musical theory to design and implementation of Digital Musical Instruments. Rust is a key ingredient in the Muses project, providing a robust and performant foundation for cross platform, desktop, and embedded system development. A high-level overview of the schedule is: - Introduction to the Muses project - 100 foot view of the Muses architecture - Experience using Rust for audio and interface development - Demonstration The demonstration will include the following physical components: Custom interface - Hardware encoders + arcade buttons - Sensel touch interface with custom interface - STM32 based embedded hardware platform, all running Rust Raspberry PI for Sound - Pure Data for sound synthesis - Rust based driver for communicating with custom hardware - Rust based Open Sound Control (OSC) server for custom control messages The framework also includes an approach to automatically generating interfaces from a DSL for SVG interfaces, written largely in Haskell, but with a tessellation pipeline written in Rust. However, while this will be mentioned in passing it is not the intention of this talk to cover this aspect of the system in detail. (For more information on this, see the provided link for the project website and associated papers, also linked from the site.) Expected prior knowledge / intended audience Knowledge of programming will be expected and prior use of C/C++, Rust, or other systems programming language would be useful. Audio topics will be introduced through out the talk and it is not expected that audience members have a musical background. Speaker bio Dr Benedict R. Gaster is an Associate Professor at University of West of England, he is the director of the Computer Science Research Centre, which within he also leads the Physical Computing group. He research focuses on the design embedded platforms for musical expression and more generally the IoT, he is the co-founder of Bristol LoRaWAN a low power wide area network for Bristol city, is the technical lead for city wide project on city pollution monitoring for communities, having developed UWE Sense a hardware platform for cheap sensing. Along with his PhD students and in collaboration with UWE's music tech department, is developing a new audio platform based on ARM micro-controllers using the Rust programming language to build faster and more robust sound! Previously Benedict work at Qualcomm and AMD where he was a co-designer on the programming language OpenCL, including the lead developer on AMD's OpenCL compiler. He has a PhD in computer science for his work on type systems for extensible records and variants. He has published extensively, has given numerous presentations, including ones at FOSDEM on Rust and LoRaWAN.
33
35
Thumbnail
23:38
52
Thumbnail
30:38
53
Thumbnail
16:18
65
71
Thumbnail
14:24
72
Thumbnail
18:02
75
Thumbnail
19:35
101
Thumbnail
12:59
106
123
Thumbnail
25:58
146
Thumbnail
47:36
157
Thumbnail
51:32
166
172
Thumbnail
22:49
182
Thumbnail
25:44
186
Thumbnail
40:18
190
195
225
Thumbnail
23:41
273
281
284
Thumbnail
09:08
285
289
Thumbnail
26:03
290
297
Thumbnail
19:29
328
Thumbnail
24:11
379
Thumbnail
20:10
385
Thumbnail
28:37
393
Thumbnail
09:10
430
438
BuildingDigital signalMusical ensembleCodeFocus (optics)Programming languageTheoryPhysical computingComputer networkExecution unitInclusion mapBitIntegrated development environmentMusical ensembleSoftwareStudent's t-testBuildingWebsiteCollaborationismVapor barrierLink (knot theory)CodeMultiplication signProjective planeControl engineeringPhysical computingComputer scienceMoment (mathematics)Materialization (paranormal)Mixed realityDemo (music)Open setInteractive televisionAreaVisual programming languageControl flowUniverse (mathematics)Maxima and minimaComputer programmingMachine visionData miningSource code
Interior (topology)Nachlauf <Strömungsmechanik>Execution unitMultiplication signLaptopTelecommunicationView (database)Chemical equationMusical ensembleRight angleProgramming languageCoefficient of determinationPerspective (visual)Row (database)Point (geometry)FrequencyCodeForm (programming)Standard deviationEvent horizonComputer musicClassical physicsQuicksortComputer animation
WhiteboardComplete metric spaceSoftware protection dongleElectric generatorBuffer overflowData managementOperational amplifierScanning tunneling microscopeAbstractionExistential quantificationFlow separationSoftwareMultiplication signComputing platformDevice driverDemo (music)QuicksortProjective planeBuildingMusical ensembleProgramming languageRevision controlLaptopKeyboard shortcutDifferent (Kate Ryan album)Channel capacityConnected spaceVirtual machineVisual programming languageBlogFeedbackPressureSoftware testing1 (number)WindowTouch typingDigitizingTouchscreenField (computer science)Parameter (computer programming)Modal logicBitComputer hardwareBinary codeComputer programmingKey (cryptography)Mobile appGoodness of fitCodeTDMAType theoryKernel (computing)MetreService (economics)Real numberStudent's t-testReal-time operating systemPunched cardPhysical lawLine (geometry)Boss CorporationStack (abstract data type)Limit (category theory)Execution unitSystem callIntegrated development environmentFactory (trading post)Crash (computing)GodComputer animation
Canadian Light SourceProgramming languageUniformer RaumFeedbackSupersonic speedTexture mappingCodeDegree (graph theory)Point (geometry)Information securityComputer programmingFunctional programmingMultiplication signExpressionExterior algebraMusical ensembleVirtual machineBitWritingFamilyControl engineeringMereologyAbstractionUniverse (mathematics)SoftwareStack (abstract data type)Programming languageExtension (kinesiology)Range (statistics)Fitness functionComputer scienceData managementRoundness (object)Message passingProgrammschleifeStudent's t-testPower (physics)QuicksortTorusType theoryFluid staticsInterpreter (computing)WordPolymorphism (materials science)Data miningPhysical systemMetaprogrammierungAdditionMathematicsDiagramCartesian coordinate systemError messageScripting languageInteractive televisionProblemorientierte ProgrammierspracheWebsiteAssembly languageFeedbackSampling (statistics)Digital divideProgrammer (hardware)Physical lawECosCycle (graph theory)Right angleLine (geometry)MappingPeer-to-peerInsertion lossWhiteboardGraph (mathematics)
FeedbackSupersonic speedTexture mappingTwin primeBlock (periodic table)Computer hardwareArmPoint (geometry)Virtual machineCuboidArithmetic meanProgramming languageRaster graphicsLibrary (computing)NumberPhysical systemCodeOpen setDifferent (Kate Ryan album)Control engineeringProblemorientierte ProgrammierspracheMathematicianSymbolic computationTerm (mathematics)MappingFunctional programmingComputer fileReal-time operating systemMicrocontrollerDevice driverExtension (kinesiology)NeuroinformatikTelecommunicationExpressionMusical ensembleProcess (computing)Computer programmingAverageWritingBitInheritance (object-oriented programming)Group actionPlastikkarteArtistic renderingSpacetimeLine (geometry)Graph (mathematics)Touch typingLevel (video gaming)Moment (mathematics)Capability Maturity ModelSensitivity analysisWindowGame controllerMessage passingSingle-precision floating-point formatMathematicsFitness functionTemplate (C++)Greatest elementMereologyComputing platformRaw image format2 (number)Latent heatEmbedded systemService (economics)GodSampling (statistics)System callDifferential operatorForm (programming)Right angleComputer animation
Physical lawGame controllerInterior (topology)PolygonRepresentation (politics)Compilation albumShape (magazine)TessellationAssembly languageProgramming languageText editorWeb browserComputer programmingShape (magazine)Address spaceTouchscreenLine (geometry)FrequencyPosition operatorMusical ensembleRow (database)Point (geometry)Term (mathematics)Right angleProblemorientierte ProgrammierspracheMappingPhysical systemTriangleWeb 2.0Type theoryGame theoryPhase transitionPrimitive (album)Computer hardwareLevel (video gaming)BitControl engineeringCurveNumberRepresentation (politics)NP-hardStandard deviationMultiplicationTouch typingAssembly languageOperator (mathematics)Multiplication signParameter (computer programming)Pointer (computer programming)Image resolutionCASE <Informatik>Workstation <Musikinstrument>Shared memoryGroup actionOrder (biology)Latent heatGraphics processing unitData typeVideo gameInformation overloadRaster graphicsMessage passingSocial classSet (mathematics)Vector spacePolygon meshExpressionExtension (kinesiology)Square numberSpline (mathematics)MetadataMereologyTesselationComputer animation
Shape (magazine)TessellationAssembly languageBlock (periodic table)VelocityStreaming mediaFunctional programmingProcess (computing)Function (mathematics)Line (geometry)Table (information)CircleGraph coloringSoftware bugProgramming languagePhysical systemShape (magazine)CompilerPoint (geometry)Functional programmingRectangleMusical ensembleTriangleProcess (computing)NumberRoundness (object)MappingBitWeb applicationAssembly languageCartesian coordinate systemSquare numberState observerArtistic renderingSlide ruleProjective planeBuffer solutionQuicksortCuboidPhase transitionCodeSoftwareNichtlineares GleichungssystemRaster graphicsThread (computing)Fiber bundleUniqueness quantificationMereologyDemosceneOrder (biology)AeroelasticityPixelPairwise comparisonRepetitionGeometryElectronic mailing listGradientRight angleRippingSoftware testingSound effectConnected spaceFamilyReal-time operating systemInheritance (object-oriented programming)Web 2.0Interactive televisionPersonal identification numberMedical imaging
Process (computing)Function (mathematics)Line (geometry)Punched cardControl engineeringMultiplication signCodeWebsiteComputer fileThread (computing)Frame problemLoop (music)Process (computing)Slide ruleVariable (mathematics)RecursionImplementationInteractive televisionPlastikkarteSound effectCompilerStrutReal-time operating systemFunctional programmingSingle-precision floating-point formatVideo gameInterior (topology)Flow separationQueue (abstract data type)Point (geometry)NeuroinformatikWorkstation <Musikinstrument>View (database)MereologyCartesian coordinate systemScripting languageService (economics)Inheritance (object-oriented programming)CausalityTask (computing)Computer animation
Universe (mathematics)VideoconferencingLevel (video gaming)Musical ensembleControl systemCodeFamilyMedical imagingInformation securityCartesian coordinate systemBitMereologyComputer animationLecture/Conference
Point cloudFacebookOpen source
Transcript: English(auto-generated)
Hi, um Yes, I'm sorry, I'm not very well actually so I would have brought some demos it's just been a bit of a rough couple of days just got a cold, but I just not feeling the best and I Traveled on Friday. So I'll apologize. I'm from the UK. So it was a pretty miserable day on Friday
I'd like to I think myself a fairly inclusive and open person and I'm yeah sad times at the moment But anyway, we'll move on for that And so I'm gonna talk about and I would have had some demos and played some music for you. But yeah About a project that I've been working on for a while I'm at the University of West of England with some PhD students of mine Some of them musicians and composers some of them are computer scientists and we've kind of have this mix
collaboration group and in particular We've got this music project which look is looking at Go materializations of Musical controllers in particular but also DSP code and everything and stuff Although that's more my area rather than some of my students. They're much more interested in composition and in
interaction so HCI stuff and We've kind of built up this infrastructure that is around physical computing so a lot of stuff some of the stuff I'll show you in pictures is actually building up physical
Instruments, but the idea is rather than the musician having to go and buy something off the shelf or more expensive They want to make them so they could potentially build it now, of course, that's not for everyone I'm not saying that but the idea of trying to breaking down that barrier of having non engineers build build instruments and things and you know We see software like that some things like PD or max MSP for visual programming is supposed to intense like that
but it's still very much in the programming environments rather than in actual physical computing and of course the maker community has broken that down quite a lot for I mean still a lot of people Who are you know tech savvy and things like that? But they're starting to break down those barriers and some things and so we're looking at that intersection really interested and
In a minute, I'll just give you a flick to that This is the website where you can find out lots of stuff. There's the github links with all the code You'll see that it's not just rust code and that's what I will talk a little bit about that in a minute but So, I mean my my motto is over the years. I've worked in industry for a long time
It's just used the right thing to solve the problem Of course what you think is the right thing is not always well other people's things the right thing But that's a separate story and you know I'm no more right than anyone else's so I'm not going to kind of advocate one thing or another I'll just show you the kind of things that I'm interested in and I've been using rust for but in particular I'm really keen on rust as an intersection with other languages
I don't want it to be the solution for everything and I don't want the other things I used to be a solution for everything because I don't think they are So I've just where it makes sense. It's what I do So there's a long history of musical instruments obviously and I'm mostly thinking from a Western perspective I don't at all want to say there aren't many other kind of
ways and Ways of looking from entomological point of view about musical instruments. I just don't have that much experience in it So I wouldn't want to talk about it and present it here but from a Western place, you know We've got the kind of classical periods particularly from kind of around Shakespearean time in my history You know when musical instruments started be particularly since we had the printing press
You know and music became be able to distribute the kind of musical form of the instrument became quite fixed You know the piano didn't emerge first It was the printing press that emerged and the ability to distribute music made wanted to standardize Things and I'm sure we can all agree that standardization is good in some ways But it also does hold us back in other ways, you know, and and so of course we've got to the modern times
Electronics, you know the the 19th century had lots of bad 20th century Sorry had lots of bad things caused by many of the innovations but also hadn't many incredible things and it's I'm sure as we work In technology, it's a kind of difficult Balancing act, you know, sometimes I feel that we should just throw it all away
Not by the next phone or whatever and other times and thinking wow, this is amazing, you know, so anyway in the UK we had this thing called the BBC Radio workshop was where they did the Dalek soundtrack, you know for Doctor Who and all of that sort of stuff Which is famous and so they started to build these studios I even managed to get a picture of a woman rather than a guy which was great
actually that the Radiophonic workshop was a real place where that actually women had quite a lot of very innovative Delia Derbyshire is the famous one But there's a lot of others as well and of course in the 80s We had people starting to break away from the more conventional instruments and finding and being able to make instruments out of other things You know particularly that obviously one is the the record deck and making that into an instrument in its own right and
Today we have many people live coding is a popular thing And we've got this kind of world where people are sitting in front of their laptops and you know I've been to lots of musical events. It's not that exciting to see People sitting in front of their laptops, but the music they make and this the stuff they're using is incredible, right?
You know, I'm not a live coders myself But I love the kind of philosophy behind what they're trying really that's sorry Yes, I think it wouldn't go to sleep when I've got it in full screen in Chrome, but it still does anyway Yeah, I can probably turn it off. I'm not going to worry about that now, but One of the things we're really interested in so I'm really interested in digital using digital
synthesis and digital generation of musical instruments and so how we can connect to these digital devices But in maybe a less conventional way not sitting in front of a laptop But also not sitting in front of a keyboard and so we're interested in building things So there's been a lot of work particular There's this out of Queen Mary in the UK this thing called the Bella board
Which is very much inspired by the Arduino kind of maker environment, but it's got a very high-end You know DSP and you know, particularly the audio the op-amps and all that, you know, they're non cheap You know, it's actually it's a very simple board But you still pay 150 pounds for it because the op-amps they're using are expensive not cheap, you know
And it's a full 24-bit DAC and ADAC and so forth and stuff and things It's really nice little board and it can support it multiplexes the channels So you can have full 96 kilohertz kind of speed But if you want to have 16 channels you get down to like 22 kilohertz depending on what you'll do But it's really nice and it's you know, it's a full Linux stack It's based on a BeagleBone and they modified the kernel so they get real-time and so forth
And of course, it's a Linux stack so we can build and compile using our favorite languages Whatever that may be whether it's C++ or Rust or anything else for that matter They do a lot of stuff with PD which is a visual programming language and things because a lot of musicians aren't Gonna be further than that. So these are kind of conventional interfaces. I won't spend too long on these stuff. Hopefully your things
This is the kind of field the maker field. These are actually all Bella based instruments that people have built Most of these are not, you know, not all of them are engineers Most of them aren't engineers, most of them are musicians who have just ended up programming from a necessity
Looking at some of their code is you know, interesting let's say but this guy up here He's built this a complete retro drum machine He completed it and as far as I know before he started this project took him about two years He'd never had any programming experience at all and he managed to do it You know, he you know stack overflow helped him a lot and something's great talking to him It's amazing project, you know to think that this guy is just a musician
I spent all of his time mostly playing analog Instruments like guitars and pianos and then he just decided he saw a talk on Bella I don't know even though probably a kind of conservatory or someone talked about it He was inspired to build it and it's pretty amazing that we've made that accessible, you know now He first programmed it in PD Then he programmed it in C
So you can imagine the experience he had it was not a good experience and he's got you can find his blog So if you can go off my website, you can find his blogs about it and it's really great to read Yeah, I think he now complains that he didn't have enough time making music but you know, anyway, all right These are the sort of things we are making. I'm not really going to talk about this much time I'm all going to talk about the kind of language side of things and everything, but these are the sort of things
So this is a pressure sensor so that rather than being capacity, you know We have a lot of iPads and stuff and things people doing capacity type But it doesn't have any kind of haptic kind of feedback. And so that's the kind of a real issue so we took this pressure sensor and we've worked out different ways of Generating Musical instruments or interfaces in particular, you know, not very much
I'll show you in a second the separation for it and up here. For example, these are free These are some 3d printed designs that I've designed that they're connectable They can be a horizontal or vertical and they connect together with just Lego pieces But the key thing about it, of course if I brought demo I could have shown you is that they're all based on magnets
So originally I designed them to being using a whole sensor So you could just detect the magnetic field when they move so that but these actually don't have a whole sensor They rely on the pressure and so but the only reason we're using the magnets is to give the haptic feedback So the button pops or the slider you get just a nice little feel of though actually being resistive You know rather than just sticking and this is a little 2d one
Which just drags it centers back in the center if the magnet was in So obviously this picture is taken about the magnet in it But the magnet was there it would be centered and then you push it and so you get it's very simple and just literally Underneath they're just felt and things So the bit that this talk is more about is how do we describe these interfaces in software? Such that they can be materialized in in hardware and in particular
This is a made by a company called sense all in Silicon Valley and I think they make touch pads, you know for things like Mac books and stuff and things and They have a really nice Very, you know, it's a you art they have a little dongle which you can either plug straight into a USB and get a USB connection to it or
They have a little dongle which takes the USB and just turns it into a straight to a you art so then you can connect directly to a external piece of hardware like a embedded STM or anything that supports you know, it's quite trivial So all of the software that we've written that connects to this device Not the stuff that necessarily generates the interface is all written in rust
And so that's what I mean, I'm not going to go into a lot of depth about that today But I mean the punchline which I have at the end is that for me the killer thing about using rust For this project other than I really like the abstractions which is what I'll talk a little bit about in a minute Is that I developed it all on here for about a year and a half
Nothing else didn't test anything anywhere else It was all just driving on here because you know, I don't use anything else So I used some Linux VMs for my students, but to be honest, I quite like my Mac I don't use Xcode just you know, just at the prompt with clang and stuff and We've rust and when I ported it to the Raspberry Pi to Windows
That's the only two platforms so far we haven't done it because sound is not so great on Linux Two days, I think maybe overall with you know, it just it just I really didn't have to do very much You know, there's a few things that didn't work, you know, particularly the audio drivers You have to build by hand and it's all you rusted to point it to it and stuff and things
But it was pretty amazing. I was really impressed and You know the greatest thing for me about rust is is cargo and the package manager is just amazing That being said I went to build the demo yesterday and of course I've installed the latest version of rust for other reasons and nothing builds anymore because of all the packages are out day And so that is still a pain
but it's a lot lot better than using C++ without a package manager and building everything by hand and I mean it's crap. It's still crap and even I was just working with someone just the other day looking at building Helping them with a C++ app that they got we've started to use the Microsoft C++ package manager Which I didn't know about And it would only install 32-bit binaries on the 64-bit Windows machine
So I still haven't you know, it's still not great. Whereas the rust one's amazing So I understand that and I use Haskell a lot. And so I use the one called stack I know people use there's another one called cabal and I'm not going to get into the political arguments of that But it's still pretty amazing. Right and in the way that it puts everything silos here. It's just great I just want whoever I've heard the person's here develops it but thanks because it's really great. So cool
Anyway, so we're gonna go back in history just for a little bit just to sort because there's some more a talk About how I benefit and what I find I like about rust and why I use it So hopefully you can guess where I started programming. I'm quite old now, but you know, I haven't had my BBC micro for a long time, but my brother and I used to fight over it and
he makes films now, so he doesn't do any of this stuff, you know and But I did a lot of this I loved it just typing in code for magazines, you know I'm sure we can all say, you know history and obviously at some point Typing in basic was not enough. And so you could do in-line assembler inside the basic interpreter a pretty cool and
Six five oh two. It's great. You know, this is where I all started it's why I went to university not until I was a bit older, but it was and Of course actually the first programming language I use the universe was lisp Which I still love to this day because of its lovely print, you know braces, but anyway bizarre
But of course we did this I did a lot of this it was a you know computer science and maths degree We had Solaris machines. We did a lot of C, you know and programming and C and I'm not gonna do not I still love C today, you know, it's an amazing language It has lots of flaws and we wouldn't design the same today and thrust I suppose but I still love it Okay, and then I went and left after I finished my piece. Obviously I did some scripting. I'm still mostly do bash
Not really a person. But anyway, I Did my PhD and I did Haskell completely And so this is I just want to come a little bit about this the reason I'm putting this up And I still use Haskell to the day I still love it and You know, it's I suppose the alternative to Python or whatever. I use it for those sorts of things
but of course I worked in industry for 15 years and Yeah, I use this language and you know for a long time. I really thought it was amazing and I really liked it and You know, I'm not gonna deny that, you know, I was a bit naive. I don't know I just you know, there's some good things about it I felt called I could do template metaprogramming and you know outsmart people was
stupid, but you know I did But and I also did some of this stuff. I mostly worked in a graphics group in in MD for 15 years and I was involved with this which has its own things. I'm not going to talk about these things And then of course I discovered NPM. I admit that discovered NPM before I discovered and Kabul and stack
I couldn't find a stack logo stuff at the Kabul one. This is the Haskell package manager and I hadn't done any rust at this point But I was back at university and I was working in this domain and I was pretty frustrated to be honest It's you know, it's a bit like fitting a peg into a round hole, right?
It's a bit apparently this is I actually use this one because apparently this was from a brexit website This is how some people perceive the UK and European Union Think lots of expletives about now and swear words and what you yeah Anyway, but this is a bit like C++ and Haskell rather two things are counted to each other They don't many of the abstractions. I think in Haskell are really important. They're really interesting
I don't want you know, there's lots of extensions and GDA T's if you're at GDA G a DT's which generalized abstract date, which are all great But they're a bit of an intellectual exercise in in showing how smart you are rather than whether they're actually that useful And there's debates about putting them into rust and stuff, but whatever
At the same time Haskell does provide really nice abstractions is very quick to work with just like people tell me about Python I'm not going to make any stances against it. I like the static type system versus dynamic so In 2015, I came to my first ever postum I just back the friend of mine who's still here today a colleague is brought me then with some students and This is where I heard someone talking about rust. I just overheard it in the Mozilla dev room knew nothing about it
And so this has been my journey since then and actually I really really think that I've been able to kind of fit that square peg into a round hole in some but since rust does that really well and that's why I really wanted to say thing and It's not the borrower checker, you know, I'm to be honest. I'm not that interested in the borrower checker
I can see that it's great for security or that but to be honest. We didn't have it. I'd still use rust I don't really care. I've spent three years actually learning how it all works and fighting it and reading papers to understand it So that was a pain, but I don't really care that much about it to be honest But I admit that I don't write software for many people of myself So it probably doesn't matter that much to me, you know where I can see that it's great applications. It's really useful
The thing I would say and I wanted to make clear about that is that it's the thing that means I can't teach rust to first years Don't in my university. I think it's just too much. I can't teach rust to first years because of It's a lot to take in
Particularly if you haven't had a lot of computer science and programming experience and most of your experiences in Python or something I personally feel that it's too much. You know people they find it hard my students find it hard They don't even read the error messages, you know Trust trying to get them to look in if they start to read those complicated, you know Be right reading the messages template metaprogramming. They're not for the faint-hearted and they're you know, it's not for the faint-hearted
I mean, it's just a comment and so I don't think I use the borrow checker to anywhere the kind of Power that you could use it and it's just because it's not interested the end the abstractions I'm interested in are things like traits and the you know the parametric polymorphism that you can do over those traits and things like that and build it up the
The kind of additions of kind of you know, obviously the range loops but moving those more into having nice Where you can return things and get the nice build-ups of Functional kind of style expressions and things so that for me has been the bit that was missing from C++ They course they're putting it in Well, you know it's like this again, isn't it?
They're putting it into C++ some somehow and I find myself writing more C code than I do C++ now because of of that It's just so complicated and it doesn't really seem to fit that well Anyway, so that's why I was gonna that's an aside. I want to talk about it. So I ended up using rust so back to the main part of the talk and so here is
A diagram that comes from this guy called Thor Magnussen. Who's a famous musicologist stroke computer scientist stroke musician Who's a professor at Sussex University in the UK and he describes an instrument? To Look something like this. So you've got this kind of the sound engines here the sound production things
you've got obviously you've got your synthesis whatever over here, whatever is creating the sound and And you've got this gestural control Here, which is the interface that I showed you the bit where you're performing interaction with the instrument itself And obviously there's feedback. There's audio feedback coming from here and there's hopefully haptic feedback coming from the interface itself and
Really I've kind of completely separate. I don't really think they are separate I'm not trying to argue they're separate but it's much easier I just like discretizing examples and it's a if we separate those two things out and a lot of my work has been looking at
that Dividing that line and of course the mapping then becomes a serious problem and it's not one that's that easily solved at this point So you'll see I have a whole and I'll talk about just at the end a whole rust based system Which is purely on this side of the world Which is just doing synthesis DSP as he integrates this language called Faust, which is a purely functional language
For DSP just it allows you to write DSP graphs in a functional expression. It's just a very mathematical. It's really nice we've got this little what of it generates C++ modified it so it generates rust now and It's just you know generates a effectively a trait with some simple methods for initializing it and then a function to ask for
Number of samples, it's not Particularly difficult all the smarts is done in the files compiler, which I don't have anything to do with They're all based in France and super smart mathematicians Working at a conservatory, which is kind of slightly odd Anyway, but and a lot of this here has been building physical interfaces
But using rust to interact with the physical world Okay, and I'm not going to talk about it today, but we've also got an embedded platform, which is all hardware Which is based on STM or arm based microcontrollers mostly m4s and stuff that we use to do to connect to the physical world for actuation and sensing and
I Did develop some of that initially started doing name rust I started working on that writing, you know from bare machine writing it up But I gotta be honest after about a year I kind of gave up and I've just write it all in C now just because of the code is really simple it's not very complicated and
We use ST chips and they provide reasonably good libraries and so There was just a lot of work and I'm know some brilliant. There's some brilliant work going on And I guess when that's got to maturity That'll be great Just I can just start off and I won't have to do all that work of getting the registers the map and all that So I you know, I'm gonna put my hands up here
I wish I hadn't had to go back but I chose just for practicality reasons to go back to using C for for speed of use and I think that's really my message about My use of rust is that where it's been amazing is that you know, you've got a credible ecosystem even though it has not that old and It works mostly across most platforms. I've tried and you know, it can be packaged up
It's really great and that is a bit more tricky on the embedded world today Okay, so I probably won't talk too much about this but if you imagine that we've got on interface like this, so I've got this whole system where you describe them in terms of Extended SVGs and you can you can draw them or you can write them in a little domain-specific language Which I'll show you in a second and you can mark them up to say whether these are generating
What kind of gesture you can do whether they're generating something called MIDI or OSC which is open sound control Just a way of communicating standard to communications and you take these SVGs and then you run it through this system Up here. So this is it's a domain-specific language
It's actually an embedded domain-specific language, although you wouldn't know because I've embedded it in Haskell So this is the point where you can see the different things with rust coming in. I think using it where it makes best One of the things I found limiting at the moment well, one of the things that Haskell is incredibly expressive with that you can embed a
Domain-specific language in Haskell and you and kind of rework the syntax, you know, it's not racket So you can't completely rewrite the syntax but using this thing called template Haskell which brings in template metaprogramming We won't go there, but it's quite nice of being able to define your own it. No own Syntax not completely you have to borrow some Haskell syntax and rust
I've still found that because of it still looks see ish in some ways You still end up with a syntax that looks quite programming language II even though you can definitely embed it so I think that's a trade-off and that's why that top one up there is Haskell rather than Rust and you know, it's not performance is not critical at that top one. It's not doing anything
This one down here is doing path tracing and Rasterization, it's I'll show you the pipeline in a minute. So performance is is important here and it wouldn't be so simple in Haskell I mean, it's still offline. So we're not doing real-time rendering or anything like that And so you could probably get away with writing in Haskell, but just didn't feel natural to me much better fit with
Rust and there's also this amazing library called lion or Leon. Yeah lion I think it is which is a 2d path rendering library, which I've modified slightly, but mostly I've used it out the box it's just a wonderful library developed by this guy Nicholas and And again, depending where we're going This is all offline and generating this this generates things for it's called a roly light pad
Which I'll show a picture of a minute, which is a little embedded Controller which has a touch sensitive interface with 15 by 15 LEDs and you can upload little C programs onto it They're not C, but they see like it's like a little embedded language that you can upload onto it
So here we're actually generating code, you know, just text files that are then uploaded onto their thing So we're not doing any real-time processing or handling. So speed again is not the critical thing symbolic Processing of what's outputted here, which is I'll show the JSON file in a second over here, though This is the thing that connects to the touch sensor that I was talking about
This is real time. So this is basically a driver, you know, it's drivers talking with a very low-level USB driver And this is doing all the mapping from the rasterize eight well, I'll show you what I mean by rasterizing the second but effectively it's rasterization and Here so performance is critical because you know We're actually this is running live with a musician playing it all the processing of when you touch something
Mapping it to the semantic meaning of you know, are you in the right place on the sensor? What does it mean? What's the action you want to perform is all happening in real time and effectively? Hopefully you all know that USB USB Jitter is pretty awful on particularly on Windows and you know plus or minus three milliseconds is good and
Even on the Mac, it's around two milliseconds, but it doesn't jitter which is quite nice And and then you've got the latency of evolved as well. I've taken that and generally on the Mac actually It's around recent upgrades around seven milliseconds on average without jitter, which is pretty good
They say that you want to be between you know But when you perform an action to hearing a sound you kind of want to be within ten milliseconds But if you think that it's taking seven milliseconds to guarantee that we're going to get thing You haven't got that much space to do any a computation on the other side, you know, you're still doing synthesis I mean obviously you're creating more than the same. So this is performance was really important here
And initially I started in C++ and it was just you know C++ and you know I'm fortunate enough to be in a research group and so I get to explore things and play around with stuff And so and I've been to Fosham and so rust is where we are here and it worked. It's work brilliant Now, you know as I say, it's that whole stack Has just been great
Okay, so I won't go into a lot of languages here But just to give you an idea These are the kind of things that people write in the language if we actually have a visual editor as well so you can just use it in the web browser and draw but I'm quite interested in what you can't do in a visual editor particularly program Programmatic You know expression
So for example, I've got this little example where you have an interface of sliders that is drawn in the sine wave And that would be pretty hard to do I mean you could do it in something like illustrator or in our little thing But you wouldn't be exactly following the sine wave. You'd probably be off and you know, it'll be a pain Maybe you could be use a b-spline or something but and so we have a little language that you can write this is like a stop button and
You know, it's got size it's a square one for this case and You could have something like a pad and then you can start talking about The kind of address this so this is the an LSC address. So this is effectively the message I'm doing all right for time on it. Yeah Okay, so and I'm not going to go into a lot of details about this But you can compose them together and build up more complex interfaces. I mean, it's not rocket science, you know, it's the standard stuff
This does you know? Use some stuff in Haskell like dependent types and stuff to make this all kind of fall out And it uses this special extended use dependent type. We can overload the syntax which gives us these special operators which allow us to have
Arguments that aren't ordered, you know, you can actually write these arguments in any way And so there's it. I mean, it's more of a if you're interested in type systems it's quite fun, but it doesn't really matter to the user you can build up interfaces and There's a little Domain specific controllers. What's interesting here is that we have a representation these all just map
To standard SVG types, right, you know, that's a really nice thing about SVG's is that of course They're just XML and if you extend it Illustrator will still display or your web browser will sit and it will just ignore any metadata that it doesn't care about I mean, it's great, you know, and obviously nothing to do with me people doing this for years
It's just really handy particularly because this is kind of a graphical thing that users can see their interfaces and things like that so it works quite well and Obviously we have things like this it Well, I didn't realize when I started off doing it I should mention that I mean this is nothing to do with rust or anything Was that initially I just used these three standard?
SVG types which aren't using Paths right they're not doing B splines or anything like that or any kind of thing but it and it's really easy to map if you imagine or particularly if you know anything about rasterization and Desolation once you take this shape you triangulize it You can then rasterize it into effectively an array and you just get a bitmap of where people touch right because it gives you a kind
Of mapping from a position on the on the screen to a point in an array And once you've got a point in the right course You've got a pointer to a class and you know or whatever a data type and then you can perform an action and you get Effectively or the one look up, you know for your touch which is really important, but it turns out
That actually because we're doing tessellation which I'll show you what that means people who don't know what that means a minute Is that you can take arbitrary paths and so people could start to draw weird, you know, they're still vector drawing So they're not completely weird shapes for their interfaces That all works brilliantly the course the difficult part is then to recognize gestures because you know
If you've got some weird curve like this How do they want to interact with it? Is it following the path or do they want to interact it with you know, multiple fingers things like that? And so I still haven't solved that problem yet I'm on we're working on that and I I've kind of got a bit of a mapping but it's still something to work out So this is the SVG that's created, you know, we won't worry about that
so this is where rust really plays its key role in this pipeline is that so I Don't know how many people are familiar with the graphics pipeline Yeah, oh so quite a few is great So hopefully you'll realize that this is effectively a graphics pipeline, you know, my backgrounds in graphics And so I do this So effectively we take these shapes up here is what and what you do if you're rendering a 3d game or a 2d game
Particularly if you're going to run it on a GPU is you take some primitives some shapes up there and they're normally they'll be effectively they'll be described in terms of Triangles so this phase would have already happened You'd have a mesh and you just you just render that to the GPU and if you haven't you'll do this phase
So this phase is basically shape tessellation, which basically takes an arbitrary shape I mean, it can't be completely arbitrary But as long as it's described mathematically either as a B-spline or as a standard geometric shape like that and you divide it into triangles Okay, so modern GPUs of course support this actually in hardware as well
So you can do normally you'd send it a triangle and then maybe you divide that into more triangles So you can increase the resolution without having to send too many triangles in the first place, but we're not interested in that What we really do is we have this shape assembly phase. We label it with a number Here because we want to have a unique ID when it comes into the system itself
We tessellate it and this is where lion comes in. So I'll just give you the little thing here He's done all the work hard work for this. There's some really smart way This is all the stuff they're doing for the web GPU stuff and things like that and related You know in the five, I don't know if he works for Mozilla But quite a few of the Mozilla team are working on this and related things so we tessellate we generate basically a set of triangles and
Then you have this phase of rasterization So this is the bit in the GPU where they're actually generating the pixels, you know They've gone from these geometric shapes effectively a triangle and they generate a bunch of colors that sit on the thing the observation here is That we don't really care about the colors because all we want to do is generate a lookup table
From the thing so we just we just write out our number that we generated in that shape Assembly and I'm using shape assembly because they call it primitive assembly in the Graphics pipeline so it's just a bit of a play on that. It's just really assigning a unique number okay, and so you can see that we've just generated this lookup table and clearly this will be the size of our
Device that we want to target whatever that might be. Okay, so I'll move on quickly This is what it might look like when it's tessellating obviously tessellating Squares and rectangles or rectangles and triangles with trivial. It only becomes complicated when you're doing
Non square shapes or rectangle shapes as he circles pretty straightforward It only really becomes complicated if you have a squiggly thing like that because you might have a you want to draw a line And it will go outside of the thing and so you have to start chopping that up and that that's where it becomes complicated That's where a few bugs have come up in line
Where they miss a point and you end up with a fat line rather than the thin line things like that but anyway, that's most of that's been resolved now and This is literally what happens, you know with your thing and if we are rasterizing it This is what you look like, you know, you've gone When you think about the size of the sensor, so this is you know, it looks like an 8-bit retro image, you know
But it's like that. Okay All right So then finally we get to the end bit. We're just really That that would have been a nice bit of Jason which would have shown you the
well, I Have to fix that another that but up here you can see there's the buffer up here So this is the buffer which is that is the one-to-one mapping with the it's the array basically the maps that's been rasterized for all possible shapes and so all you'll see is a a 250 by 180 array of
Shape ID numbers so, you know if it was a circle it would have five in it if it was going round and things like that so there is one drawback with this whole system and which I will mention and Is that hopefully you can see here. I'm not going to have aliased all these points Okay, because this is nicely aliased here And so you kind of get this when you look at it from far away
It will look like it's nice and round I have to make a decide whether this is the in point and this is the out and So it's potential if you're doing very Fine grained interfaces, you might get behaviors that you're not expecting So and I don't really have a good solution for that. Anyway, I'll move on quickly because I haven't got that long This is what these two interfaces look like. This is the very light pad this little company from London. We make these
Overpriced box. So finally, I'm just going to finish how long I got left about five minutes. Yeah, cool So this is a little application I've developed it's completely written in rust about 25,000 lines of rust it's a hundred percent rust and
It doesn't build anymore because I upgraded the compiler. But anyway, but it's actually but it's really nice So it uses that line. So it's all 2d based rendering path rendering to generate all the graphics Step sequences it uses the simple threading library, you know the cues and everything and but and it runs
Really really well the performance has been amazing on my Mac and something really great because you know it was a question when we started this project whether rust was quite ready for doing real-time audio and I know a few people from Ableton have been exploring it their big audio company, but it's just been great and it's been a real pleasure to write it and and Everything and what's been really nice and I'll just finish up on this
There's this group as I said In France from Paris in France who have developed this functional Programming for DSP programming language for sound synthesis and audio processing and it's a really nice. It's very mathematical So if you're interested if you're good at DSP and understand it
It's brilliant place to start because you're almost writing the DSP equations and stuff and things, you know and it has delay lines and all that sort of stuff and as I said that they You write something that looks a bit like this and so very simple code if the stupid thing wasn't wrong You can see that there's a simple little oscillator. You can set the frequency and
The only thing that I don't like about it Is that they've had kind of assumed that you have this little kind of web-based interface or little gooeys that they generate for interacting With it and so that's okay And there's ways of linking that with C++ software or whatever Which if I went to the next slide and it was working well, you'd be able to see the C++ code But I don't know why it's done that. Sorry about that
But you can see that if I can show yeah up here So this one here it generates these codes and that comes from the fact Of course, I just want to go back one slide That they have this H slider up here which describes it a kind of an interaction with the outside world You know, it's a side effect effectively, but it's all functional And so they need a way of connecting that and to do that
They introduce these variables when they generate the C++ code or and when I generate the rust code you generate something as well And then they have a way that you have to have like this UI that you pass in which is the thing that's controlling it which actually has been a real pain for us because When you build up an audio application, that's not really how you want to pass around your day
Do you really want to separate it as you know? The UI wants to run in a completely separate thread that isn't real time dependent Whereas the audio thread has got to be you know, real time doesn't want to be and so it can miss control signals Because it might not look at the queue because of time things and everything
Whereas this is kind of all got it fixed in so what I ended up having to do was Sorry about that, it's really annoying, but basically if I slide over to here You'll see that there's these functions here. So when I generate the the rust code you basically describe a JSON file which describes the
The site the variables that you want to change and then it generates the rust code looking at that and will generate just a bunch of Setters and getters. I mean fairly straightforward. There's nothing smart about it. It's really very simple code It just has a simple strut plus a simple implementation. The implementation is initialize the audio process
that you know the Synthesizer or whatever variables need to be initialized and then it has a compute function that you call For how many frames you want? So maybe you've got a 64 frames that you need and you just call it and it is a simple loop The really nice thing about the fast compiler is because it's the way it compiles is that there's only a single outer loop
There's no nested even though you might write it with recursion inside it It can propagate it all the way to the top So you end up with a single loop and so it's incredibly performant and of course You know LLVM can rip through it and optimize it really well. There's no aliasing. There's nothing So it just the code it generates is great. And so it's a really nice interaction and
It's benefited that rust was written in LLVM because I now just generate from LLVM because the files goes to LLVM as well so that's been really great and You know another great thing about rust is that it chose to use LLVM which I love, you know, GCC is the haunt of my life and Anyway, so that's it really just wanted to say and I just yeah the I suppose the punchline
I already gave it to you was that It was really brilliant. I just the package that's the thing for me about it. Everything else. I love in rust I've used it all the time, but cargo just it's just been the best thing, you know an NPM, you know JavaScript hey, I still quite like JavaScript. I can't deny it. Anyway, that's it
Okay, yes
So
The question is that in rust there are Some other work going on with particularly in video and audio and image based control systems Is there an overlap with this work? I guess there probably is I must admit I'm not that familiar with them. And so but I don't see any reason why there wouldn't be I don't I The some novel work in this but the rust part of it probably isn't that novel other than I think it's a great
Application for rust, you know, we wanted performance But I really care about having some of those high-level abstractions and that for me is the main That's why I would encourage people to use rust again. I know a lot of People and I hear people at my university particularly people in the security groups talking about rust because Microsoft's now talking about it
You know and I absolutely that's great. I completely see that it's just I fall asleep when they start talking about it So it's just not me, but that's just a personal thing and not a criticism of rust at all when the borage I mean, it's amazing. It's just not my interest. So but yeah Okay
Yeah, if my cold actually I feel a bit better now, I think just cause it's wet on set good but yes, yeah, definitely