WebMIDI
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46873 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020182 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
Slide ruleCommunications protocolMIDIGoodness of fitMereologySlide ruleBitLevel (video gaming)HypermediaMIDIWeb 2.0Communications protocolKeyboard shortcutComputer animation
00:38
Slide ruleSoftware developerGame theoryEmulatorSlide ruleTheoremFigurate numberTouch typingProcess (computing)Software developerSign (mathematics)Term (mathematics)MIDIWebsiteTheoryComputer animation
01:25
MIDIFile formatComputer hardwareCommunications protocolSample (statistics)Message passingVolumeFunction (mathematics)Junction (traffic)Web browseroutputMIDIComputer hardwareFile formatWeb 2.0Standard deviationCommunications protocolCodeSequenceData storage deviceMessage passingNeuroinformatikLogic synthesisTrailForm (programming)Volume (thermodynamics)Latent heatWeb browserQuicksortOcean currentBitMereologyDecision theoryHexagonNumberRevision controlMultiplication signSerial portSingle-precision floating-point formatComputer programmingWordPoint (geometry)Video game consoleNavigationVideo gameArithmetic meanMultiplicationVirtual machineMatching (graph theory)Public key certificateComputer clusterMathematicsComputer animation
04:57
MIDIHypermediaMultiplication signJava appletScripting languageOrder (biology)Video gameWindowMessage passingWeb browserMIDIDemo (music)Particle systemInformationBit rateVolume (thermodynamics)Web 2.0Context awarenessCommunications protocolGreatest elementBitComputer animation
06:19
Message passingMIDIParameter (computer programming)MathematicsPressureRange (statistics)Meta elementEvent horizonType theoryChord (peer-to-peer)Analytic continuationBinary codePoint (geometry)MereologyOperator (mathematics)Electronic mailing listMIDILatent heatNumberRevision controlLibrary (computing)Communications protocolMessage passingLogicComputer programmingParameter (computer programming)BitMultiplication signDatabaseSystem callFundamental theorem of algebraSingle-precision floating-point formatPrice indexPressurePhysical systemString (computer science)Set (mathematics)Exclusive orStreaming mediaVolume (thermodynamics)Computer fileKeyboard shortcutMathematicsLevel (video gaming)Patch (Unix)Web 2.0Presentation of a groupChord (peer-to-peer)CodeComputer clusterMeta element1 (number)Event horizonAreaTouch typingComputer animation
10:51
Form (programming)Condition numberWaveMultiplication signHarmonic analysisWeb 2.0BitWeb browserKey (cryptography)VideoconferencingForm (programming)Message passingFront and back endsComputer animation
12:10
WaveformWeb 2.0WaveMultiplication signHarmonic analysisHookingGame theoryCodeSensitivity analysisGame controllerLoop (music)WindowThomas BayesNumberWeb pageWeb browserMessage passingMedianOpen setComplete metric spaceSampling (statistics)FamilyVideoconferencingEnumerated typeVirtual machineWordMobile appKeyboard shortcutMIDIRoutingGraphics tabletVariable (mathematics)PasswordComputer animation
15:46
MIDIPointer (computer programming)Form (programming)Drum memoryExecution unitDigital image correlationMagneto-optical driveFirst-order logicWindowVideoconferencingRootSquare numberReal-time operating systemWeb 2.0Point (geometry)CASE <Informatik>Multiplication signLevel (video gaming)NeuroinformatikMetropolitan area networkFamilyConnectivity (graph theory)QuicksortSoftware bugMIDIComputer animation
18:00
MIDIPrice indexTwin primeComputer multitaskingLibrary (computing)Design by contractGraphics tabletMultiplication signHypermediaBit rateMIDICommunications protocolWeightGame controllerMaxima and minimaExecution unitQuicksortKeyboard shortcutComputer animation
19:38
AbstractionLink (knot theory)outputFunction (mathematics)Expandierender GraphQuicksortPrice indexNumberNoise (electronics)Virtual machineWeb browserLibrary (computing)Web 2.0Computer fileFunction (mathematics)Physical systemSlide ruleExclusive oroutputMIDIValidity (statistics)Keyboard shortcutLink (knot theory)AbstractionComputer animation
21:22
WritingGraphical user interfaceProcess (computing)Library (computing)Point (geometry)ArmHypermediaMultiplication signStreaming mediaCASE <Informatik>Cartesian coordinate systemCommunications protocolVolume (thermodynamics)Musical ensembleEvent horizonExecution unitProjective planeAbstractionGame theoryText editorFunctional (mathematics)Series (mathematics)Physical systemBeat (acoustics)MereologyWeb 2.0Message passingSocial classLogic synthesisForcing (mathematics)Translation (relic)Different (Kate Ryan album)MIDIFault-tolerant systemBand matrixBit12 (number)Head-mounted displayEmulatorStability theoryComputer animation
25:38
Open sourcePoint cloudFacebook
Transcript: English(auto-generated)
00:05
Good morning everyone as that part of the slide says my name is Steve and as that part of the slide says I'm a little bit of a geek if we've all turned up to the right room I want to be talking about web MIDI for the next 20 minutes or so We're gonna go through who I am what I've done to be on the stage
00:21
What is MIDI in itself so we can talk about what is web MIDI? And we'll talk about the protocol of MIDI and some examples which means I may be playing some music which is why I have a keyboard and a microphone It's just a good excuse to show off really isn't it? So who am I? What have I done to deserve to be on this stage or as it should probably be called
00:42
The ego slide where the speaker brags about its achievements for 20 minutes Not gonna do that because I haven't done anything Industry veteran by the way just means I'm old Developer and entrepreneur means absolutely nothing it means no one would give me a job so I had to create my own But the important thing is I have a term I mean
01:03
This is the only musical instrument you can play without actually touching it Even though if you go to the Museum of Music in Brussels, there is a sign on the theorem that says do not touch Go figure The important thing I guess for the purpose of this is that I do like composing music. I'm just not very good at it
01:25
so I Thought okay. I will do what I can which is code So for many many years MIDI has been the standard for musicians to Store their music into sequences and the playback now
01:41
MIDI means multiple things to multiple people When you say MIDI, that's not enough MIDI sometimes will mean the serial protocol that goes down the cable between Synthesizers between computers and synthesizers or drum machines or other equipment like that MIDI can also mean the file format and there were three forms of file format
02:02
They've been interestingly labeled format zero format one and format two Zero is pretty useless everything's on the one track But it is very easy to pause We file my format one has got every piece of music on a separate track which makes it very easy and many file format
02:20
Too used by almost nobody but you've got to know it exists. Some people mean that when they say MIDI And MIDI is also the hardware specification There is a chunk of the spec which says exactly how much current has to go through the cable What sort of tolerance is it has to be how fast it's running and all of that stuff We don't need to worry about we're in the safety of the sandbox of a web browser
02:43
so we don't need to worry about any of them, which is lucky because web MIDI is an API for that protocol and Nothing else All the bits that would be useful that are necessary and a sequencer are not available in web MIDI Thank you for coming to my TED talk. Goodbye
03:02
It's just an almost nothing in there so if there's nothing in there it will at least be easy to explain Does anyone decide for the message yet? That is a MIDI message If I present it in hex, does that help? No, at least one person's nodding
03:21
well The first part of this message is the command 1 4 4 99 and the second one is the channel now a couple of things you can work out from this Hex numbers 4 bits long from 0 to 15. That means there were only 16 channels That's part of the spec. It's been there since the year dot is not going to change
03:43
Unfortunately, however MIDI read the MIDI spec version 2 has just been announced So this talk will be completely redundant probably by the time I've finished giving it The first bit is the command and again you'll notice It's a single nibble in hex, which means there are only 16 possible commands It's another caveat. So
04:05
two possible things This command Dictates these bytes So serial protocol they'll come in one after the other and it'll your program will see that first byte It says what is this? It's a note on message at which point it understands that the following two bytes will be a note value and a volume
04:38
So the obligatory hello world apologies for the use of the word navigator in there, it's probably not really necessary
04:48
But this is the thing that will take in a message Passes it out and says well, we know note on is this 90 and we can track it out to the console So to prove that works
05:01
Probably shouldn't say that and then do a live demo. I should do the live demo and then so let me clear that If you look onto the bottom window One message for on one message for off still only three bytes each time 64 volume one
05:22
Where's the time where's the timing information in that message? See that can anyone tell me where the time is gone The time is in real life All the time in web MIDI is controlled by how much time you wait until you send the next message There is no other way around it So you are forced to rely on JavaScript giving you very precise timing in order to create very precise music
05:49
This is why I titled it garlic bread Is anyone aware of the Peter K sketch the stand-up he does where he goes garlic bread Because it's just incongruous you have a MIDI
06:03
protocol that is for music that is real-time that needs to have very precise timings and Then you go and stick it in a web browser Which is sandbox and is not very good at precise timing Which makes it a little bit painful at times if you try to do anything live
06:21
So let's continue with that so It's if it's a serial protocol. There is one problem with handling it. How do you know if you're in the middle of a message? or not We've established. This is the note on command. This is the note on message Well, what happens if the first byte we read is 60? What command is that?
06:43
What's it suddenly going to start doing at that point? What if we come in here? What about a hundred? Well, the clue is by writing it down in binary So if this most significant bit is one it means this is a command Something is happening. Now. You should be paying attention And if the most significant bit is zero, it means this is the data for a previous command
07:05
Very simple to pause. It's one operation and present 0x80 But I just said there were 16 possibilities for this number if we're using the most significant bit Then actually we only have eight possible commands and that's it. The entire specification is just eight commands
07:27
And that and that that's these ones so note off is the most important one because it's the first one No idea why note on is the second one Although you will find and this is where it gets fun
07:40
that hundred if First byte is the command the second two bytes of the data bytes now Any number of data bytes you like they're controlled by the command This is the volume Some keyboard manufacturers have decided that if you put zero as the volume on a note on Message that means note off
08:02
Because obviously it's logical, right? So you've got no on a note off the key pressure just for hitting it harder essentially Parameters and programs for changing the sound because although there were no sounds in the MIDI spec and there were no mid sounds in the MIDI files There is still the indication that there would be a sound so you say I'm just telling you the note
08:24
I'm saying this note goes on this note goes off I'm not telling any more about it But I will tell you I would like the synthesizer to play a piano or a string sound Doesn't tell you what the string sound is or how the string sound is formed. Just something in the general area of strings
08:41
Changing the pressure more after touch thing the pitch wheel obviously this expensive keyboard does not have pitch wheel But if you do then there was a special message for it and then there's the meta events for everything else It's like they started designing the keyboard. They thought yeah, we need notes on we need the ability to change the program We need the pitch wheel. Oh, dear. We've run out
09:01
So everything else is lumbered in this bottom stage And there are just hundreds of these meta events system s exclusive messages that send patches Just full of them This one. I know it's it. I know it's dot H file because this is that this list comes from my original C version of this library And it just lists them all in there
09:23
So what is message what we're missing in web MIDI. We've established the timing message. It is missing That is just something you have to time yourself When you deal with MIDI files, and I'll show you some bits later You have to specify that time so you have to count the time yourself You say this note comes on then this note comes on and there is half a second between it you then write that in
09:46
The duration again, if you don't have timing messages, you can't have duration So you have to handle that yourself think about how long each of these notes are going to be Cords Cords are kind of fundamental music
10:00
This is harmony every single piece of reasonably good music will have harmony and quite a lot of bad music will have harmony as well But there is no way of describing a chord You have to say this note on then this note on and then this note on At the same time or as close to the same time as you can possibly manage in your code
10:20
Ideally, you'd want that time to be zero though. At least shorter than 10 milliseconds And there were no conventions. So if you use one of the system exclusive messages, it's up to you How you decide to handle that? Every keyboard manufacturer has their own set of things that they put into those messages And if you want to process them or understand what that keyboard is sending you're on your own
10:45
so we've got the protocol is a stream of messages and I was going to give some examples, but I Will have to go out of here for that So we shut that down
11:02
This is the one that I've been fiddling with for the time. It's This one and I put this up as loud as I can and then flip this on
11:41
Now you can tell that there is a little bit of a latency on me pressing the keys and hearing the notes which is unfortunate But that's unfortunately nothing of my doing I'm not that bad a coder It is just the time that it takes For the message to come through and the browser to react and the sound engine inside the web audio engine to play those sounds
12:01
Because there isn't anything much is really doing there That's a web audio backend being triggered from from the from the JavaScript in the browser Now if I that's not good this was just an example this has got nothing to do it this is an example I was playing with earlier. I was curious about waveforms and how waveforms are created. So I built this web audio thing to
12:22
Create waves using different harmonics. I just needed time to hook it up to the setup today But if I show you another one like this soundboard, for example, if I play notes on here now
12:44
I didn't get the looping quite right on it, but All I've done here is I've listened for the note down message and when I get that note on message I trigger the sample loop. I don't do anything else when the note up comes. I don't stop it when the second note down comes I
13:02
Stop it. That's logic. That's one variable. That's one check which you can have hours of fun with Now while I'm doing that you might hear something else going on
13:29
There's a bass sound going on under sit underneath each of those samples that bass sound the bass sound from that window
13:42
When you open web MIDI and you say I would like to do some web MIDI, please mr Web browser and the web browser goes sure which would you like and it will allow you to enumerate Every MIDI device plugged into your machine Which is great. It means if you've got multiple keyboards, you can have one keyboard doing this thing and monkey They're all doing that thing works brilliantly
14:02
Unfortunately, it works brilliantly for everyone else as well So if you happen to have two apps that were running in different browsers, it means you can see the other person's traffic I Say this because I'm using web MIDI for music stuff. Mostly. Well as much as you can call that music. It's just a loop
14:21
But because I'm using if we say I think very much in the music sense, but these things are cheap, right? It's a ten of anyone's money There is no reason why you can't take that apart use the switches and hook them up to something else You might like to do a game controller, which is more accessible than traditional game pads That would make a very cheap and easy accessible game pad controller and it's controllable through the web browser with web MIDI
14:47
Unfortunately, if you were doing anything sensitive like you are requiring someone to put in a passcode using those controllers it would also be visible in another window and Currently as it stands
15:01
There is nothing to stop you It will be visible in every window and you can't stop you can't say oh I want this for myself That controller is available to everyone and you can start that up on your web page without a click Without the user having to perform things if you've done any kind of web audio things in the browser You'll have probably noticed that it doesn't work until the user had actually clicked on something
15:23
It requires the user to interact with the page before it gives you control or for web audio device Not the same for web MIDI you can just look at that traffic at any time another example we had which is probably the interesting one I suppose is
15:44
That one I think and of course it's still running in my other window, so let's shut that down
16:20
Oh Which is about the closest I'm gonna get so I'll leave that there or quit while I'm ahead Put that back on Now this is obviously a proper example of a base sense, which I didn't write but it is quite fun and is it is again using web audio and Web audio as I say you've got to click here before you can actually make sounds
16:42
But it's a nice example, and you can see that it is possible to build quite big and fun things and at this point I realize the person who's written this might be in the room No good That would be awful it's happened before actually sort of mid lecture activity thing I was at a computer thing
17:00
Funnily enough me computer thing who'd have thunk it Was it a computer thing and I was we're talking about old computers as my friends often do and I happen to mention this trick on the zx81 about how you could put components across the chip to fix the square root bug that existed on the zx81 at the time and I was explaining how it worked and why only worked in
17:21
That case and then I found out the guys that next to me built it And you know the mansplaining level was a 10 out of 10 at that point. I mean fanboy was off the charts But it's what it's one of those things. I'm sure I'm going to give this talk and the person who wrote it's gonna be here Which is a shame because it's really nice. I mean that's
17:41
And it works real time so I remember back in the 90s. You could make a whole album with just that I
18:04
Don't think I should have done that let's do that instead. Oh so guy and we've got some library links, so So as I saying I use this for music
18:21
But you could create a game controller by ripping out the innards and putting that into you know some sort of Something else as we say it's not keyboards drum pads They're very very fun. They're very easy I like those sort of things the Novation long pad which I can't quite afford but they are very pretty and I have lots of lights on them and
18:42
You could do it can control lighting rigs When I did some theater stuff a while back the lighting rig that was a big desk, but controlled by MIDI same protocol note on meant light on Note off meant light off But it's it's doable. It's it's it's a standard protocol that seems to have gone beyond what it should have ever been doing
19:04
The rate of MIDI is very slow. It's about 33100 valve rate essentially Well, she's slow when you think of how many notes could be playing in a piece of music Even a fairly straightforward Symphony with only maybe 50 or 60 players
19:20
There's 50 60 notes in their minimum You've only got 16 channels and you've got multiple notes So you're probably not going to get all of them playing at the same time Because you can't play all of them with zero time gap. There will always be a time gap between each note So it'll never be quite right This is the the slide you might want to take pictures off if you want to go and play with that synthy thing later
19:45
also, there is No file handling. Oh, sorry. Did I not did I go? For those that want to take the picture I'm ready for my close-up
20:02
So file handling is missing if you want to save the MIDI file You can't if you've generated something nice. You've got to save it yourself. Don't worry. I've written a library for that There'll be a link later for as well The Sysex abstraction. So any as I say any manufacturer such as Korg and Yamaha They will have their own system exclusive data
20:22
Which that you just have to go and find out for them and any actual sound the actual noises themselves not in the spec It's just an indication that there is a piano sound here If you do want to have a look my MIDI library is the second one. These other things are also perfectly valid MIDI libraries
20:41
My one was based on the C++ library that I wrote many many many years ago I think my original MIDI library is actually older than this conference and there's not many things that are So those were available if you've got the pictures now. Yep So my sort of basic conclusions is the web MIDI is really interesting for input
21:02
Lots of things support it not just the keyboards the drum machines and all that sort of stuff Very good for handling. You can build your own but the output is more interesting Because now the browser can control synthesizer rigs and it can control drum machines But the number of people wanting to do that is probably even fewer they want to do input
21:22
So with that I shall leave it open the last five minutes for questions, I should update my FOSDEM scorecard That's now the 12th different talk I've given and I still haven't got good at it So we've got a microphone if there's questions
21:45
So it kind of looks like someone took a naive translation of a wire protocol and dumped it straight into JavaScript without any kind of design process Was there actually a design process or did someone manage to persuade the Chrome team just to put in this API that no one had really
22:00
thought about I don't know Why or how web MIDI came to be I don't know the people I suspect that looking at it They said it's not so much dumping the wire protocol, but dumping what every other MIDI library does Every other MIDI library just gives you that raw data. Here is the stream Partly because you can't process the SYSX messages
22:23
There's no simple way of abstracting that other than say it's a SYSX message and here is the binary blob There could have been abstractions. They could have created an API which said on You know with with an event and you say I want this function called when I get a note on event But they didn't do that
22:41
I think because they wanted to maintain a direct relationship with all the Preceding libraries and the way that people wrote MIDI applications But that's my conjecture Yes, my library as attached off off the github there does abstract away a lot of this ugliness And it handles that odd stupid case as well
23:02
Where if the note is on but the volume is zero it actually means note off Which is one of those things that Fox me for ages There's also the odd thing as well in the MIDI spec that if you do a note on message and then another note On message you don't have to say note on again. You can just keep going So you don't get that most significant bit being one
23:24
but for the next message So you just see note on is a three byte message. Why am I getting five bytes? That's just because there's that quirk that was in the original MIDI spec because it will cut down on the bandwidth used You don't have to send a byte and back in 1982 when this MIDI was created that one byte was important
23:43
Now it's not but we're stuck with it So we've got these little caveats in the original spec that carried through to web MIDI. I have a microphone for you
24:05
So how stable is the web MIDI stuff could it be used for editor librarians for older? synthesizers like I have Some old stuff that I need to use an Atari emulator to actually edit stuff on could maybe be used to reliably do that
24:22
You think is it stable enough so is web MIDI stable enough to use as a librarian tool and so forth it is stable enough Web MIDI as a spec isn't gonna be changing. The API is not gonna be changing There will be upgrades for web 2.0 which is coming through the pipeline now
24:42
But it's more the case of do you want to spend that amount of time writing a librarian tool When there might be better ways of doing it You it would probably be good for you It would be good for me some of the time But because it's coming quite late into the game I'm concerned that it might only be used for fun projects like this
25:03
Rather than serious projects that have already got librarian tools and other system tools But I'd like to see it more done more I know how soon as someone puts a VR headset down. I know that interest is going there and not here
25:22
You still have two minutes of interest. Okay, so can we muster two minutes of interest? No, no Thank you a lot gracefully. Thank you very much. Thank you