Accessibility in MuseScore
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47042 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 202014 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
OnlinecommunityPositional notationOpen sourceFreewareWritingPositional notationComputer programmingOnlinecommunityMusical ensembleRevision controlOpen setFreewareOpen sourceWebsiteSoftware development kitDifferent (Kate Ryan album)Computing platformFormal languagePresentation of a groupPoint (geometry)Computer animation
00:33
Open sourceSoftware frameworkFormal languageComputing platformMobile appOpen setOpen sourceEvent horizonElement (mathematics)Point (geometry)MereologyCartesian coordinate systemQuicksortBuildingProgramming languageBlock (periodic table)Mobile appGUI widgetDifferent (Kate Ryan album)Computing platformCross-platformFormal languageJava appletSoftware frameworkScripting languageAutomatic differentiation
01:29
outputFunction (mathematics)Focus (optics)Maxima and minimaComputerComputer programPhysical systemMenu (computing)PixelSet (mathematics)Structural loadQuicksortDescriptive statisticsDifferent (Kate Ryan album)Keyboard shortcutVideo gameRoboticsElement (mathematics)Computer programmingoutputForm (programming)SpacetimeInclusion mapInterface (computing)Focus (optics)Menu (computing)Software developerInformationLevel (video gaming)Right angleBasis <Mathematik>NeuroinformatikComputer configurationPixelProcess (computing)BitMereologyComputer fileStress (mechanics)Multiplication signPhysical systemDivisorTerm (mathematics)Interactive televisionVisualization (computer graphics)Hard disk driveDisk read-and-write headComputer animationXML
04:47
Computing platformKeyboard shortcutConvex hullWindowAddress spaceRadical (chemistry)Cellular automatonPattern languageComputing platformDot productAxiom of choiceNeuroinformatikRow (database)Touch typingDigitizingComputer animation
05:13
Single-precision floating-point formatCellular automatonPattern languageLaptopFeedbackCellular automatonDot productNeuroinformatikDigitizingPattern languageComputer hardwareRow (database)Electronic visual displayGoodness of fitSpacetimeReading (process)Computer animation
06:28
Duality (mathematics)Keyboard shortcutPhysical systemSoftware frameworkData modelConvex hullCartesian coordinate systemWindowComputer hardwareKeyboard shortcutSoftware frameworkTheoryComputer programmingComputer programSoftware development kitDirection (geometry)Library (computing)Lie groupComputer animation
07:07
outputGUI widgetStructural loadInformationElectric currentMenu (computing)Interface (computing)Category of beingElement (mathematics)Right angleArithmetic progressionGreatest elementState of matterElectronic visual displayInformationComputer animation
07:39
Machine visionGUI widgetStandard deviationDefault (computer science)View (database)Arithmetic progressionMusical ensembleCuboidGUI widgetTerm (mathematics)GoogolMachine codeCollaborationismSelf-organizationMachine visionComputer animation
08:17
Machine visionCollaborationismDecimalStandard deviationGUI widgetKeyboard shortcutView (database)Set (mathematics)GUI widgetSymbol tableKeyboard shortcutView (database)CollaborationismStudent's t-testLetterpress printingPositional notationCASE <Informatik>Mountain passState of matterMusical ensemble
09:07
Machine visionVisual systemScripting languageFeedbackCASE <Informatik>View (database)Symbol tableProduct (business)BitKeyboard shortcutLine (geometry)Function (mathematics)CollisionComputer animation
10:02
GUI widgetStandard deviationMiniDiscField (computer science)MathematicsCASE <Informatik>Multiplication signoutputGUI widgetFreewareCuboidTerm (mathematics)Descriptive statisticsStandard deviationCursor (computers)MetadataPhysical systemBitShooting methodComputer animationXML
11:33
Network topologyView (database)GUI widgetMusical ensembleArrow of timeCategory of beingView (database)GUI widgetDescriptive statisticsQuicksortBitTunisKey (cryptography)Computer animation
12:12
GUI widgetComputer configurationView (database)Execution unitMenu (computing)Key (cryptography)Electronic signatureMultiplication signElement (mathematics)Expandierender GraphCategory of beingClefBitInheritance (object-oriented programming)GUI widgetDescriptive statisticsArrow of timeComputer configurationObject (grammar)View (database)Standard deviationSymbol tableInterface (computing)AbstractionComputer animation
13:22
Arrow of timeKeyboard shortcutData structureGUI widgetKey (cryptography)Object (grammar)Key (cryptography)CuboidNetwork topologyKeyboard shortcutArrow of timeTerm (mathematics)Area2 (number)Computer animation
14:04
View (database)Keyboard shortcutArrow of timeWordLine (geometry)Different (Kate Ryan album)Right angleSystem programmingChord (peer-to-peer)Execution unitView (database)CuboidEndliche ModelltheorieFeedbackCursor (computers)Key (cryptography)Staff (military)GUI widgetMultiplicationExtension (kinesiology)Level (video gaming)MeasurementWordKeyboard shortcutSingle-precision floating-point formatMusical ensembleLine (geometry)QuicksortSymbol tableSet (mathematics)Focus (optics)LinearizationWater vaporMultiplication signComputer animation
16:01
View (database)Term (mathematics)InformationComputing platformDefault (computer science)Scripting languageEvent horizonSoftware frameworkSemantics (computer science)ImplementationAbstractionMathematical analysisOpen sourceView (database)Cross-platformInformationGUI widgetDifferent (Kate Ryan album)MeasurementOpen sourceNumberKeyboard shortcutBeat (acoustics)Staff (military)Form (programming)Key (cryptography)Descriptive statisticsScripting languageMultiplication signArrow of timeError messageLink (knot theory)WindowLine (geometry)TheoryGreatest elementMathematicsComputing platformRight angleInterface (computing)MetreFunctional (mathematics)CuboidProcess (computing)HypermediaEvent horizonPositional notationStandard deviationMachine codeElement (mathematics)Cartesian coordinate systemQueue (abstract data type)Software development kit
19:41
GradientMusical ensemblePositional notationGradientMultiplicationFocus (optics)Event horizonRight angleInsertion lossMusical ensembleResultantGoodness of fitGraphics softwareStudent's t-testSoftware testingMaterialization (paranormal)TheoryView (database)CASE <Informatik>Network topologyCursor (computers)MathematicsOrder (biology)Web pageControl flowReading (process)Default (computer science)Video gameLinearizationUniverse (mathematics)Nonlinear systemFlow separationScripting languageSlide ruleSocial classReal numberLetterpress printingOptical disc driveComputer animation
23:26
Point cloudOpen sourceFacebook
Transcript: English(auto-generated)
00:05
So I'm Peter Jonas, this is Mark Sabatella, and we're talking about accessibility in MuseScore. So what is MuseScore? Well, it's the world's most popular music notation program. It's free, open source, under GPL version 2. It's available on all platforms, translated into 40 languages,
00:23
and it's written in C++ and Qt. And we also have a website, musescore.com, which is home to the largest online community of sheet music creators. And what is Qt? Well, it's Qt, but often called QT, and you might hear us call it that by accident at some point during the presentation.
00:42
It's a cross-platform UI toolkit that basically enables apps to run on lots of different platforms. It's open source under various GPL licenses, and it offers two, there's two parts to Qt. There's the traditional C++ framework, which basically adds event handling to C++,
01:01
and widgets, which are like UI elements that are all ready to go. They look like native UI elements that you can use in your application. It also has a QML language, which is sort of a JavaScript-based programming language. And rather than giving you pre-made widgets,
01:21
like in the C++ framework, the QML gives you the building blocks to create your own custom widgets and custom UI elements. So now we'll talk about some general accessibility concepts. So what is accessibility? Well, it's basically inclusiveness, so designing for everybody,
01:40
so in particular people who struggle with the traditional interface of a mouse and a keyboard and a monitor and speakers. Because everybody's different, we all have different abilities in terms of sight and hearing, or even in terms of movement. Some people might struggle to use a mouse or a keyboard and various other different factors.
02:02
And one in five people have a disability, so it's quite common, and we all know that technology is an essential part of modern life. But our focus is going to be on accessibility for blind people, and the reason for that is that it's one of the most difficult forms of accessibility to implement,
02:21
because we're not just sort of adapting our interface, it's a completely new interface that's not based on visual interaction at all. But it's also more rewarding than other forms of accessibility in the sense that when you get accessibility right for blind people,
02:41
you often find that it fixes other accessibility issues in the process or makes them easier to fix. And it also leads to better overall design as well, even for sighted users. So when we talk about accessibility, we talk about assistive devices. So this is a device that helps somebody to use technology,
03:04
as the rest of us do on a daily basis. And the main one for blind people is a screen reader. So this is just a program that runs in the background and announces what's happening on the screen, on the computer. So it's a synthesized voice, like a robot voice,
03:21
and it will announce whatever element on the screen currently has keyboard focus, what is receiving input from the keyboard. So imagine if you're on a button, the screen reader will tell you it's a load button, and there's two pieces of information there. Firstly, the button is called load. That tells you what it does. And secondly, it tells you it's a button,
03:41
and that allows the blind person to know what they're supposed to do. So they're supposed to click it maybe by pressing the space bar. And there's sometimes a description that gives you a bit more information that it loads a file from your hard drive. So the experience of using a screen reader, you can imagine it a bit like using a telephone menu system, like press one for sales, two for marketing, and so on,
04:03
in the sense that the blind person, because they can't see, they have to listen to all of the options before they know what to do. And they have to keep in their head a mental map of what all the options are that are available. When it comes to programming for a screen reader,
04:21
the screen reader is not able to see pixels, so it's up to the developer to tell the screen reader what elements are available on the screen. But you don't tell the screen reader what to say, you just tell it what's there. And then the screen reader is clever enough to work out what to do based on that,
04:40
and this allows the users to adjust various settings like its speed, pitch, verbosity, and so on. So here's some popular screen readers on the various different platforms. So on Linux we have Orca, Mac OS has VoiceOver. Windows has a choice, and that's down to, in the past, the built-in screen reader on Windows
05:01
hasn't had a reputation for being all that great. But it's actually improved quite a lot in the last couple of years, so we might see it start to overtake the other two there. Another way for a blind person to interact with a computer is via a Braille terminal, which is a row of Braille cells.
05:21
Each cell is a pattern of raised dots, and the blind person reads the dots by touch, and each pattern represents a letter or a digit. The advantage over a screen reader is obviously this is silent, but it also means that the blind person can read it at their own pace,
05:41
and they can skip back and forth, whereas a screen reader just reads at its pace. And of course, a Braille display will work for somebody who's deafblind, whereas a screen reader's not going to work for somebody who's deaf. But the disadvantage to a Braille display is that it's another piece of hardware you have to carry around.
06:01
You can't get them built into consumer devices. You need to learn Braille, and they're extremely expensive. But the good news is, as developers, you don't need to get hold of a Braille display. Because they mainly just repeat what the screen reader is saying. So you can test with a screen reader, and you can ask your blind users
06:20
to tell you if the Braille display is giving something different to the screen reader. So get feedback from the users in that way. So how do you program for an accessible device? So if you're programming for an ordinary monitor or keyboard or so on, you don't do that directly. That's all done through a UI toolkit.
06:40
And the good news is that the same is true for accessibility. So you can use your same toolkit, so QT, and it will, in theory, take care of accessibility for you. And so no need to import any more libraries. But if a problem occurs, it could occur in any of these layers in the toolkit, the framework, or the device.
07:03
And it's not always obvious where the problem lies. So that's a disadvantage there. So I'll give you a quick tour of MuseScore's interface. So in the middle, we have the score view, which displays the current score. At the top, we have some toolbars for saving and loading scores and adding notes to the current score.
07:20
On the left-hand side, we have the palettes where you can add elements to the score. On the right, there's the inspector where you can change the playback properties or display properties of things that you've already added to the score. And at the bottom, there's a status bar that gives you information about the element that's currently selected. So that's the interface for a sighted user. And now Mark's going to tell you
07:41
about the progression of accessibility in MuseScore. Thank you. So accessibility didn't happen all at once. When we first were developing MuseScore, it basically was not accessible at all. I mean, we used the Qt widgets and so forth, but if you don't give them appropriate names,
08:01
the screen reader doesn't know what to read, and it'll read the value of a spin box but nothing else. And then our use of the widgets wasn't really well optimized in terms of their arrangement on the screen, how you could navigate through them. So really, it just wasn't very useful at all for blind or low-vision users. For MuseScore 2, which came out five years ago, we really made a concerted effort,
08:21
and it was some collaborations with a few other organizations, including Google Summer of Code, where we had a couple of students who implemented some really good features for us. So one of the things that happened was implementing what's called modified stave notation, which is this kind of large screen or large print editions that's not just larger.
08:42
It's like scaled in particular ways to make it easier to see. So we had to implement a lot of settings for that. And then just making the score view itself accessible so that you could read the score, because the score view was a custom widget before. It had no accessibility whatsoever. So we implemented the ability to navigate it by keyboard and ability to have the screen reader read it,
09:02
but still not for the palettes, which is how you add symbols to the score. So editing wasn't very accessible. For MuseScore 3, which just came out last year, and we're continuing to improve, we've really taken it up a notch, and now it is the case that you can really edit scores. And we've done a lot of things, including removing the need to have to see the score to be able to edit it.
09:24
I mean, if you put symbols on the score that used to appear where they felt like appearing and there would be collisions, and now we resolve those automatically so you can be pretty trusting that you're going to get a good product just by entering the things with the keyboard. And we've totally re-implanted the palettes using QML, and Peter will talk about that,
09:41
to make them screen reader and keyboard accessible, because they weren't either. And we've really improved the navigation and screen reader output for the score view. I'll talk a little bit more about that. And the bottom line is now we really have a pretty good story where blind users can both read but also create and edit scores.
10:05
Okay, so have a look at MuseScore Wizard, which is where you go to create scores. And this is kind of the best case scenario in terms of accessibility. So we have some text input fields there, and they each have a label next to them,
10:20
so title, subtitle. This is all the basic metadata about your score. And these are all standard Qt widgets. And as a standard widget, they basically give you accessibility for free. So the screen reader knows how to deal with these text fields But there's a problem, though, which is that those labels next to each text field
10:41
are not attached to the text field in any logical sense. So the screen reader doesn't know that title goes with that particular text field. So when the cursor is in the text field, it will tell you it's a text field, but it won't tell you what you're supposed to do. But the solution is, within Qt,
11:00
you can give the text fields an accessible name and an accessible description, and that allows the screen reader to know what they're there for. And you can also use the buddy system, which does allow you to link the labels with the text fields. So this idea of accessible names and descriptions,
11:23
it works with other widgets, not just text fields, but any standard widget, like a combo box or a radio button. And the screen reader will know what to do as long as they have the name and description. But sometimes you need something a bit more complicated, and you might need to use an item view. So this is a way of combining a bunch of items together
11:46
within one large widget. So now, rather than just the widget needing a name and description, every single item needs a name and description. The items are not widgets. This is a sort of technical distinction, and it means that you can't access them with the tab key,
12:03
so you have to use the arrow keys to change between different items in the view, and there's various other properties you can't set on them. So now we'll look at something a bit more challenging than the new score wizard, which is the palettes. So the palettes here is how you add elements to the score, and you can see that the design that we were given
12:22
for sighted users is like a tree of grids. So we have these categories of palettes, clefs, key signatures, time signatures, and each one expands into a grid of symbols, and the difficulty here is that there's no standard widget that will give you this appearance in Qt.
12:40
So your options are to combine multiple widgets to try and create something that looks like this. You could create a totally new one by re-implementing the interface of QAbstractItemView, which is like the base class of all item views, or you could leave the widget mentality of Qt and try the QML framework, and that's what we did,
13:03
and it turns out it's more flexible than the Qt widgets, so in the QML world, everything is an item, and you're able to set its accessible name and description. You can set whether it appears in the tabable, whether you can reach it with the tab key or whether you can reach it with the arrow keys.
13:23
So on the left here, you can see what the palettes would look like in terms of tabable objects. If each was reached with the tab key, then it would take 50 presses of the tab key to get from the search box at the top all the way through all of the items in the palettes and back to the top again, but if we make use of the arrow keys as well,
13:43
then what we can do is we can combine the whole tree area as just one tabable object, so now it's just four presses of the tab key to do a complete circuit, and while you're in the tree, you can use the arrow keys to navigate, so you can improve the experience
14:00
for blind users and keyboard users using that method, and now Mark's going to talk about the score view. So the score view is the view of the score itself, and here's where we really had to think outside the box of how to do things. We want it to be keyboard navigable, but we also need to have screen reader feedback, so the basic model we knew we wanted
14:21
is if the score view is showing you a particular note is selected, and you hit a cursor key like the right cursor key, it would move the focus to the next note, and hopefully the screen reader would read, and there you go. I have it set on UK English.
14:41
That's why it said crotchet instead of quarter. All right, so the navigation model, we can use a text-based model for this to some extent. I mean, text is linear. Music is at least somewhat linear, so we can have characters in text kind of map to notes in music, words where you'd use control left or right to navigate by text.
15:03
That's a measure or a bar in music, and then lines of text, we can compare those to staves in music, but it's unfortunately not quite that simple. Like for piano music, this is piano music. There's two staves here in parallel, one for the left hand, one for the right hand. There's nothing linear about that.
15:21
They happen at the same time, and even within a single staff, there's multiple notes happening at the same time, and there's other symbols that are sort of outside of that flow, like this symbol here is spanning several notes. So somehow you have to come up with a way of linearizing that so that people can understand how to navigate it,
15:41
and so what we ended up doing was coming up with a set of commands that just do exactly that. They linearize this so that as you move through it, it'll know I'm going to go from this note, this rest, to this symbol, to that note, to this, and then back here in a very standardized way in which you can learn to read the score that way. It's not dissimilar to how Braille might linearize it.
16:01
So now that we have the keyboard navigation happening well, now you need to get the screen reader working, and so the main thing you have to do is make sure you're presenting the right information to the screen reader so it knows what to do, and so Qt provides what's called the Q Accessible Interface, and if you define certain functions, then in theory the screen reader can access those functions.
16:22
The most important is to choose a role for your widget, and this is how the screen reader knows how to interpret it. Like Peter was saying, a button means, oh, that's something I can click. Well, there's nothing for score, right? So we had to choose which role fit the best, and trial and error told us static text worked the best. We pretend the score is just text,
16:41
and then the screen reader knows how to read text. So what we have to do is provide that text, and there's different aspects to that. You want to have that widget give it a name, and then there's value and description that go along with these widgets. So we are using for both the value and the description information about the score, and so what that might look like is at the end of every command,
17:02
when I hit that right arrow key and the note moved, at the end of every navigation command, every edit command, we run something that will calculate what to give to the screen reader, and it's basically a description of the selected element, its time position, you know, what beat and measure it's on, and what staff it's on,
17:20
but we also optimize this for the screen reader. The raw form is in the status bar, but for the screen reader, we optimize it so the most important information is first. A blind user will typically will start moving through very quickly like that so they can just, oh, I heard what I needed. I'm ready to move on. I heard what I needed, ready to move on.
17:40
So we want to make sure that worked, and so get the important information first, and really avoid repeating things unnecessarily. If you're navigating between notes within a measure, you don't need to repeat the measure number. We already know what measure we're in, or the staff name for that matter. So we avoid that, and then once we've calculated that information, you send a Q accessible value change event,
18:02
and this is what notifies the screen reader, the value of that widget has changed. In an ideal world, that would be all you need. It's not an ideal world. The reality is different screen readers choose, do I care that the value of this widget changed, because it doesn't know what a score view is, it doesn't know what anything is, it doesn't know what's important to read and what's not.
18:22
So NVDA works out of the box, and NVDA is an open source screen reader for Windows, and so this allows all, Windows is our biggest platform, NVDA is free, so it allowed us to hit a lot of users at once, but the other screen readers and the other platforms, they have different interfaces, different APIs for how they work, and only some of that is abstracted by Qt.
18:41
So we were able to create scripts for Orca on Linux, and JAWS, which is the most popular screen reader for Windows, which is the most popular platform. So now that we have those scripts there, we can now say we fully support Orca as well as JAWS, as well as NVDA within MuseScore.
19:00
So the bottom line here is that implementing accessibility, we're not talking a ton of code, there's nothing about it that's hard, except wrapping your brain around what needs to be done. So it can be hard to find the information, the toolkits are great, they provide some information, but there's still a lot of platform dependencies in here, so you just have to be prepared for some trial and error.
19:21
There's a couple of tools, Exerciser and Accessibility Insights will give you some information about your application, what the widgets are, that's what you're seeing at the bottom of the screen, and what's going to be presented accessibility-wise. This is a link to a whole lot of some of the different standards, the different toolkits and so forth.
19:42
I want to just wrap up by talking about a real-world use case. I teach music theory at a university. This is a blind student we had this last semester, right as we were wrapping up this work. So she got to be kind of our first really important test case, and I was able to create all my teaching materials for the course within MuseScore,
20:01
and then print them to give to the sighted students, and they took their tests on paper and pencil. This is her taking the final exam in MuseScore, right next to everyone else in the class, taking it this way, and she was able to read the test, fill out the answers within MuseScore, and then I was able to take the results and grade them and add them as comments within the score
20:20
and hand her back her grades and all that, and it all just works beautifully, and as you can imagine, it's really actually kind of life-changing. So thank you so much, and I guess we have one minute for questions even. Yes.
20:53
Yeah, I mean, to some extent,
21:07
you have to assume the blind user is maybe creating music for sighted musicians because we're not going to create Braille scores for them, and so you hope that we do a lot of good things by default, and we do, but there's definitely situations
21:21
where you're going to want a sighted person to act as your final editor to insert page breaks in reasonable places and things like that. I guess that's the best way I could answer the question about making that accessible. We had another question over here somewhere. Yes.
21:46
Yes, so it has its own cursor for some reason, so with all the other screen readers, you have a tab cursor. You press tab. Something gets focused, and the screen reader thinks that that has focus.
22:02
VoiceOver has a separate cursor, and so you can have more than one thing that has focus on the screen at once. Also, it was not scriptable. I think it was mentioned on one of Mark's slides, so we haven't been able to get it to work with the score view because it's not seeing this value update event
22:22
that is being sent. What I'd say about that is you can issue the command in VoiceOver that says read what's selected in the score, but that value change event is ignored even though the Apple documentation suggests it's the right thing to do, and so we have no real recourse there to understand what to do.
22:42
Yes. Okay, one more. Yes. Yes. Mm-hmm. We will linearize everything in your score for you,
23:02
so yes, there's a lot of things that could be nonlinear, multiple stays, multiple voices within a thing, multiple markings attached to things. We come up with an order and say we are going to walk that tree for you in some linear order.