Polishing X11 and making it shiny
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 97 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45740 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201035 / 97
1
2
4
5
6
8
14
15
16
17
23
29
38
41
42
44
46
47
48
50
53
54
62
63
64
65
66
71
74
75
77
78
79
80
82
84
85
94
00:00
Right angleMereologyDemo (music)Computer animation
00:31
WindowBitNeuroinformatikExpected valueVisualization (computer graphics)Java appletFrame problemProduct (business)Mobile WebRandomizationVideo gameFlash memoryGodGraph coloringSet-top boxPurchasingGroup actionData managementComputer animation
02:51
WindowPresentation of a groupRandomizationInternetworkingResultantArmVideo gameSemiconductor memoryMusical ensembleTouchscreenElectronic mailing listRaw image formatVideoconferencingFrame problemServer (computing)Direction (geometry)QuicksortGravitationPosition operatorSet (mathematics)ImplementationTablet computerContent (media)GodFlash memoryCollisionStatistical hypothesis testingComputer fileCuboidOrder (biology)Open setComputer animation
06:23
QuicksortMereologyRotationAsynchronous Transfer ModeFlash memoryRoundness (object)Metropolitan area networkParameter (computer programming)Client (computing)Subset1 (number)CodeTouchscreenExtension (kinesiology)Video projectorComputer hardwareMathematicsPattern languageGame theoryCartesian coordinate systemGroup action2 (number)CASE <Informatik>Right angleAbsolute valueEntire functionOrder (biology)WindowSingle-precision floating-point formatPoint (geometry)Multiplication signVideo gameImplementationConfiguration spaceRow (database)Machine visionAcoustic shadowSemiconductor memoryINTEGRALBuffer solutionSpacetimeTransformation (genetics)Sweep line algorithmGoodness of fitRandomizationPatch (Unix)Mobile appSet (mathematics)Computer animation
12:13
SequenceData managementKey (cryptography)WindowLevel (video gaming)QuicksortClosed setPoint (geometry)Proper mapMobile appReal numberFlash memoryGoodness of fitOperator (mathematics)VideoconferencingRight angleFrame problemGravitationBitServer (computing)Event horizonTouchscreenRow (database)Multiplication signAuthorizationGroup actionSoftware bugWeb browserCartesian coordinate systemMereologyMobile WebOrder (biology)Client (computing)Numbering schemePatch (Unix)Overlay-NetzGraph coloringPower (physics)Asynchronous Transfer ModeContent (media)Entire functionVideo gameData structureMoment <Mathematik>Staff (military)Sheaf (mathematics)Uniform resource locatorSign (mathematics)AdditionWater vaporSuite (music)WordLiquidComputer animation
21:16
Water vaporMultiplication signDevice driverWindowLine (geometry)Computer configurationZoom lensProper mapUser interfaceSynchronizationTouchscreenBuffer solutionPoint (geometry)IntelComputer animation
23:39
DebuggerData managementSoftware bugWindowFrame problemClient (computing)Goodness of fitTouchscreenComputer filePatch (Unix)Multiplication signDifferent (Kate Ryan album)Electronic mailing listArtistic renderingMereologySet (mathematics)Presentation of a groupOcean currentImplementationSpacetimeArmWeb browserRoutingVideo game consoleFerry CorstenFlow separationAxiom of choiceParallel portState observer9K33 OsaCommunications protocolBitQuicksortAsynchronous Transfer ModeCursor (computers)Tracing (software)Server (computing)Computer animation
29:16
TheoryComputer animation
Transcript: English(auto-generated)
00:00
Right, so my name is Daniel Stone, I work for Collabra Limited, they're all fantastic, and yes, that's part of our logo. So if you came expecting whizzy graphical demos and stuff, this is not it, and you might probably want to go to a more visually interesting talk.
00:23
This is about as shiny as my talk gets, so sorry about that. I picked the title in about three minutes. So basically, the whole thing behind my talk is every frame must be perfect. I've been working on consumer and mobile devices for the last four or so years of my life,
00:47
and it's been absolutely wonderful. So the problem we have is that the desktop is allowed to look fairly rubbish. You're allowed to have incorrect frames, so when you're resizing your windows are allowed to look horrible,
01:08
it can look jittery, you're allowed to flush out random bits of colour, you can bring a window up and it can go black and then be repainted. And this is alright, partially because desktops are really fast, so you don't entirely notice it,
01:24
and partially because people just have really low expectations of computers and I can't fault them. Sadly, when you buy a consumer device like the wonderful Nokia N900, which you should all go out and purchase several of,
01:44
really they're wonderful, they've got Java on them so you can wander around the conference like, hey, where are you? I'm in a hack room. Cool, so am I. So if you get just a tiny little glitch or a visual artefact on that, it's really, really jarring,
02:03
because you don't expect that your phone does that. And the same goes for anywhere else where people tend to put X, like set-top boxes, or anything people pay money for that doesn't have a keyframe, basically. And so, yeah, that causes, when you're working on the product management,
02:24
come up and say, oh my god, this must be fixed next week. Never mind that it's going to take about four weeks, because, you know, it's a tiny flash of black, how hard can it be to fix that? So that's the kind of thing that gets you critical, oh my god, we're shipping next week,
02:41
and we can't ship with this device. So maintaining X for a consumer device is really, really fun, and I can highly recommend it. Right, so the rough laundry list of problems, this has come from having worked for Nokia
03:02
for two and a half years before that on all the internet tablets, and Collabra subsequently working on the N900 and a few others. Most of the things I've come up with, come up against, sorry, and also when I started at Nokia, I did make a fairly concerted effort
03:23
to hunt down all the other phone and consumer device X3s I could find, and see just exactly how other people had pushed with them. I tried to find all the other PAP sets again for this talk, but sadly they're all hidden, all the SDKs tend to be hidden behind,
03:44
being aware of the leopard sign, and downloading, you know, one gig of someone's SDK in a raw file didn't seem like a great idea at positive. This is mostly the stuff from Nokia that I've had direct experience with,
04:01
and what I remember from looking at rubbish PAP sets about four years ago. So the first problem is that X isn't Wayland. Wayland's guiding design principle is that every frame has to be perfect. Everything that gets presented to the screen is exactly what you want to see.
04:21
There's no tearing, there's no sort of intermediate frames, being that, you know, you resize and window gravity kicks in, and sort of shoves your content somewhere else, and then you redraw your contents, and all that kind of thing. So if X was Wayland, it'd be great, but it's not, so here we are.
04:44
Window reconfiguration is the hard one. When you're resizing and moving windows, you're almost guaranteed to get god-awful results between start and finish. Random reconfiguration is absolutely horrible,
05:01
and that's been a large chunk of my life I'm never going to see back again. Video doing it correctly is very hard. Our server implementation, it was written by people who generally don't care about this kind of thing.
05:21
Composite does fix most of these problems, except that sometimes you don't want it. On ARM devices, we're often extremely limited by memory bandwidth, so if you need to do a double copy, generally your UI just gets hideous and slow. So you need to enable and disable composite on the fly,
05:45
and it turns out that enabling and disabling composite results in your screen looking rubbish in the meantime, or flashes of black and other music. The initial and final presentation of the server
06:00
when you start and shut down is fairly ugly, especially when you shut down, you kind of get all the windows undecorated in some random order, and then you get your nice little shut down screen that we paint onto the frame bottle,
06:21
and everything tears. Right, so basically all of this is fixable to some extent, right? There aren't any particular fundamental problems that mean we can't solve it. So, as I said before, everyone hacks it to be shippable.
06:44
So if you look at the Xorg list, there's a bunch of people saying, I've got this tiny problem, how do I fix it? Generally there's not much good advice forthcoming, and from the patch sets I've seen, they hack around it in the worst ways possible.
07:01
And thankfully for them and their sense of shame, it's generally really hard to find these patch sets. But it's also bad because it kind of perpetuates the problem. And yeah, the ones you can't find are probably even worse. So ideally we'd be able to fix this stuff up
07:21
so people who want to use X in a mobile or consumer device can just install it and ship it, and they don't need to spend four months looking at code going, I don't know what this does, but it flushes black, so I'm going to comment it out and hope for the best. Right, so Randa comes first
07:41
by virtue of being the worst offender here. It's built on the assumption that you never want to change screen size, which seems odd for an extension which exists to let you reconfigure your screen. It's built on the assumption that all you ever want to do is say,
08:01
plug in your projector every now and again, or change size once a week or whatever. But it turns out we need to do this fairly frequently. It seems to be less popular these days, but suddenly a few years ago a lot of people tried to make games
08:21
and they were ridiculously slow, so they just resized the screen to roughly half. And in the N900 we switched between portrait and landscape quite a lot, so when you start the phone application it does a rotation transition, and this used to take, what was it,
08:40
like three seconds in the worst case? Yeah, so it could actually make you miss your call, because by the time you've rotated into the phone UI they've given up. And yeah, the current implementation is just an absolute sledgehammer. Something changed.
09:01
By God, let's re-do the entire thing. In order to rotate, I think you previously had to re-layout all your visible windows six times, and re-layouting in GTK still takes approximately forever.
09:21
So that was quite deeply unpleasant. And it's sort of exacerbated by there's no single random entry point that you can say, here's my new screen configuration, please make it happen. If you want to rotate, you have to disable the CRTC,
09:41
then change your screen size, and then re-enable the CRTC with rotation. So if you're doing sledgehammer reconfiguration at each step, then it's pretty awful. And it also means that if you want to rotate and you're doing a nice whizzy transition between portrait and landscape,
10:02
you have to have a natural black point in the middle where you can disable your CRTC, quickly rearrange everything, and then turn it back on. We used to, it was a nice sort of gradual sweep between portrait and landscape, but luckily we changed the transition
10:21
so it kind of flips around and then it goes black very nicely and everything happens under the garden and then it comes back up again. And it's also built under the assumption that no one actually ever wants to rotate. Rotation is generally punitively slow.
10:42
And it works by having a shadow buffer and then every now and again when we're kind of done with clients and we get around to it, then we do a rotated leap into the transformer space. And this is partially because there are very few devices left
11:01
with hardware rotation engines. So it was designed for the desktop which has no hardware rotation and even on the desktop it's unusably slow. And composite, switching between the two modes is fairly punitive. If you want to switch between
11:21
composited and uncomposited, you get flashes of black and then background and then other and misc. Saying don't do that then is a really attractive argument. I spent about four months saying don't do that then, but sadly in the end really for performance reasons
11:41
we do need to be able to switch composite on and off pretty much a bit. So you have to do it which means we have to fix it. It's largely hackable around from the client side.
12:00
We spent about a week or so and the transition between composited and non-composited is now seamless on the N900 but it's really not obvious how to get to there. Right. Window reconfiguration is again, fairly awful.
12:22
Luckily you don't tend to do it that much in the mobile world which is really nice. But basically it's specified to look hideous in between start and finish. You've got your backgrounds and bit gravity which basically guarantee incorrect frames.
12:41
When you resize something and stretch it it's going to move around the contents for you. It might be painting the background for you when you bring a window back up and that doesn't really give you anything useful. All it means is that you get a flash of grey.
13:01
Apps, it's partially the application's fault. For example, on the exton application on a certain unnamed mobile device I'm not going to name it because it's otherwise perfect and I don't want to slander its good name. But if you set a black background for the exton it still leaves the window background
13:22
as white. So when you bring it back up it's going to go white, white, exton. And that looks awful. When you're reconfiguring windows especially if you're trying to move and resize windows at the same time which we do when we transition
13:40
between full screen and not as we've got a title bar on the top then you basically need to be using composite during this stage. So if you're in un-composited mode what you really need to do is get into composite mode and try not to cause artefacts there reconfigure your windows
14:01
under the curtain wait until they're all done, paint the contents and then back out of composite mode again. And yeah, all apps and toolkits are ridiculously stupid here. They do all the wrong things which is partially our fault for never telling them what
14:21
the right things are which is partially ex's fault for often not having the right answer. But they really do make our life difficult here. And if someone wants to fix GTK to be smarter, please do. Yep?
14:41
Yes. At some point. I've got the entire limo patch set which I'm going through now and cleaning up and trying to get out and also to document the pitfalls that we've encountered
15:00
and how to work around them from the client side. But that sort of relies on someone doing it, so hopefully it will do. Yes? Can you name some of the worst problems and proper solutions for stuff that GTK is doing wrong? Or any other toolkit
15:21
you're intimately familiar with? Some. One or two. The worst ones. The main one I find absolutely infuriating is using a background. And, sorry, telling X that you have a given
15:41
background colour. And ten points for having the background colour you tell X not be your real background colour. So, as I said with the extern thing, if you set the background to black when you bring it back up it's going to flash white as X paints the background the apps have told us to paint, then it will flash black
16:01
then you'll have the extern. That's the main one that really irks me. But a lot of them are I'll have to sit down and dig up my notes but I'd love to coherently yell at the GTK people. It would be very bizarre.
16:22
Window managers are also fairly dumb. In fairness this is mostly our fault because it's impossible to write a really good window manager. You just can't really do it. But a lot of the window managers are, sweet, I've got an event I'll chuck that on the screen right now.
16:41
And then if it's sort of got about five events in a row it'll do that five times in a row so you'll get five intermediate frames instead of batching them up and going I'll wait until we can present something useful. So this does seem to be getting a little better but it's still fairly dumb.
17:03
Window managers just present exactly what you tell them to in order, which is not what they should do. I've never used, I've not used enlightenment since 1999.
17:35
Okay, enlightenment may actually be the way forward. You think that gets released after?
17:42
Yeah, any day now. Do you still have a bug when you close applications? Sorry? You still have a bug when you close applications because you're excellent at them you don't have to draw them you sort of just do whatever it wants. Nice. Don't close applications it'll be fine.
18:01
Yeah, so even when you've fixed up all these little niggles in the X server and stuff it's probably still going to look awful unless you've got a friendly toolkit and browser rendering engine and app and other app author around you where you can say you're doing this, please make it so
18:21
oh, please fix it. And when you tell them that usually they say I'm not really sure exactly what's going on I think I'm telling it's the right thing and basically everyone took part. Video is pretty harsh as well.
18:42
So we have overlays which have usually the most tedious colour you can think of a very tasteful lime green incidentally when I was about five I used to always change my mother's background to lime green and the rest of the colour scheme to fluorescent colours
19:01
and she didn't smother me somehow which is amazing because every time I play a video and I see colour keys, it's ridiculously jarring. So thankfully we have textured video going through the 3D engine instead of overlaying but 3D engines especially in mobile devices will tend to
19:21
use all the power in the world. Overlays are extremely efficient with power and throughput textured video not so much but it does mean that you don't have to have the colour key, you can use it in other parts of your UI instead of just having your video right on top of the screen and nothing else
19:42
So if you very very carefully use the API and make sure that the end to the API, make sure that your order of operations is excruciatingly correct so you always start video and make sure it's there and then
20:02
paint the colour key behind that and then when you're done always paint something other than the other big colour key that you want to show and then stop your video. Then you can almost not show a colour key I say almost because sometimes
20:22
X will stop your video for you and the colour key is there and there's nothing you can do about it. So that would be nice to fix at some stage if someone video happy wants to look at XP.
20:40
Frequently some people use a colour key close to black and then you pop a dialogue up over the video and your dialogue is sort of half a video. It is a kind of ghetto transparency I guess. Most people really don't take much care
21:02
in how they sequence it, they just say oh cool, XP, this is really easy start video, stop video and in the meantime you've got the colour key flashing everywhere and chunks of rain over here it's deeply unpleasant. Right, so
21:20
tearing is starting to be a very big problem for us as well. Previously going back a few years you had kind of QVGA sized screens and very much static UIs. You pressed a button something would happen, that was it. But now we've got queued screens, the
21:40
N900 and most of it all of its predecessors are 800 by 480 and everyone wants nice, whizzy animations all the time and you know your background is kind of water ripples and you know, you press something and you get a nice little animation coming up and everything
22:00
is great. But unfortunately this means that you're almost suddenly going to have tearing so your animation mostly consists of a few jagged ugly lines going right through it. And also you're probably texturing from the entire planet. If you press the upper left window switch button on the
22:20
N900 it'll do a very nice zoom out and a gradual blur over the background and it'll show you every window you have. So it's actually starting to get complex and dynamic enough that we're hitting the tearing quite badly. So your two options are either you fix it or you never ever do a horizontal
22:42
animation. Ever. Yeah. If you happen to ship backgrounds with very strong vertical lines and then you can move them horizontally all the time turns out this looks horrible and you should never ship that thing. But luckily DRI2 makes it possible
23:02
to avoid tearing properly. With the swap buffers stuff which has just been merged to master and will be in the next release then you can sync everything properly to be working to avoid that. So that's been something we've been working on for a while and it's fully supported and working now in the Intel driver.
23:24
admittedly that is only on your desktop but you can avoid tearing now. In fairness we're only 30 years behind here. It's an improvement right?
23:42
So those are the sort of more fundamental issues which we can't really fix without bumping the protocol or becoming wailing. There's a bunch of much smaller little implementation issues which we can fix relatively
24:00
easily. So the first one being the initial presentation when you start the X server. It's not the rootweave anymore. We don't show that nice little black and white X cross cursor anymore. So we have got there. But the problem is when you start something up it'll give you a very nice little
24:20
smooth animation. There'll be a corporate logo. It'll be very sort of relaxing and calming. And then the X server will pop up and just smash black all over the screen or something. So a fairly popular hack is to let X take an image file as a background
24:41
and hopefully it's the same one as you've been using for the rest of your startup animation. And we should probably promote support for that. I've got that on my to-do list because I've seen that at least 10-20 different path sets where people have put in support for having
25:00
an initial background file. Again, same problem with shutting down. This is partially not X's fault because people tend to kill the window manager and then X, which is quite provably the wrong answer. But shutting down X, you tend to get undecorated windows all over the shop
25:21
and then you get a nice gentle shutdown. Well, I was thinking about the frame or the background image thing. What about grabbing the current image from the frame buffer? Grabbing the current image from the frame buffer is a very good choice as well.
25:42
It's a much better choice than the most impressive one I've seen and I'm not going to say who it was originally. It would start the X server which would clear the background and then in parallel it would sleep for almost exactly as long as the
26:01
X server would take to start up and clear the background and then a tiny bit of slack in there and then repaint the same thing back over from the user space. It looked awful. Grabbing it from the user space does make a lot of from the current frame buffer makes a lot of sense.
26:21
But then you want it to actually be something useful rather than say, the current text mode console or whatever. So that's kind of a more difficult one. Server side cursor rendering isn't great. Luckily we don't need to use cursors very often
26:40
in mobile devices but it almost kind of works on the desktop but it's slow enough on mobile devices such that especially if you're using a particular browser which likes to repaint its entire screen all the time you just get the cursor flickering horribly. That's quite a joke.
27:01
So I've got a patch set which allows client-side cursor rendering so the compositing manager can just paint the cursor when it paints the frame anyway. Hopefully that's no fault. It's also impossible to debug rendering issues.
27:22
If someone looks at something and says shit black's popping up here or part of my window's popping up here and it shouldn't be then how to get from there to actually fixing the bug is really not clear at all.
27:42
Partially because the whole client-server separation with pretty much zero debug tools for that whatsoever makes it a bit tricky, partially because the server's hard to debug anyway and so there are xmon, xpy
28:01
xscope and about a million others that will tell you exactly which commands the client's sending over the wire. So far I've not managed to make any of them work. xscope I think will work for about three minutes and then randomly die. So one of the things I have on my to-do list
28:21
is xscope that actually works, including telling you exactly what you're painting to whichever target you want and exactly what provoked it. So yeah, that's the JavaScript problem. As I said, toolkits
28:40
not great. Browsers, gecko is quite terrible. It's yeah, I don't want to get into that but don't use gecko. But then it's impossible to prove without any useful traces. So I can show
29:00
up to the browser guys and say hey, your rendering is rubbish, can you please make it less painful? And I'll say prove it to me, show it to me exactly what we're doing wrong. And then it kind of slings away with the target when it works. And that's pretty much all I've got. It is a short talk
29:22
because in theory it is going to be talking about Cairo-GL. I don't know where it is so that's great. Does anyone have any questions in the meantime?