Lumen
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47021 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Web browserProjective planeText editorLine (geometry)Lecture/Conference
00:34
Embedded systemBuildingLogical constantClient (computing)FluxLipschitz-StetigkeitSet (mathematics)Multiplication signWeb 2.0Service (economics)Client (computing)Physical systemProjective planeScripting languageRevision controlForm (programming)Formal languageSoftware frameworkException handlingWeb applicationBinary codeTerm (mathematics)Game controllerFunctional programmingFluxLogical constantWeb browserMathematicsMobile appImperative programmingProduct (business)Erlang distributionCompilerQuicksortSingle-precision floating-point formatAbstractionMicrocontrollerImplementationRun time (program lifecycle phase)Extension (kinesiology)Computer animation
02:40
File formatBinary fileVirtual machineRead-only memoryControl flowControl flow graphIntegerArchitectureType theoryWeb-DesignerWeb browserMereologyCore dumpTranslation (relic)Interface (computing)CodeStandard deviationGoogolForcing (mathematics)Software developerBridging (networking)Software testingExploit (computer security)Flow separationWeb 2.0Group actionType theorySemiconductor memoryFunctional (mathematics)Process (computing)Formal languageData storage deviceAddress spaceException handlingModule (mathematics)Latent heatFile formatWorld Wide Web ConsortiumIntegerXMLComputer animation
04:25
Compilation albumType theoryInferenceWeb browserEinbettung <Mathematik>ParsingStreaming mediaJust-in-Time-CompilerMereologyMultiplication signFunctional (mathematics)Web pageType theoryScripting languageClient (computing)Process (computing)Semiconductor memoryRevision controlEntire functionCompilation albumStructural loadData structureQuicksortLevel (video gaming)Binary fileSheaf (mathematics)WebsiteEmailEquivalence relationComputer animation
05:08
Constraint (mathematics)Concurrency (computer science)Data modelInterior (topology)Function (mathematics)Event horizonDifferent (Kate Ryan album)Thread (computing)Multiplication signWeb browserWindowNormal (geometry)Web 2.0Machine codeTerm (mathematics)Run time (program lifecycle phase)Event horizonScripting languageGraphical user interfacePoint (geometry)Compilation albumQuicksortAlgebraic closureWeb pageStructural loadCodeClient (computing)Erlang distributionProcess (computing)Rule of inferenceProxy serverSet (mathematics)Reading (process)NumberComputer animation
06:51
Machine codeRun time (program lifecycle phase)Type theoryScheduling (computing)CodeGaussian eliminationPerturbation theoryBytecodeLoop (music)File systemNetwork topologyTrailLibrary (computing)Multiplication signComputer fileSet (mathematics)Just-in-Time-CompilerGame controllerQuicksortTotal S.A.Thread (computing)Heat transferScheduling (computing)System callWeb 2.0MereologyType theoryTerm (mathematics)CompilerVirtual memoryMemory managementMachine codeWeb applicationCodeInformation securityGame theoryRun time (program lifecycle phase)Binary fileWeb browserGaussian eliminationTime zonePhysical lawSemiconductor memoryScripting languageFamilyBlock (periodic table)Process (computing)GodInteractive televisionLevel (video gaming)Point (geometry)Computer animation
10:13
CodeOverhead (computing)Variety (linguistics)Latent heatWeb 2.0Endliche ModelltheorieCodeQuicksortErlang distributionLibrary (computing)Module (mathematics)Mathematical optimizationLoop (music)Structural loadSystem callMereologyPerturbation theoryGaussian eliminationFormal languageProtein foldingLogical constantFunctional (mathematics)CoalitionMultiplication signContext awarenessTheory of relativityData storage deviceParameter (computer programming)NumberEntire functionCartesian coordinate systemInstallable File SystemCompilation albumCASE <Informatik>Level (video gaming)Default (computer science)Operator (mathematics)Insertion lossComputer animation
12:35
Term (mathematics)Memory managementProcess (computing)Category of beingBinary fileSpeicherbereinigungThread (computing)Scheduling (computing)Scheduling (computing)Web 2.0Multiplication signErlang distributionSoftware testingThread (computing)System callParallel portBinary codeProcess (computing)Inheritance (object-oriented programming)Core dumpSemiconductor memoryRun time (program lifecycle phase)CountingPerspective (visual)Memory managementOverhead (computing)CodeSpeicherbereinigungPlotterEvent horizonWeb browserFunctional (mathematics)BefehlsprozessorFrame problemCartesian coordinate systemSheaf (mathematics)Network topologyData managementCharge carrierDifferent (Kate Ryan album)Product (business)Scripting languagePlanningRow (database)CoprocessorDecision theoryRing (mathematics)Task (computing)TwitterDirected graphBitTerm (mathematics)Arithmetic progressionWebsiteComputer animation
16:23
Process (computing)Process (computing)Module (mathematics)Scheduling (computing)Cartesian coordinate systemDemo (music)Network topologyAdaptive behaviorWordKey (cryptography)WeightScripting languageComputer animation
16:50
Process (computing)Data conversionTerm (mathematics)Type theoryGreatest elementChainFunction (mathematics)Ocean currentParameter (computer programming)Frame problemCurveData conversionProcess (computing)Memory managementDemo (music)Pointer (computer programming)Ferry CorstenSpecial functionsChainResolvent formalismSystem callGreatest elementSign (mathematics)Scripting language2 (number)CodeData storage deviceMobile appInsertion lossLibrary (computing)ResultantData structureRow (database)Computer animation
18:42
Video game consoleFunction (mathematics)Multiplication signPoint (geometry)Process (computing)Letterpress printingUsabilityFunctional (mathematics)Frequency1 (number)Table (information)Interpreter (computing)Group actionCompilerProxy serverIntegerAutomatic differentiationRun time (program lifecycle phase)Electronic mailing listCodeDemo (music)Erlang distributionNormal (geometry)Computer animation
19:57
Execution unitNormed vector spaceComputer fileCore dumpInterpreter (computing)Normal (geometry)HexagonMixed realityTask (computing)Erlang distributionWeb browserEquivalence relationInteractive televisionDivision (mathematics)CodeMessage passingLine (geometry)Web pageSource codeSoftware developerSource code
21:32
Revision controlScripting languageInterpreter (computing)Level (video gaming)QuicksortBootingCompilerCartesian coordinate systemBinary fileMobile appTerm (mathematics)Loop (music)Parameter (computer programming)Goodness of fitInformation securityPoint (geometry)Computer fileErlang distributionParsingBinary codeVideo game consoleRight angleLecture/Conference
23:34
FacebookPoint cloudOpen source
Transcript: English(auto-generated)
00:05
So, our next speaker is Lukimov. Luk is one of the creators of the Lumen project, which he will tell you all about. He's also the creator of the IntelliJ Elixir plugin, which he will talk about tomorrow in the free editor's room. If you want to catch that, that's also going to be super cool for sure.
00:24
And yes, so let's give it up for Luk with Lumen. As he said, I'm Lukimov. I'm known pretty much everywhere online as chronic death, because I'm filled with autoimmune diseases.
00:40
So, let's start with an overview before diving into the details. Lumen is a new compiler and runtime for Erlang Elixir and anything else we can convert from Erlang abstract form. It targets things that are difficult or impossible to target with beams such as WebAssembly, XA6, single executable binaries.
01:01
So, not eScripts, but just single executable, like competing with Rust or Go, that sort of thing. Or true embedded systems without an OS, such as microcontrollers. Rich web apps represent a significant portion of the work we do. And with the client-side ecosystem in constant flux and long-term maintenance, an extension is quite painful, because you go away from a project for a while that's written with a JS framework,
01:25
and it just will not build. Too many things have moved on. Meanwhile, the server-side ecosystem is more stable. Like, people have not really had upgrade problems with Phoenix, even though we've added new features. So, the one thing that's really missing is we can't take those server-side languages and put them on the web.
01:44
But WebAssembly changes that. We can use previous back-end-only languages for client-side apps. And the reason why we're doing it now, even though WebAssembly hasn't really even reached 1.0 yet, is because we don't want Erlang and Elixir to be passed over because they don't have a WebAssembly target.
02:01
There's already targets for C, C++, and Rust, and people have done toy versions of Python, Ruby, and PHP, but we want to make sure that Elixir and Erlang have a true client. It's not just a huge 100-megabyte download to get a Python CLI and a REPL in the browser.
02:20
We want this to be something you can use in production so that, once again, people don't push aside functional languages. So, the WebAssembly spec even says, we want to make sure that this doesn't just work for imperative languages, but all the implementations are just C, C++, and Rust. So, if we get in there now, we make sure that they stay compatible with functional languages.
02:41
Not everyone in the audience may have seen Lin Clark's excellent articles on WebAssembly on the Mozilla Web Developer Vlog, so let me give you a quick introduction. WebAssembly has a format specification and a test suite, and development happens as part of the W3C community group, while formal standardization occurs under the purview of the WebAssembly working group.
03:01
So, these working groups are the reasons why browsers remain compatible. It is, you know, Google and Apple and Mozilla fighting to get a standard that everyone can agree to, so we don't go back to, like, it works in everything but IE. Or, it works only in IE.
03:20
The Lumen core team is part of that working group, so that we can advocate for functional features being there now and not being an afterthought. Or, it works, but you have to ship way more code to make it work. The overall goal of the designs are to make a safe, fast, sendboxable language for the web.
03:40
Separate code and data means you can't address code, so you can't store code, you can't do go-to for exception handling, but it also means we can't get exploits that require ROP gadgets, where you jump to the very end of a function to set up registers to be in a certain state, which is a big problem with x86.
04:01
The caveat with JSFFI right now is that we can only pass integers over the bridge, but JavaScript can see the entire memory of a WebAssembly module if it's shared, and so you can say, I'm here, and I'm this length, and read the bytes out. There is a proposal called interface types, where this translation will happen automatically, but for now we depend on the Rust WebAssembly support to do this translation for us automatically.
04:26
In browsers, WebAssembly is faster than JS because, as a binary format, it can parse faster. It's also set up to allow streaming compilation because there's a section that implements the equivalent of a header in C, so we know all the types of functions, and then each individual function can be stream-compiled,
04:41
and you don't have to have the entire thing download before you start compiling, which happens with even the most minified version of JavaScript. That being said, a lot of people think this means that it's a completely different stack in the browsers. It's not. Once you get down to the level of it's parsed and it's in some sort of structure and memory, both of those things can go through the JIT,
05:01
so we still benefit from JIT even though we get a binary format that's faster to parse and compile when it's for that first load effect. Now that you know what WebAssembly is and its benefits, let's see how Lumen targets it. Code size is critical. It directly impacts time to load the page as well as compilation time in the client.
05:20
Load time is a major portion of the time to the first paint, and any delay there is noticeable to the users. Threading in WebAssembly is very different animal than politics or Windows threads. Right now, the browsers take the requirement to have threading support in WebAssembly to mean we can use web workers, and web workers and browsers, for sandboxing reasons, are when it has processes, not actual threads.
05:42
We and other people are pushing for them to get real threads, but for now, they are web workers, and so it's not as nice. Going the other way, the main thread is very different than the main thread in normal OSes because that is the one that interacts with the DOM, and if you're doing work in the main thread, you freeze the UI,
06:01
to the point that Firefox or Chrome will pop up, like, do you want to kill the scripts in this page if you freeze it for too long? So there's a lot of gotchas. Async APIs require a callback, which interact differently with GC than normal closures. We are able to have async callbacks for the things that browsers support
06:23
because the Rust WASM support allows us to do that sort of thing, so we can use futures there, but not everything supports that, so for actual events that have to be concurrent, we need special support in the runtime we ship to the browsers to allow the Erlang code to be woken up from the scheduler
06:40
and run immediately in a blocking manner, which wouldn't be the normal way to run a process. And once again, for FFI, we need to translate between the JavaScript values and the Elixir values, or the Erlang terms. If you're wondering why we chose to build a new compiler at runtime, rather than try to port the Beam to WebAssembly, so like, if you have something, so like the toy examples for PHP, Python, and Ruby is,
07:04
they take a tool called MScript in, which existed before JavaScript, when they were just AMJS, it'll take any C code and just, it's run up on the browser, but it's huge, and usually pretty slow. There are some bins of caveats where people have like, gotten Unreal Engine games to run on their own scripting, but it took a lot of work. It doesn't just happen that it's in the browser
07:22
and it's as fast as native. Much of the Beam runtime support depends on APIs that are not available or entirely unsupported in WebAssembly because they would violate the security of the web, so like, init assumes you have a file system to read, or the way the memory allocator works is, it does a thing that makes total sense with virtual memory
07:42
on an OS where you ask for like, two gigs of memory, and you only write to the parts you need and the browser only gives you the parts you need, and it's perfectly okay that you ask for two gigs, but for safety on the web, when you ask for memory, in WebAssembly, you get that back immediately, and it is zeroed out, so you can't do that sort of, you shouldn't even fake those sort of calls on the web.
08:02
In WebAssembly, like I said, the way the main thread works is it's blocking, but also if we did ever have a scheduler on the web workers, they couldn't do DOM access, so we have to know which thread you're on to know if the API call is valid, or like, automatically go over to the other thread
08:23
when we need transfer control for a DOM call from a web worker. Additionally, JS values need to be tracked by the runtime, so we have to have sort of like, almost a NIF term type to keep track of the JS values so that they garbage collect correctly when they're on the other side of the JS shim that Rust generates for us.
08:45
Using the beam in the browser means that we would need to ship all the .beam bytecode files to the browser, but as an example, individual libraries, like the Timex library that allows you to do human readable times and time zones, takes a 1.1 megabytes, and like, 1.1 megabytes is,
09:04
you should feel ashamed that your client-side web app is that big, so if that's one library that's not shipping the same library, that's one like, extra support library on top, that's not viable to be competitive. Remember I said we want this to be a competitive thing, not a toy set of tools.
09:20
So one way we could do this and what even JS can do now is dead code elimination through what JS people call tree shaking, but beam files aren't really set up for tree shaking because we want to be able to do hot code reloading, and so there's no real way to set up to tree shake, or to do dead code elimination.
09:41
The other problem is we'd be running the beam VM interpreting, we would be running bytecode in a VM, and that VM itself acts as bytecode in WebAssembly to the WebAssembly VM in the browser, so it's VM on VM on VM. There's a lot of indirection that will just slow everything down. And additionally, because that beam bytecode is opaque,
10:02
it's not going to magically JIT your code, even if it's in hot loop and Erlang, into native code. There's just too many levels of indirection. So we won't get the benefit of the JIT then. So I've explained issues with the beam itself, but I haven't really explained how the design of Lumen
10:20
would solve those issues. First, Lumen drops support for hot code loading, which gives us a benefit. The reason why this is okay on the web is we can't really hot, we can't really replace a WebAssembly module. We could have a WebAssembly module of code to download a new WebAssembly module, and then do it again, but it's not really the same as on the beam,
10:41
where all your code would just know to call this one. It would have a different ID, there would be no replacement, it's much more cooperative than it would be where it just works on the beam anyway. But because of this, we can do full ahead-of-time compilation, so we only pay for what you're actually using. If you have an OTP application,
11:01
but you're only using one function, we only have to ship that function. This also matters a lot, because we don't have to ship the entire standard library. We only ship the BFG you're actually calling. And Erlang is huge. The Erlang module itself is huge, and a mess, and a lot of unrelated functions to each other. Additionally, it ships a bunch of checks,
11:21
some functions that no one should use in a modern context, because it's like MD5 and Adler32, which are completely unsecure. There's stuff we just don't want to ship. But initially, because we're ahead of time, we get everything that Rust and LLVM can do. So we get dead-core elimination, not just of functions, but individual instructions,
11:40
arguments, stores, and loads. It can do instruction combining if a better FUSE function exists, and it can do loop optimization or vectorization, sort of like how the Pelemate team is doing in Japan. Like I said, we're built on top of LLVM and Rust, and we use Wasm bindgen, which is the part that allows us to call DOM APIs
12:01
in a more transparent manner. Being on top of LLVM is kind of the default for new languages. Erlang just existed before it, so it's not built on it. But because of this, things like constant folding or dead-code elimination just happen at the LLVM level, and we don't need to do that.
12:20
Which means we're not, in that case, we're like five years after the language is out, then it becomes fast again, because it finally gets all those optimizations that are in every language already. So we're not falling behind that way. On the Lumen core team, I'm primarily responsible for the runtime and the bifs.
12:41
So the runtime is composed of five layers, the memory management terms, processes, schedulers, and bifs. The first layer is memory management. Memory management is the layer. The runtime was a work between Paul Schoenfelder and myself. Paul looked at how the code is written in Beam and poured that to like chunk sizing, super chunks, super carriers, all that stuff.
13:01
It's in us. So you don't have to worry. That will behave unexpectedly from how you're used to thinking about memory and GC in the Beam. And all that memory management and all the bifs are properly tested using Rust prop tests. So it's all,
13:20
we know it's segfault safe because we have to use unsafe code to do memory management in Rust. It's not safe for us. But I'm not just saying that like, oh, we profited, therefore it must be safe. No, no, I'm saying it found some segfaults. And so now that we run the prop test, we can trust that we've eliminated the segfaults.
13:40
From perspective of everyone here that's just writing Erlang or Elixir, the memory for processes are the same. Processes have heaps. They have reference count binaries when there are 64 bytes. And each garbage collection is per process. The processes have similar features to those in the Beam and from Erlang Elixir code, it's gonna behave identically.
14:01
So you don't really have to think about it. Lumen scheduler works similar to the Beam scheduler in that there is one per thread. Each time the scheduler runs, it checks if any timers to the timeout exactly what is processed. Right now we haven't implemented dirty schedulers because for the main web target, there's not really a concept of what would that mean to have a dirty scheduler. We don't really think that would be safe
14:21
because you're gonna try to freeze a thread so that'd be bad. How the schedulers work, of course, differs on WebAssembly versus native. On web, we have to worry about the main thread is the one we get and everything else has to be a web worker. And the Rust Wasm ecosystem deals with that for us.
14:42
On native, there's no special threads. And from the testing, because we have thousands of tests, we know that Rust generates a new thread for every test to make it cleaner and let them run parallel. So we know we can spin up 10,000 schedulers in 13 minutes
15:02
and nothing bad happens. So having lots of schedulers works just fine. So WebAssembly calls are blocking. There's no built-in support for declaring a function as async the way there is in JavaScript. Even if we could declare an async wrapper, we want the timers to work without polling and without hoping the event listener callbacks
15:22
are called when enough schedulers wake up. So we want the scheduler to somehow to be running in the background all the time. For WebAssembly, so that we don't plot the UI thread, we use a requestAnimationFrame, which is like, people in JavaScript used to use setInterval and just keep calling the code over and over again, but browsers started to block that because it was bad.
15:42
And so now the way to do it, even if you're not doing animation, is to call requestAnimationFrame and just keep rescheduling it over and over because then every time it does a paint, which unfortunately browsers can do whatever they want, but they say just assume 60 milliseconds, but they don't allow you to actually detect the frame rate, which is kind of annoying. So for us, we just assume
16:02
we have 16 milliseconds to work with, and so that leads to like a three to 4% CPU overhead at idle to just check if any process should be woken up or if any timer's timeout. We can optimize it later, though, based on knowledge about the timers. The runtime needs two ways to interact through the web through calls, JS calling into the lumen
16:21
and lumen calling out to the JS stuff. As described in the scheduler section, once the WebAssembly module is instantiated, the scheduler started it and just went from the background. It actually has no process. It does not have a knit process. It does not have an application tree for the demo I'm about to show you. This is just like running one application by itself. We will eventually support all that knit application stuff,
16:42
but it didn't need it for this, so we didn't go through the extra trouble. So we do these steps to happen asynchronously. So to call asynchronously, we use the await keyword in JavaScript. Using the lumen-web library, we spawn a special process that is immediately executed.
17:01
This allows us to do the JavaScript to term conversion in that process's heap without having to make a junk heap somewhere if we just did as an apply. And then we're going to put a special function call that will get a promise in the bottom, and on top of that, we'll do the apply,
17:21
because that way, when the apply returns, that special call at the bottom will be able to shove that over the wall to JSLand. To generate the promise, we actually give promise to our executor, and we only need to record. The executor doesn't need to be anything special. It is just a struct that holds the two callbacks that we get from a promise, which is you can call resolve or reject.
17:42
Resolve is when everything went good. Reject is when everything went bad. And reject just happens automatically if your process dies on exit, or abnormal exit, I should say. Normal exit's fine. Back in JavaScript, chain run one returns but a promise, so the wait curve just waits.
18:02
In any given frame of the compiled code, we're going to get the arguments from the stack, replace the current frame, and pointer and next label, and do that. Let me just, got like, yeah. Yeah, I got two minutes. Okay, so we're going to jump straight to the demo. So this is, oh, wait, was I not on that screen?
18:24
No, oh, no, right, because it's, yeah. Sorry, I forgot how this works.
18:47
Okay, so here is it working. We have full DOM interaction, so I can do a form, and it will generate tables.
19:01
And LuminaWeb gives you all this, so you can call these as normal Erlang functions. What this is doing is, it's a sponging demo where we, we enum reduce over a list of processes, process N gets process N minus one's PID, and then I give it zero, and it goes back the other way, adding. And so this shows both that we can spawn processes,
19:22
and we can pass integers, we can pass integers when they get too big, we can print, and if you want to, oh, I'm out of time.
19:41
Okay, we also have support for, and so this shows that the runtime works, but the actual, like, Elixir code was being translated by, with Rust, we're still working on the compiler. But there's an interpreter that works, and so if I go here, you can see it says Elixir in your browser.
20:02
I go, right, so this Erlang file is just sitting on my disk, and if I reload the page,
20:23
the interpreter can read that file using the, the normal JavaScript APIs that get a file, and we're able to read it into the interpreter, the interpreter compiles the AST from just Erlang source, and is able to run the code, and this is printing into a div element, so it's doing DOM interaction also.
20:49
Yeah, that's the equivalent of what it's doing. So this was converted from Elixir code, and I just did it in Erlang code because it's faster to convert. So one of the core team members of Elixir has a thing that will decompile any beam file back to Erlang,
21:02
and so we use that to convert the Elixir to Erlang for this. It's just a mix task called mix decompile, and so it's on his GitHub. I don't think it's a hex package yet. Questions, I guess?
21:40
Right. Yeah. So my follow-up to that. We can easily do the REPL with the interpreter. Oh, do we expect there to be a REPL? We can do the REPL easily with the interpreter
22:01
because we can just ship you the interpreter, but with the compiled version, it would be harder because we'd have to ship a REPL loop that we normally wouldn't ship. So I would say that you'd probably have to, unfortunately, use the interpreter for the REPL version. We might because we want to support you being able to give
22:23
almost like a boot script so that you don't have to enable, you know, the same reason why you boot your app when you have a release with everything booted, and sometimes you do a clean console with not everything booted. We would probably be able to support that because it's not going to change what you ship. It's just what you have running. And so we potentially will need a term parser
22:42
so that you would be able to change the arguments to the applications when you boot them with the boot script. But the boot script is usually in term to binary format. Already it's not like human readable Erlang term format. And so we will definitely support term to binary boot script,
23:02
but we don't know if we'll get to a point where you could just do a REPL that a human could read. For stuff that's compiled with interpreter, yeah. Like, I'm loading the file here, but it doesn't have to be loaded for a file. I also could have just put in the text in the console, and that would have also worked. It just would have been more error-prone, copy and paste to our problem.
23:22
Anything else? Oh, we do have stickers for Lumen, if anyone wants stickers. Yes, of course.