How to design powerful DSLs for users
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 287 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/56901 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
DiagramEngineering drawing
00:37
Software developerConstraint (mathematics)Web browserAsynchronous Transfer ModeLetterpress printingPasswordInterface (computing)Data bufferCountingPhysical systemSpacetimeWindowCompilerComputerLatent heatTime domainLemma (mathematics)Task (computing)Computer fileMarkup languageDistributive propertyLimit (category theory)ImplementationParsingIntegrated development environmentInformation securityUsabilityLibrary (computing)String (computer science)outputMacro (computer science)Formal grammarRegulärer Ausdruck <Textverarbeitung>Token ringParameter (computer programming)Formal languageConstraint (mathematics)Web browserSource codeChainCompilation albumBinary codeWave packetPower (physics)Right angleCellular automatonComputer programmingForm (programming)Programmer (hardware)Atomic numberSurfaceExpressionCartesian coordinate systemCodeArithmetic meanElectronic mailing listProblemorientierte ProgrammierspracheEndliche ModelltheorieKey (cryptography)Variable (mathematics)CompilerPhysical lawFunctional (mathematics)Point (geometry)BitAsynchronous Transfer ModeMultiplication signDependent and independent variablesCASE <Informatik>Set (mathematics)Latent heatComputer fileSheaf (mathematics)Substitute goodString (computer science)Markup languageWebsiteData storage deviceProgramming languageEmailConfiguration spaceFormal grammarLibrary (computing)Domain nameSoftware developerRecursionProjective planeProduct (business)Computer scienceHacker (term)Boom (sailing)Game theoryView (database)NumberQuicksortSelf-referenceText editorScaling (geometry)Operating systemWordAbstract syntax treeSource codeXML
09:29
CompileroutputFormal grammarRegulärer Ausdruck <Textverarbeitung>Constraint (mathematics)Asynchronous Transfer ModeDefault (computer science)Proxy serverPasswordInterface (computing)Data bufferLetterpress printingCountingWeb browserTouch typingPhysical systemWindowComputer-generated imageryRead-only memoryDistributive propertySineComputer fileBookmark (World Wide Web)Presentation of a groupForceAddress spaceInternet service providerConfiguration spaceImplementationCAN busMathematical singularityWritingOrdinary differential equationGreatest elementPhysical systemMacro (computer science)Computer fileSampling (statistics)CompilerSingle-precision floating-point formatCASE <Informatik>Bookmark (World Wide Web)Vector potentialoutputRecursionThermal expansionQuicksortGame controllerFormal languageRule of inferenceRegular graphCodeElectronic mailing listConfiguration spaceCartesian coordinate systemFrame problemLetterpress printingExpressionWindowAreaInternetworkingPower (physics)Point (geometry)NeuroinformatikMereologyMedical imagingNetwork topologyWeb pageRight angleMatching (graph theory)Forcing (mathematics)Presentation of a groupFile formatDifferent (Kate Ryan album)Asynchronous Transfer ModePredicate (grammar)Regulärer Ausdruck <Textverarbeitung>Distribution (mathematics)Computer programmingSubject indexingData storage deviceSound effectAtomic numberProxy serverComplete metric spaceNumberWeb browserBitGreen computingSymbol tableString (computer science)Information securityContext awarenessNumbering schemeMultiplication signIntegerDataflowSurfaceKey (cryptography)Bit rateExploit (computer security)Expert systemGodSystem callUser profileFilm editingTelecommunicationDressing (medical)Online helpExecution unitXMLComputer animation
18:21
Letterpress printingAsynchronous Transfer ModeWeb browserTouch typingCountingInterface (computing)PasswordWindowSource codeXMLComputer animation
Transcript: English(auto-generated)
00:39
Hello everyone, my name is John McCurris and I'm gonna give you a
00:45
brief explanation of how we designed our DSLs for the next browser I'll explain to you some of our design goals what we were thinking about matters And hopefully you can get some tips and you can use these Exact same techniques in your own applications to make them more powerful for your users. So
01:06
The first things first. What is the next browser and who am I? Well, the next browser is a it's a list based power browser We often call it the hackers browser It's a totally completely customizable and it can do anything you want
01:22
That's that's the premise of it. You can tweak it in any way So the second question is who am I? Well, my name is John. I'm a computer scientist I'm a next browser developer Obviously, you can see my email here If you want to reach out some questions, you have anything John at Atlas that engineer and you can also check on my website
01:46
HTTP colon slash slash John dot Mercurius dot online so before we get into How you can use DSLs in your program to make them better for your users. The question is what is the DSL?
02:03
So a DSL According to Wikipedia, it's an interestingly recursive definition They say a domain specific language is a computer language specialized to a particular application domain That's kind of wonderful Because it's self self referential. It doesn't really say anything
02:22
so Why is that? Well, that's because Actually a DSL is a really ambiguous thing. It's actually any small language That is tailored to a specific purpose It's a language that is limited in scope. It doesn't have a lot of bells and whistles. Maybe it has a small standard library
02:42
Maybe it only has three or four Functions exposed. It's incredibly tiny. That's the key of a DSL to be very tiny to solve a specific problem DSLs are often employed by end users or When I say users, I mean engineers
03:02
programmers sometimes even actually Non-technical users as well though they do Some people do try to make DSLs simple enough that literally anyone without any formal training in programming language usage compilers, etc can use them so
03:25
Why do these languages exist? Well, the first one obviously is they're incredibly simple right So that makes them really good for Configuration files things that the user is touching directly or things that the user is gonna look at right? They're gonna see okay
03:42
We have this specific Settings file, let me tweak some some parameters here. Maybe there will be some settings file for a game The word they can edit some arguments and change the behavior another thing that you may be more familiar with as an engineer is
04:01
Document markup, I'm sure many of you have used markdown or org mode, which is actually what we're looking at right now so this is a org mode document with its own DSL and This is extremely useful and of course the the most famous HTML so
04:20
Okay, we know what they do we know who uses them Now the question is why would we use a DSL? Instead of a general-purpose programming language, why are we kind of restricting ourselves to a DSL? Well number one, they're incredibly simple When we make a more complex language when we expose more complexity, they become harder to learn and harder to use
04:45
Because we can design them Ourselves for our application. Usually we can tailor the syntax and make it very straightforward Specifically for what we're trying to do next is a
05:01
Reality is that we don't typically want to ship a complete compiler bundled with our application this would be Pretty dangerous for a lot of reasons if you can compile arbitrary code Within your application you open up a gigantic attack surface
05:20
for people to exploit and And Additionally, we don't want to embed a compiler within our program because it's a gigantic undertaking compilers are very complex things compiler tool chains are also very complex and Going from a piece of source to a binary is a really non-trivial thing to do
05:43
so Make a DSL we don't have to make a crazy compiler. We don't have to make You know, we don't have to add Alexa tokenizer Make ASTs and do all this is this crazy stuff. We just look for some key value pairs and about it being about a boom
06:01
We have ourselves a DSL that's perfectly tailored for our application Okay, it sounds good so far I can understand why people are doing these things and This is the product that comes out of it is something exactly like the Python ini files So it's just a simple almost key value store where we have some simple string substitution going on
06:24
We have maybe some sections And this is great for 99% of use cases. This is amazing, right? We've allowed the user we've exposed a whole bunch of you know, extra fancy switches to them Well having to make a complicated UI in fact this also scales
06:41
Infinitely because with the simple text editor you could search for a particular argument or or or value rather You could search for a particular variable and do whatever you need to do However, if you want to do something a little bit more complex than assignment, that's when things start to get a little bit more tricky and
07:02
When you're making the you know, or rather when you're trying to make the world's most hackable browser Now we start to think hmm, maybe we would be limiting ourselves a little bit if all we could do was was a Set the values of variables That might be a little bit tricky. So let's let's talk about it. What is the radical approach?
07:26
So we had a couple of constraints that we discussed that were based on the models of typical applications Now because we're you know in a very atypical application our constraints are significantly different. So what if we have a compiler, right?
07:41
what if we have The ability to compile any form within our Compiled program so next could come with a compiler, right? Why not? It's not too hard to do that in this Okay fair enough. So the next thing is well, what if we trusted the user right? What if we gave them the power the
08:02
responsibility to Evaluate any arbitrary list. Well, what would that be? Like? What could they do and What if we didn't have to invent a new syntax, I mean lots of people have already spent a lot of time perfecting syntaxes and Thinking about it. Why do we have to invent yet another one? Why can't we just reuse the existing work of someone else?
08:24
Well, maybe we can so let's kind of go through these points a little bit more in depth So the first kind of sub point is about using lists syntax so rather than Coming up with a whole new syntax Let's just use lists syntax right because lists syntax allows you to represent any arbitrary data and
08:44
Any arbitrary program so we've covered everything that we could possibly want to do, right if we want to save Data or I mean technically everything is data or code We can do that. No problem. We don't have to add any anything and
09:01
Additionally one of our kind of desires of making a DSL is incredibly simple syntax And I don't think it gets any simpler than Lisp So what is really? list syntax like well, it's just you know, a bunch of s expressions and an s expression is just an atom and then you can think of it as just a
09:24
Form a form is composed of atoms or other s expressions Wrapped in parens. So this is kind of another recursive definition But it's it's it's only two rules and just to clarify something an atom is you can think of it
09:44
there's a couple of things that can be atoms but Simplified for your for your thoughts think okay. We have maybe a string but we have an integer. Maybe we have a symbol These can be atoms and they can be nested infinitely with expressions
10:00
so We have a simple syntax, right? So the next step is a compiler and I'll kind of explain how this works. And again purpose of this Presentation is to show you our approach and help you understand it. I'm simplifying a lot of things and I'm not
10:22
necessarily talking with the you know, the most correct list terminology for the sake of effective communication, so You can think of it like this we have a list program running and Then we just take a snapshot of everything. Okay, and we save that snapshot That's what compiling a list program for distribution is
10:43
We start with the compiler We go compile this compile this compile this compile this compile this compiles a whole bunch of things And then we save the whole world including the compiler in a specific file So now we have a list compiler. This is pretty standard Okay, there are ways to not have the compiler in the list of image there are ways to remove things there's tree shakers
11:05
We're not going to cover that but stuff is too safe for next We include the compiler in our list image because we get it for free for no extra effort on our part so The final point and this is the most difficult one to swallow, right?
11:21
You're trying to make a secure application or trying to make a reliable Application remember one that's not gonna suffer from so many potential exploits, right? You don't want to open up a huge gigantic surface area. I Get it But in next we trust user input. It's their computer They're allowed to do what they want to do with it if they're gonna do all sorts of mistakes. That's fine
11:44
Ultimately, we give them the power. Of course, they could choose to do nothing with their user input files and And You know reduce their risk or they could do whatever they want. They could copy and paste blindly without examining snippets of code from the internet nonetheless
12:00
Let's let's just consider that for our application. We trust user input and let's see what happens You know, what did we gain from all of this? so every single configuration file in next is Literally just lisp everything and
12:20
You can embed any arbitrary list code you want and then any configuration file so in the right hand Well Emacs window Regular person frame we have an example of a configuration file. So Rather than just allowing the user to set arbitrary keys and values. We let the user write
12:43
Literally anything so here I have a sample where I've defined a command this is a lisp expression that we could have also written to next and now I can run the command hello and It'll prompt me for something and it'll print it right
13:01
This wouldn't be possible if it was just an I&I file We can't anticipate all the potential use cases for next we cannot Anticipate how someone is gonna need to mold it to their workflow so Every single literally every single configuration file is lisp
13:21
And that extends to everything so I provided here a little sample from our bookmarks Our bookmarks file is just lisp again. If you wanted to you could make a macro that does all sorts of expansion and generates a whole bunch of Bookmarks you could do that. You could have another macro that fetches a whole bunch of
13:44
URLs from the internet and populates your bookmarks So let's go a little bit more practical beyond just the initialization file so the initialization file is an example of someplace So the initialization file is the init.list file here on the right
14:01
The user configuration. So the init file is an example of something that Typically would have been an I&I file. That's a pretty pretty common use case people can understand that right they have a Emacs init file where they can arbitrary execute arbitrary lisp. That's fine. That's fine. But there are use cases
14:21
outside of the init file for example, we have a mode index called auto mode and auto mode It's effectively a predicate that enables and disables different modes for different URLs or hosts or whatever
14:41
because We don't know how the user wants to enable and disable modes, we allow arbitrary lisp and This sample that I provided here We have you know tomato cucumber salmon. These are exact matches. So when we visit the page cucumber it's gonna enable dark mode the Wikipedia page on cucumbers, but
15:06
There are more cases so a Great example of this is actually we had one user who wanted to Enable proxy mode for onion addresses, right
15:21
Because we didn't limit them because we didn't say to them you have to follow this format or whatever They have provided they made their own predicate. They used regex in this case so match regex and then they said then then start by proxy and then they can do all sorts of You know things like disabling force HTTPS or whatever
15:42
So that's the power of allowing the user to execute arbitrary lisp in files outside of just the initialization file so and I wrote this here to emphasize the user can write any
16:02
arbitrary predicate using the complete lisp language Amazing at least I think it's amazing, so What's the whole point number one? We didn't have to write a compiler. We use the compiler Sbcl specifically, it's already fast. It's secure. It's efficient
16:22
Secondly, or maybe thirdly Our users don't don't find themselves hitting a dead end with a particular DSL because of something we didn't anticipate It can use all of lisp to do whatever they want to do whenever they want to do wherever they want to so I don't know I think that's kind of
16:46
Kind of important personally when you again when you're trying to make a you know The most hackable browser out there. Anyways, the next thing is We don't invent a new language that the user has to learn, you know They may already be familiar with s expressions another lisp
17:03
guile scheme Something like that It's a simple syntax. It's the same thing. It takes minutes to learn and it's incredibly powerful and Finally and most importantly for our context the users are fully in control of their system
17:22
so I Hope that you know that was useful for you. I hope you kind of understand why we didn't Try to reinvent the wheel why we decided okay Why make another DSL? When we can just use this amazing wonderful compiler already built into our system
17:44
Maybe it makes sense for you to maybe you can use a a general purpose language as well for your configuration or for your data store or whatever Because you saw we're using also for our bookmarks we're using it also for our auto mode rules
18:02
all sorts of actually, even our history is a list file anyways, the point is Hopefully that's interesting to you. Hopefully it gives you some ideas. I hope you enjoyed. Thanks for listening Thank you so much for your attention and for your time, and I will catch you later. Bye. Bye