We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Idioms for building distributed fault-tolerant applications w/ Elixir

00:00

Formal Metadata

Title
Idioms for building distributed fault-tolerant applications w/ Elixir
Title of Series
Number of Parts
163
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
This talk will introduce developers to Elixir and the underlying Erlang VM and show how they introduce a completely new vocabulary which shapes how developers design and build concurrent, distributed and fault-tolerant applications. The end of the talk will focus on Elixir and include some live coding.
28
30
76
128
MultiplicationAsynchronous Transfer ModeServer (computing)Integrated development environmentThread (computing)InformationWave packetComputer programmingFormal languageMultiplicationSoftware frameworkVirtual machineProduct (business)NeuroinformatikGoodness of fitTraffic reportingRight angleDirection (geometry)SoftwareWritingExterior algebraCore dumpAreaRevision controlPoint (geometry)Web 2.0Condition numberFunctional programmingThread (computing)BitProgramming languageRuby on RailsMultiplication signImperative programmingSoftware bugConcurrency (computer science)Web applicationComputer animation
Computer programmingFunctional (mathematics)Transformation (genetics)Web browserBefehlsprozessorIntelRead-only memoryFunctional programmingSoftware maintenanceSoftwareProduct (business)Moment (mathematics)Server (computing)TelecommunicationConnected spaceState of matterMobile WebCASE <Informatik>Point (geometry)Different (Kate Ryan album)Virtual machineObject (grammar)Cartesian coordinate systemErlang distributionProgramming languageBuildingConcurrency (computer science)BlogInformationFormal languageFunctional (mathematics)Thread (computing)FacebookMessage passingCore dumpOrder (biology)Operator (mathematics)Software bugMaxima and minimaMathematicsElement (mathematics)Web applicationPhysical systemAsynchronous Transfer ModeRun time (program lifecycle phase)Latent heatSet (mathematics)System callMultiplication signInternet der Dinge1 (number)RoutingComputer programmingSemiconductor memoryRight angleNetwork topologyWeb browserLevel (video gaming)Process (computing)Chord (peer-to-peer)Transformation (genetics)MereologyMedical imagingClient (computing)DatabaseAlgebraic closureCAN busSingle-precision floating-point formatStructural loadComputer architecture19 (number)Mobile appFile formatDistributed computingInstance (computer science)JSONXMLUMLDiagram
Cartesian coordinate systemOrder (biology)Virtual machineBitDifferent (Kate Ryan album)MereologyState of matterFault-tolerant systemCodeView (database)Right angleWordJava appletTheoryComputer programmingJSONXMLUML
CodeProcess (computing)Core dumpMessage passingMultiplication signGoodness of fitImperative programmingZoom lensOrder (biology)Run time (program lifecycle phase)Physical systemCodeVirtual machineOperating systemWeb 2.0Connected spaceMereologyStatisticsCartesian coordinate systemEmailInformationDatabaseSequenceThread (computing)Software frameworkLibrary (computing)Crash (computing)Exception handlingNeuroinformatikIndependence (probability theory)Point (geometry)Concurrency (computer science)Block (periodic table)Server (computing)Office suiteWeb applicationFormal languageReal-time operating systemOcean currentPlastikkarteRight angleCASE <Informatik>Video gameFile formatDemosceneDivisorJava appletCategory of beingSystem callMathematicsComputer animation
Process (computing)Message passingConcurrency (computer science)Distribution (mathematics)MereologyMultiplication signPerspective (visual)Product (business)NeuroinformatikFormal languageOperator (mathematics)Process (computing)Computer programmingVirtual machineState of matterLoginNetwork topologyMessage passingConnectivity (graph theory)Concurrency (computer science)FreewareException handlingCategory of beingPhysical systemSoftware bugType theoryCartesian coordinate systemComputer fileFault-tolerant systemSoftwareBootingRun time (program lifecycle phase)AbstractionCommunications protocolWordCodeVirtualizationStrategy gameGroup actionUniform resource locatorDatabaseSlide ruleDemosceneField (computer science)Arc (geometry)Dependent and independent variablesArmExtension (kinesiology)Basis <Mathematik>Natural numberLink (knot theory)DivisorSystem callTheoryInsertion lossCASE <Informatik>Crash (computing)Table (information)Roundness (object)Right angleBuildingReplication (computing)Real-time operating systemAreaGraph coloringTime travelFile systemComputer animation
Meta elementPattern languageFormal languageModule (mathematics)Software testingMathematicsOperations researchExecution unitCASE <Informatik>Computer programmingRun time (program lifecycle phase)System callLibrary (computing)Virtual machineCodeSoftware developerGoodness of fitSpacetimeCASE <Informatik>Formal languageSampling (statistics)WritingSoftware frameworkCompilation albumMacro (computer science)Execution unitSoftware testingProgramming languagePattern languageNeuroinformatikComputer configurationSyntaxbaumDomain nameDifferent (Kate Ryan album)Series (mathematics)WeightExpressionRepresentation (politics)Error messageModule (mathematics)Task (computing)WebsiteFreezingFamilyComputer scienceNumberRight angleTraffic reportingLetterpress printingCausalityLocal ringView (database)Link (knot theory)Operator (mathematics)JSONXMLUMLComputer animation
Error messageElectronic mailing listTraffic reportingSoftware developerBitSoftware testingDomain nameTask (computing)Run time (program lifecycle phase)Category of beingDynamical systemQuery languageLink (knot theory)Type theoryDatabaseInjektivitätFormal grammarMereologyCodeLibrary (computing)Macro (computer science)WeightMultiplication signAuthorizationFormal languageCompilation albumInterior (topology)Right angleDomain nameProduct (business)Military baseGroup actionBeat (acoustics)ExpressionJSONXMLComputer animation
Domain nameProduct (business)Formal languageOverhead (computing)Different (Kate Ryan album)Arithmetic meanRight angleSoftware developerMultiplication signFreewareCuboidTerm (mathematics)Mixed realityData managementSocial classSoftware frameworkDemo (music)CodeSoftware testingComputer programmingRadical (chemistry)Type theoryOrder (biology)JSONXMLUML
CompilerSoftware testingModule (mathematics)CodeDatabaseValidity (statistics)String (computer science)Codierung <Programmierung>Source codeDevice driverMereologyInformationInteractive televisionErlang distributionThread (computing)Table (information)Distribution (mathematics)Execution unitCASE <Informatik>Random numberComputer virusIntegrated development environmentoutputPhysical systemLogicTotal S.A.Link (knot theory)Maxima and minimaPointer (computer programming)Multiplication signGoodness of fitType theoryCASE <Informatik>Functional (mathematics)File formatInformationProjective planeConfiguration spacePhysical systemDirectory serviceLibrary (computing)CodeGastropod shellRight angleSocial classComputer virusPoint (geometry)Module (mathematics)Network topologyStrategy gameInteractive televisionIdentifiabilitySoftware testingPrincipal ideal domainMereologyCartesian coordinate systemSemiconductor memoryFeedbackMixed realityProgramming languageProcess (computing)Online helpIntrusion detection systemState observerText editorString (computer science)Message passingKernel (computing)Run time (program lifecycle phase)Crash (computing)Ferry CorstenVirtual machineLetterpress printingSoftware developerRevision controlStructural loadElectronic mailing listState of matterLeakBlogProduct (business)Wave packetPhysical lawCategory of beingTorusSystem callTraffic reportingAreaFormal languageBitPlanningCoprocessorForcing (mathematics)NeuroinformatikComputer animationJSON
Computer virusSource codeState observerDemo (music)Software engineeringTask (computing)Content (media)Reading (process)Streaming mediaTelephone number mappingIterationSlide ruleInfinityTraverse (surveying)Right angleWave packetComputer fileLine (geometry)Multiplication signMultiplicationCASE <Informatik>Source codeInformationStreaming mediaProgrammschleifeWebsiteTheoryElectronic program guideSheaf (mathematics)TouchscreenDifferent (Kate Ryan album)Programmer (hardware)ResultantGroup actionShape (magazine)Pattern matchingMedical imagingProcess (computing)Module (mathematics)State of matterWeb applicationFunctional (mathematics)Link (knot theory)Element (mathematics)Task (computing)Level (video gaming)Operator (mathematics)Sign (mathematics)Category of beingMetreSystem callContent (media)DemonParticle systemDependent and independent variablesPattern languageAreaTrailCommunications protocolSimilarity (geometry)Arithmetic meanWeb pageMereologyECosOvalProduct (business)Series (mathematics)BlogVideoconferencingIdentifiabilityMatching (graph theory)Reading (process)Software developerVirtual machineBooting2 (number)Network socketGame theoryComputing platformTorusPhysical systemOpen setWritingElectronic mailing listAbstractionCodeIterationData storage deviceNeuroinformatikNumberObject-oriented programmingPrincipal ideal domainCartesian coordinate systemDirectory serviceTelephone number mappingLatent heatWeightProgramming languageTupleDirection (geometry)Computer programmingFormal languageClassical physicsComputer animationJSON
Transcript: English(auto-generated)
Hello, everyone. Good morning. Thanks for coming to this talk. This talk is about the Elixir programming language. And before we get into the language per se, I want to tell a little bit my journey and how I got into functional programming
and how I got into, as we'll see, into the early virtual machine and into Elixir. Who here is familiar with the Ruby on Rails framework? Ruby on Rails is a web framework that runs on the Ruby programming language,
and about, at this point, six or seven years ago, we released version 2.2, and that version was saying that Rails was tried safe. And what was happening here is that the Rails core team knew and the community knew
that our computers are not getting any faster, so now if you want to deploy our web application into production and get good performance out of it, we need to be able to use all the cores on that machine. And using all the cores, to be able to use all the cores in the machine when you're using an imperative language
like Ruby or C Sharp or Java, Python and so on, we usually use threads, right? Threads are our main abstraction. So we now have threads, and threads are going to allow us to use all the cores on our machine, but a lot of the software in those languages,
when we start using threads, it actually doesn't work, right? Or we can corrupt the data structure, or we can have race conditions. So being thread safe here meant that you could get your web application, run into production using multiple cores, and it's not going to break, okay?
And after I joined the Rails core team, this was before I joined the Rails core team, I actually found out that Rails was not really thread safe, right? Because how can you actually prove that something is thread safe? It's really, really hard. So we got reports from people using Rails in production with threads,
and then from time to time, they had some failures, right? The so-called Heisenbugs. When you look at it, it's not there anymore. So we had all those bugs that they were very hard to find, hard to reproduce, hard to fix, and the whole experience was frustrating. It was actually quite hard. So one day I was thinking,
well, if concurrency is becoming more and more important, our machines are getting more and more cores, we can deploy easily to machines today with 24, 32, 48 cores, right? So if that's becoming more important, and the tooling where I am right now, it's not good. It's very hard to program.
I don't feel productive and so on. There are probably alternatives out there, okay? And that put me on the mindset to start looking what other people, other communities, other languages are doing to solve this problem, to be able to write software that's going to be elegant and be able to leverage the direction our computers are going today, okay?
And it was on this journey that I found out about functional programming, and we have a lot of talks about functional programming. We had great talks yesterday. We're going to have even more today and tomorrow. I'm not going to go into that, but to me there are two things that make functional programming great for handling those kind of issues.
The first one is that the state is explicit instead of implicit. So if you think about objects, right, the point of objects is exactly to hide the state. We have the state inside the object, and then we have the methods around it, and then sometimes we have the object that has another object that has another object, right?
So the state is always implicit. It's getting hidden in layers. And the reason this is bad is because a lot of the concurrency issues we have is exactly when you are trying to change the state, right? So we can think we have two threads running on different cores at the same time, and when they try to change the same state, or when they assume some order of operation in that state,
it's going to start to run into trouble, okay? So if we stop hiding the state and make it more explicit, it's going to be much easier for us to find those bugs because we don't need to go layers deep, right? The state is explicit, it's easier to see it and easier to reason about it.
And the other one is that if the trouble is in having two things, trying to change the same state, what if we avoid changing things? The maximum we can, we avoid mutating, change that place memory, and start thinking more about transformation, right? So we are no longer going to
change an array. We are going to create a new array with the new element we want to add. And when I look at those two things, I was like, wait, this makes sense. This, like, if I was using those those concepts, if I was following this philosophy and writing my Ruby software at the time,
I would say, yes, this would make things definitely easier. So after I got that, I said, okay, I kind of had like one foot into functional programming, right? So I said, okay, let me look at what functional programming languages are doing, and now when I was exploring those languages, I like to say I fell in love with one,
which is Erlang. Who is here familiar with Erlang? Okay, that's great. Just some of you, so I think this is great because you're going to be very pleasantly surprised, okay? So Erlang is a language and virtual machine created by Ericsson. And Ericsson is a telecommunication company.
And what is very interesting about Erlang is that when they designed language, when they designed the runtime and the virtual machine, they had a very specific set of requirements and use cases in mind. For example, what they are doing as a telecommunication company is that they are building telephone switches, right? So what a telephone switch needs to do is that
if you want to call someone, if person A wants to call person B, the telephone switch is going to connect them, and now they are connected and they can talk and start exchanging information. What we really want, if you are a company that is building and selling telephone switch, you want to build telephone switches that are able to handle as many connections as possible, right?
A lot of people should be talking at the same time for that switch, and the switch should be fine with that, to handle all that concurrent connections and all that load. So we start to get an idea like concurrency was at some point one of the requirements they had that, but they had other ones. For example, sometimes
you want to call someone else, but that person is already busy talking to another person for another switch. So the switches, they actually need to be able to talk to each other and exchange information. Hey, I want to route this call for you, or is this person busy, and so on. So they said, okay, we need to have now a lot of machines actually
that's running Erlang, and those Erlang instances, those Erlang nodes, they need to be able to communicate with each other and exchange information. So it started to look more into like a distributed system, right, where information is coming and going through different machines. So that's another requirement they had, and they had a bunch of other more. For example,
sometimes you need to kind of go in the middle of nowhere to deploy a telephone switch, and you don't want to, you know, it's expensive to be sending someone back every three months to give maintenance, right? You don't want that. So we want the system to be robust as much as possible, and because, you know, telecommunication is basic infrastructure, right, it should be up and running all the time.
So we want really a lot of reliability. The system uptime should be great. There is another requirement which I like to make jokes about, which is we don't receive phone calls from our, you know, telecommunication companies saying, hey, we are going to be in maintenance mode from midnight until
1 a.m., so please don't use your phone, right? That should not happen, but, you know, you have a software. Sometimes you need to update the software. So they figured out a way to update the software that's running production live without bringing the system down, without even disconnecting people that are talking at that moment right now, okay? So this is early, and then it was
built for telecommunication, and it was used for telecommunication for a while until, I think, about from 2005 to 2007, people realized that this use case we have here is not really different from this use case, right? Instead of having a telephone switch, you have a server, and you want to handle as many connections as possible,
and sometimes those connections, you know, they are just people accessing a web application with a browser, but sometimes you are connected to embedded devices. Internet of Things is very popular lately, right? Or native mobile applications, whatever you want to, okay? So it's the same. Now we have a server. Sometimes we need the server to communicate with
other servers, or to communicate with internal endpoints. We have a thousand different clients, okay? And people realize that, oh, if it was good for that, it's going to be good for this, too. And then they started to build a lot of interesting things. So those are some companies that are doing interesting things with Erlang. So we have Amazon, Facebook,
we have telecommunication companies like Ericsson and Motorola, more like recent companies like Heroku and RIAC, which is a database. It's written in Erlang 2. If you ever hosted your application in Heroku, their routing layer is written in Erlang, so it goes through Erlang nodes in there. And there is one new case that got
very, very popular lately, which is the WhatsApp case. They got a lot of attention because they were acquired by Facebook by 19 billions, and they used Erlang and they were always very open about their architecture. So they wrote blog posts of how they were able to have two million connections
on a single node. So WhatsApp is a messaging application, you start on your phone and you can exchange messages with your friends. So here what they were saying is that they had two million devices connected to a single machine and they were sending and receiving information. And then when you go into the details of the machine, you can see, oh, it's a very powerful machine, it has 24 cores,
it has 96 gigabytes of RAM, but even though it was using just 400% of the CPU and memory with that load, but when we look at it, this is exactly what we want. We want to write a software that's going to run as efficiently as possible my machine, so it has four cores, it's going to use all the four cores to do the job, but when I put it into production
and I'm giving it a very resourceful machine, it's going to use that as efficiently as possible. So, that's about the Erlang. And Elixir is a programming language that runs on top of this virtual machine.
the same way we have Java, Scala, Clojure running on the JVM, we have different languages running on the CLR, Elixir is one of the languages that run on the Erlang virtual machine. And as you're going to see, Elixir is a functional language, but that's not really what I want to focus right now, because more than a functional
programming language, it is a concurrent programming language, and more than that, it's a distributed programming language. So, this brings us to the second part of the talk. That was the introduction, and I hope I convinced you that the virtual machine is a very interesting software, how it came to be and how it is used, and it's
worth investigating. And with that said, if Elixir is a functional, but more than that concurrent, more than that distributed, it means that we can write very interesting applications with that, but in order to do that, we need to learn kind of a new idiom, we need to learn new words, and we need to
think about things a little bit differently. And that's what we're going to do for the second part of the talk. We are going to explore those idioms, and as we are going to see, those idioms, they're not only for building distributed systems, but it's about building fault tolerant applications, applications that when there is a fault in the application, the application will be able to self-heal and go back to
a state that is stable. So let's start, let's talk about talking about those idioms. So when we program in Ruby, C Sharp, Java, C, Python, C++, this is our code, right? It's just a chunk of sequential
code. And that's why threads are so hard, because when we are using threads, they are really interleaving in the sequential code, so we need to look at the code and say, hey, there are many things running at this code, we need to think which order they run and so on. It's very hard to reason about that. So that's it, we just have a chunk of
sequential code. And in Elixir, it's not really different. We have sequential code too, but the sequential code is actually, we put in something very very tiny, called process. So that's the first word that we're going to learn. Process. And when I say process for the rest of this talk, I don't
mean an operating system process, I mean a virtual machine process. It's very lightweight, it's a very cheap thread of computation. So, for example, just to have an idea of how cheap this is, in the WhatsApp case, they had two million connections. They had at least one connection per process, which means they had at least two million
processes, okay? So we can create a bunch of them. And that's what we do. So we have all those processes, right, that are executing some sequential code, but it's interesting that those processes are isolated. They are all running independently. And what it means is that if they are isolated, they don't really clash with each other, it means that we can
run all of them at the same time. And that's how we already start to leverage concurrency because we have processes that are isolated, and they are running at the same time, right? So if I have ten processes and I have four cores, the run times say, hey, now it's time for this process to run on this core, this other process on this core, and it's kind of multiplexing the process on the cores.
And because they're isolated, we get this functionality, but sometimes we want those processes to collaborate towards a common goal. And when they do that, they do by exchanging information by sending messages to each other. So every time the process needs to exchange some information, they are able to send
messages. So now we are thinking about processes, right? And we are thinking that those entities, they are concurrent and they are isolated. And that's a very good first step, but it's really the beginning only. So let's choose, so we have all those processes running, let's choose four and give a zoom to them in order
to understand what is happening here. Let's imagine also that all those processes, they are running in a node that is a web application, so we are going to have things like I need to be able to access the database, so we are going to have a process that is able to communicate with the database. We have another process that is receiving the
TCP Acceptor that is receiving connections and handling the requests. We have another one which is calculating statistics. For example, we want to know how many users are connected, so you created this process to keep live information about your running system. And then we have another process that is responsible to send emails, because sometimes the user
joins your application and you want to send them a welcome email or an email with some code and so on. So we are saying that those things are isolated and this brings a very interesting property, which is if one of those four processes crash, because they are isolated, the others
are not really going to know about it. They are not going to be affected unless they want to. And this is really important. In order to show why, let's go back to an imperative language. So in an imperative language, for example, I know this is true in Ruby, I know this is true for all the web frameworks and libraries in Java, in
Python. What happens is that every time you have a request in those languages and your library, it calls your code. But something can go wrong in your code. For example, when you try to send an email, maybe the mail server got temporarily unavailable and something can go wrong while trying to send that email and that's going to make that request
crash. But in those languages, when the mailer crashes, when sending an email, you can get an exception, right? And this exception is going to bubble up, bubble up, bubble up. And then if you don't do anything to that, it's going to bubble up to the point where your whole system is going to calm down. The whole system is going to crash. The whole node is going to crash.
And now, think about it, how can sending an email make all the parts of your application crash in those languages? It doesn't really make sense, right? So what those libraries do, the reason we don't see that, is that those libraries, they put try-catches or they put rescue blocks to ensure that anything that goes wrong in your code
is not going to propagate and bring the whole thing down. But I find that a very dangerous idea, because errors, exceptions, is one way of your system to say, hey, something went wrong, right? From the name exceptions means something that was not supposed to happen happened. And all we are doing
is rescuing those things and saying, hey, thank you for this information, but just continue doing whatever you're doing, right? It's like you're going to the doctor and saying, hey doctor, I'm not feeling well. And then the doctor says, thank you for the information, go on about your life, right? I'm not going to do anything about it. So it's a very risky idea, and here what we do is the opposite,
because if they're isolated and if the mailer crashes, nothing goes wrong, right? What we can do is say, okay, if there's something wrong in the mail, just let it crash, fail fast. And what we are going to do is that for our system to go back to work, as it's supposed to be, we have supervisors. Okay, so here I have a supervisor
supervising the web and stats processes, but the DB can have its own supervisor too, the mailer is going to have its own supervisor, and then when something goes wrong with sending the mail, what we do is that, okay, let that mailer process crash, the supervisor is going to see it and is going to react on it and is going to start another mailer
process to do everything the mailer process was supposed to be doing, okay? So that's what we do, right? We say, okay, if something goes wrong, it's fine, you can crash. We have a supervisor and it's going to notice it and it's going to have a strategy on how to bring that mailer or that part of our application back, okay? And why restarting is such a good
idea, right? Why, for example, we are programming and then our computer starts to act weird, and what do we do? We restart, right? And then, bam, it works magically, right? And the reason for that is that the people who designed the operating system, the booting process of our machine, they tasked that booting
process to exhaustion, right? When we restart, we put our machine back to the initial state that's guaranteed to work, right? And this is the thing we are doing, right? Because maybe the mailer crashed because it went into a state that it's not supposed to have gone, so when they restart it, it's going to go back to that initial state that's the one we tested, right? That's the one
we verified to exhaustion. And then if it crashes again, because it went into a bad state, it crashes again and again, we are going to have the logs and then we connect on it, okay? So that's what we do, we have those processes, and then we have, we define the supervisors, sometimes the supervisors have their own supervisors too, and we start to have those things called
supervision trees, and then at the end of the day we package all of these inside an application, okay? So we get one particular tree and then we put it inside the application, and that's how we ship our code. So if we're talking about idioms, we've learned a lot of new interesting words, right? Now we learn about processes, we learn about supervisors
which are just processes with some special responsibility, but we're also talking about applications, which is how we package our code and how we start and stop each component that runs it, okay? Runs in it, runs in our system. And we're also talking about message passing, that's how different processes communicate. And we got
a lot of interesting things from it. For example, now we are talking, we see that by just using processes, we kind of get concurrency for free. Another consequence of the system is that we are no longer saying, like, catching exceptions or rescuing errors, right? That's a bad idea. We are going to say, hey, things are going to go wrong. And what we are
going to do is that we are going to have supervisors that are going to act on that. And what this does, another property that this brings to us, that our systems they start to get more fault tolerant. And why is that? Because it doesn't matter which language you use, you're going to have bugs in your software, right? Things are going to go wrong. You can use the most sophisticated type system
in the world, right? You're going to have bugs. Your application is going to have errors. Even more when you start factoring like writing to the file system, talking to a database, doing things over the network. So we can't really catch all the errors, handle all the errors, catch all the bugs. If you want to have an application that's going to be running
all the time, we need it to be able to react to failures, right? If something goes wrong, it's going to be, hey, I'm letting that be, and I'm going to go back to the state that I know it's going to work. And that's how it brings the idea of fault tolerance. Our application can really handle failures, right? Sometimes I like to say that our application self-heal, okay?
And that's the idea behind it. And we got all those things, right? And there is something else which is, I haven't talked about, but it's in the title of the talk, so I hope you're expecting me to talk about it, right? Which is how can, from all those things I talked about so far, how does it make it easier to build distributed systems?
So the answer to this is, if you go back to that slide, we have a bunch of processes, and they're exchanging messages with each other. Because the message sending, the message passing is abstracted by the runtime, it doesn't really matter if the process you are sending a message to is in the same machine
as you are right now, or is in another one, right? Everything is abstracted for us. So if I have all those processes here and then you want to send a message to another process running on another node, the runtime abstracts that for us. And we just need to change who is the recipient of that message and say, hey, it's on that node, and then the message is going to
get there. So the runtime takes care of everything we would expect when building a distributed system. It takes care of serializing the messages, it takes care of the protocol between the machines, everything comes for free. So that was it. That was the ideal part of the talk, and
we've seen how we added those new words to our vocabulary. We are now thinking about processes, we are thinking about supervisors, and that brought to us a bunch of interesting properties. But the truth is, everything I said so far, it does not only apply to Elixir, but to all other languages that run the R-Link virtual machine.
It's really a property of the runtime, and whatever language you choose to run on that particular virtual machine, they're going to have all those properties. So you may be asking, okay, so what does Elixir bring to the table, really? And that's the third and final part of the talk, which is about the Elixir programming language, per se,
and what it brings. And every time I talk about the Elixir programming language, I like to talk it from the perspective of its goals. The language has three goals. They are compatibility, extensibility, and productivity. So compatibility is the easy one. It basically says everything I have said so far,
it's going to apply to the Elixir programming language, it's going to remain as compatible as possible to the runtime and virtual machine that we run on. So to have an idea how this affects our code is that you can call Elixir from R-Link and vice versa with no performance cost. So all the libraries that already exist in R-Link,
even more for R-Link developers, you can just call, and there is no cost, there is no bridging, and so on. Okay, so that's good. And then the second goal is extensibility. And the way I like to explain extensibility, I really like this quote from Guy Steele. It's one of my favorite talks of all times, and at the end
of the talk, it's called Growing a Language and at the end of the talk, he says, now we need to go meta. We should now think of a language design as being a pattern for language designs, a tool for making more tools of the same kind. And the way to interpret this is that today, computer science, it's so wild,
we have so many options that it's impossible to design a language that's really going to be able to tackle many of those options efficiently or elegantly. So what we need to do is that a programming language needs to be extensible. We need to be able to get the programming language and
extend it to the particular domain you're working on. And there are many ways that we can do that with different programming languages. And one of the ways we do this with VELIX series is by supporting the ideas of macros. Those who are familiar with
what is the thing that LingQ uses on .NET to learn the computation expressions where they can see the representation. Sorry? Expression trees. Thank you very much. So those who are familiar with expression trees, it's somewhat similar, and I'm going to show some examples, but it's more flexible, and I'm going to show one example exactly of how it is more flexible soon.
So here's how I would write, so this is your first Elixir code. It took us just 30 minutes to get some Elixir code sample. So here's how I define a module. Here's how I would write tests in Elixir. We define a module, give it a name, mathtest, and then we are going to use the X unit
case, which is the name of our test framework. So when I say that, I'm defining this module as a test case, and after I do that, I can say, hey, I want to test basic operations, and then that is the test, and inside this test, I can have a bunch of assertions. For example, 1 plus 1 is supposed to be equals 2. And
depending on the language that you use, after you see this code, assert something, you're kind of like saying, no, no, no, don't do that, right? Because a lot of the test unit frameworks, we cannot use assert. We should use assert equal. We need to tell, exactly, that we are trying to compare two things. If we use just assert, we get a very poor error
reporting, right? And the reason we don't need to do that, we don't have things like assert equal, assert different, assert more than in Elixir, is because assert is a macro. And as a macro, it can actually look at that code, at compilation time, and see what you're doing. So the assert macro looks at that code and says, hey, I can see here that you are trying to compare
two numbers. So when something goes wrong, it can give us a very good report of exactly what you are trying to do and what was expected and what was different, exactly because it can look at the code and see what you are trying to do, right? It doesn't rely on the developer to be giving hints of what's supposed to be done. So that's one way where we got
the language and extend it a little bit to the testing domain. Another domain that we were able to extend Elixir is sometimes we need to talk to the database, and sometimes we want to write queries that are going to run in the database. So in Elixir we have a tool called Acto, which is inspired by Link from .net.
And this is how we could write a query in Acto. We'd say from ping post, and then we get where the publish at time is less than now and the author is just there, and then we want to order by the time the posts were created. And what we're going to do is that from is a macro and is going to be able to look at this code and at compilation time it will bring this as close to SQL as possible.
So at runtime we just need to get this and run it against the database, and it's going to give all the properties that we're already familiar with Link, right? It's going to even for Elixir's dynamic language, our queries we actually do type checking because against some databases it can be a secret issue. We are going to protect against
SQL injection and so on. And the interesting thing is that when they added a link to .net, they actually extended the languages, right? They had to change a little bit C sharp grammar to let it know about how to build all the link expressions, but here in Elixir, Acto is just a library. It's not a part of Elixir itself,
right? So we had a library completely separate from the language, and it was able to add this because of macros and how flexible the syntax is. So those are two examples of where we got the language to say, hey, let's extend it to some particular domain, right? And when we are thinking about that, you already
get an idea of productivity, which is our third goal, because if I can get the language and extend it to the domain I'm working on, I'm going to definitely be more productive, right? It's going to reduce the overhead, the conceptual overhead between different domains. But there are other ways you can talk about productivity and our third goal, and I know that productivity is hard, like how we're
really going to measure that language A is more productive than language B, but there are certain ways that we are going to be more productive no matter what. There is no way we can deny that. So one of those ways, for example, is by having first class documentation. Documentation should be
easy to write, it should be easy to read, easy to access, and so on, right? The language should push us, should gear developers towards writing good documentation. If the language does not do that, it's a flaw in the programming language, in my opinion. Other things that are going to ensure we are productive is by having a very good tooling, right?
We should have a test framework out of the box, so we talked about XUnit. We should have something called interactive LXU, which is where you can go to your terminal and type some LXU and get some stuff out. We have Mix, which is a build tool and you're going to see it soon. And we also have a package manager, because if someone already solved the problem you are trying
to work on, if you can just use that in your code and it works, it's great, right? That's what we want. And if we have those things, there's no discussion. They're definitely going to be more productive. So in order to show some of those, let's do some demo. Actually, so I've rehearsed the talk yesterday, it took me like 45 minutes,
but it took just 30 right now, so we have like 30 minutes of demo, yay! So if you want to see something in particular, just ask, I will start and then if you have some questions, because we have a lot of time for fun. So, okay, before I start,
yes, in the back? Oh, yeah, okay, so you can't see it, yes, that's good. I typed like, can you read this? And then, yes, you can't, right? No, you can't, that's good, that's exactly the feedback. What about now?
Yes, great, okay. So I talked about being productive, and one of the things that we want to do for being productive is like, I installed a programming language and I want to be able to jump into a project as soon as possible. So in Elixir, we have this tool called Mix and it knows everything about creating a new project, running your tests, compiling it,
so I can say mix new foo, right, and that's going to create a project, and then I can just go into the project directory and run mix test and it's going to compile my code and show how many tests it runs, all the time, and so on. Okay, it's going to take care of all this stuff for us.
Something else that we could do, which is I can start something called the Elixir interactive shell and what I can do here is that I can type any Elixir code, right, and try stuff out, so one plus three is four, it's correct, that's good, and I talked about first class documentation.
So what we can do with this is that I can type h for help and the name of any module in my system or any code, and it's going to bring information about that. So for example you want to work with strings, you can type here eight string and it's going to print all the documentation for the string module, okay, and this is great
because as I said, we need to expose documentation on all the ways possible to developers. So in Elixir it's really easy to access that and first we started by doing that in the interactive shell, okay, but now we see people working with IDs because it's very easy
to access this information. It's very easy for them to show documentation on your editor and things like that, right, because it's all accessible, it's all really reachable. And then I can say hey, I have the string module then, what are the functions that are in there and then it shows me all the functions and then okay, I think I want split, what is
the documentation for split? And then it's going to print the documentation with a bunch of examples and something that's very cool is that you can see that many of those examples here right, they have like the same formatting as the shell, and what it means is that we can actually, we have something called doctasks which means that if you have examples in your documentation
we test those too, so I guarantee that your documentation is up to date. Because I have a lot of time I can actually show it working so this is let me maximize this so this is the project we generated, right, it has
a configure directory which however we're going to configure stuff, sorry I cannot make this larger, but it has a lib directory which is where we put our code and our test directory. So if you go to the lib directory, we can see it has a module okay and we can write the documentation here, so
we write the documentation like this this is my module okay, and now if I go back to the shell I can reload that module and then I can say hey, what is the documentation whoa, no documentation file, hold on and then I can say show me the documentation for foo
and then it prints hey, this is my module so I just wrote the documentation and I can access it if I go back there I can, we can write some tests so I can say one plus one is three, let's make it fail on purpose. The documentation uses markdown which is kind of very common for developers to know, so I wrote
like some tests in my documentation and now I can say I want you to doc test the module foo and after I do that if I leave this and run my tests we are going to get a failure, right because one plus one we said that we expected it to be three so that was our code, we expected one plus one to be equal three, but we got two
right, on the left hand side so our documentation test failed. So here's just some ideas of how documentation is first class and the build tool allows us to be really productive and not really worry about wire things up, it can just do the work we are supposed to do. Something that's really nice showing also, so I was talking about distributed systems
so how easy really is to get two nodes to talk to each other so what we can do here is what I'm going to do is that I'm going to start an interactive Elixir shell, as I was doing before, and this time I'm going to give it a name I'm going to give it a name of foo. So what I'm going to do
here is that I'm going to define a module called hello with a function called road I know this is a little bit cliche, but you're going to see there is a point and all this function does is that it prints hello world so I define the module and then I can print
I can call hello.road and it's going to call that function and it's printed hello world here. Great. It works. That's good. What I'm going to do here is that I'm going to go to another tab and I'm going to start another node but give it the name bar and when I give the node's name
I'm able to communicate with each other because they have a name now and then I can send messages between them so what I'm going to do here is that if I try to call hello.road it doesn't work because I define hello.road in the other shell, in the other node. It's not defined here but if I call node.spa and give it the name of the node
which we can see here, so this one is bar at macbook and macbook is the name of my machine and the other one was foo at macbook so I say, hey, I want you to spawn a process, we talked about process on that other node and what I want you to execute in that node is this function that calls hello.road and when I do that
we can see that we got hello.road back so what happens is that we went to the other node and said hey, execute this function, we execute that code and then when it was time to print something that other code sorry, that other node said, hey I can see that you want to print something but I can see that the request has started from that other node so all the I.O.
is redirected back to us which is really cool, that's what we want. So we can use this if some of our systems for example, the whatsapp engineers they were working on the application, if some of their systems is acting weird what they can do is that they can log into that system and get information in fact, they have something called
observer-start that okay, so for example, what you can do is that you can log into a node and then you can run this observer application and it brings a lot of information from our system, so it brings what is the run time version how much memory is being used
so right here, it's running when we start a node that's not really doing anything it takes just 19 megabytes and then we can see from each entity how much memory by each entity is being spent on. We can say oh, your machine has four cores right, and a bunch of other information we can get load charts
there's nothing really happening in this application so we're not going to see anything interesting here but we can have a list of the processes, right, of all the named processes and then, for example because everything happens in processes all these states are happening in processes if one of the processes have a memory leak you can say hey, my memory application is growing right, what is going wrong? we can come here and say, okay, I'm going to
order this by the memory usage and then I can see the process that's using a lot of memory I can double click it and find information about the project find information about that particular process. I also talked about applications every time we run our code our code exists only
inside applications and we have many applications per node on this node that I just started we have four applications running we have Alexir IX which is an interactive shell we have the kernel which is the runtime kernel that comes from the early virtual machine and we have a logger when we want to do logging and this is really nice because
with this, we can start to reason about failures too so I have my application, and this is what we are seeing here this supervision tree we are talking about and then we can start wondering wait, what happens if something goes wrong? so we can come here and say what happens if for some reason this logger process crashes
so I can send it an exit message right, and then I can see how things will react so I don't know if you noticed I'm going to do it again but you can see that this tree it starts the tree and then we have four children, right, from the supervisor node so we have that supervisor and that supervisor is supervising some four other processes
and then we have different strategies one of these strategies, for example, you can see that the identifier for this one is 22 24, 22, and this one 24, 25 so one of the supervision strategies that we have is that, hey, if any of the children dies, they can be connected in a way where I can say, if any of the children dies
I want you to restart everyone and we are going to see that when I killed logger that's kind of what happened because you see, I got new PIDs for those processes there, so the supervision says, hey, something went wrong so I am restarting this whole part of our application
and yeah, does anyone have any questions anything in particular you would like to see? or not yet? okay, that's good so so let's move on there are some final slides in here
oops okay, so if you want to learn more about the Elixir programming language if you are interested about this talk you can go to our website alexir-lang.org we have a getting started guide there that's going to take you through many of the things I talked about here right now
and explore what are the language data types, how to define modules, and so on but we also have on the side a learning section where we link to screencasts, podcasts books, and a bunch of different resources so we have many books out there the major publishing companies
they already have books on Elixir available, so we have programming Elixir from Pragmatic Programmers Elixir in Action that was released just last week I need to update the image it's finally out, which is from we have introducing Elixir from O'Reilly and many others, you can check the learning section page finally I want to thank my company
Platform Attack for investing on Elixir and allowing us to create the language, create the ecosystem and so on, and if you are interested in particular about hearing more from companies using Elixir in production we are starting a series of blog posts and videos which is very obviously
named Companies Using Elixir in Production where you can see from a bunch of interesting use cases, for example the last one that came out was from a company using Elixir and running it in embedded devices so they are building embedded devices that need to be deployed to ship boats, right?
and they were able to get Linux the early virtual machine and Elixir in this embedded device and everything boots in two seconds or two and a half seconds it's a very nice use case we have other cases using I like to say that Elixir is going to be fantastic for everything that runs on top of a TCP socket so we have people doing more
of like API backends we have companies building classical web applications we also have a very interesting use case about someone building a whole distributed gaming platform in Elixir, so there are a bunch of those cases and it's worth checking out and that's it, that was the talk about Elixir and thank you
Torben? Yeah, I... So Torben says show me an agent OK, so
so in Elixir I was saying that we have those processes, right? and those processes they really are the main entity we work with and sometimes they're not like many times actually, most of the times we are not creating those processes directly we are creating things
that already have a specific customization or behavior to that process for example, one that's going to be very common familiar to .NET developers is the idea of tasks so with a task, I can say you know, I want to create a process that's going to execute
some particular computation like, no, let's do like so here I spawned a process yeah, I spawned a process which has the task behavior that's just going to execute the code that we give to it right? we also have something like the task
async await pattern so if I want the task to calculate something I can just create a task and then I can pipe that result to task await and it's going to spawn a process to compute that something on the side and when that result's ready I'm going to read it back so task is one of those behaviors entities we have that has
already a certain shape and it makes it easy to have a process that does something in particular and one of those, which is the one that Tom asked to show is the idea of agent and the agent is useful when we all we want to do in the process is to keep some state so for example, what I can do here is that I can
create a new agent and when I create a new agent, I give it the initial state, so the initial state here is going to be zero so what I did here actually is that I called the agent.startlink function using something to calculate the initial state and what that returned to me is
a tuple with two elements OK, which is saying hey did I create the agent successfully and agent, and what we did here was pattern matching because the right side returned a tuple with OK and the identifier for that process, which we call PID and we matched that to extract that information OK
I'll go back to pattern matching soon let's just continue with the agent example, so now that I have an agent I can, for example, say hey, I want you to show me your state, so I can send it the function that's going to receive the state and return the state and now I can read it back, but I can also say I want to update the state
so I'm going to say agent update and I'm going to say that the state is state plus one now I have updated it and it said OK, we're able to update it and now if I run GET, the state's going to be one so that's a very simple process which its only purpose is to keep some state that we need to have our application running OK and just since we talked about pattern matching
so pattern matching, we've seen people talking about it in other talks but it's a very useful way to write, I like to say, pattern matching is a good tool for writing assertive code OK which that means, for example if I want to, let's say read from a file
OK all I need to do is to so when I read from a file what I expect as a result is OK, I was able to read the file and the contents of that file, so I can say file read, I think there is a read me in this directory OK, so actually there isn't, so we could see what happened here so I was expecting
to be able to read a file and open it, but there was no file with that name, so the right side returned, we can see error, inoent, and when we are comparing those two things, the OK contents on the left side with the error, inoent they don't match, so that's a way I can write assertive code
and code that only cares about certain scenarios and then pattern matching to extract information out and so on any more questions, Brian? OK OK, I see you two are trying to put me on the spot
OK so OK, so something else that we can do in Elixir I didn't talk about protocols but Elixir has the the idea, and if you're coming from link you have the same
something similar rather, which is the idea of collections so if I have a list one, two, three, and then what I can do so this is a list with three elements what I can do is that I can map over them, and when I map, I want to multiply each element per two, and bam
I've done that. Then we can do all those things all those things that we all those functional names that we get used with, like map filtering, reducing, and so on, all those operations are there, but there's something very interesting about Elixir is that we have a module named stream, OK
and with stream when we say, for example OK, so let me hold back here, so we have a list right, so I have a list, let's put in a variable and now that I have a list, I can I can do the same, like I can pipe that list to a nummap, but instead of using a nummap, I'm going to use
streamMap and the difference here is that streamMap is lazy it's not going to perform the operation it's just going to hold just going to store that information so we can use it elsewhere so when we execute this code, we can see here that we didn't get a list, we got something called a stream, and I could do a bunch of stuff
I could map, I could filter I could, so for example now that I have this stream, let's get this stream itself and pipe into a streamTake sorry, not a numTake, a streamTake OK, and get one entry, still that didn't execute anything, it's just storing the computations
so when I do a num to list finally, because say hey, I want a list out of it, we can see that it mapped the element and took just one so the stream is a very interesting way where you express the computations, but they are not executed they are lazy, right, and this brings us a bunch of interesting ideas because if we have our data source
which, for example, in this case is a list and I have all the computations, we can start saying hey, if I have the data and the computation, what if when I have to run this computation I spread the computation throughout multiple cores, right? So those ideas allow us to export this direction
it even allows us to work if infinite collections for example, let me steal an example from the documentation so we can have we have this function called stream iterate OK, that let me go to the top, OK
we have this function called stream iterate and all it does is it receives the initial state and then it goes incrementing by one infinitely, right? so if I call a num to list in this, it will never give me a result because the collection is infinite but because the stream operations are lazy I can do things as we did before, right?
I have the stream, so I want to map over it and then, oops, not enum almost OK, and then I can say enum sorry I want to do again stream, stream, I want to
take just five elements and then when I say enum to list with the stream it's not going to do the whole thing, even if it needs collection because other operations are lazy and I know, hey, I don't need to traverse everything I can just
I can just get what I need so, which directory I am? so we can also use the streams for example, to traverse files sometimes sometimes we can work with files as collection shows, so I can say hey, I want you to give me this file as a stream, and when you say take me the first five entries
it's going to return you the first five lines of that file so we have this whole, which I think what Brian wanted me to show, this whole abstraction to work with things as collections, and it can be either eager or lazy OK we are running out of time, we have just five
minutes more, so I want to thank everyone for coming and watching the talk I hope you try Elixir out, and if you want to talk about it, I will stay around the whole day so come chat, come ask questions and so on. There are also other Elixir talks happening both today and tomorrow so Chris and Torben, they are here and they are giving other talks about Elixir
so thank you very much