golang: Practical Go Programming
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 64 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45928 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201121 / 64
3
7
10
11
17
19
21
28
33
34
35
37
40
44
48
49
52
55
57
59
62
63
64
00:00
Computer programmingRead-only memoryCodeConcurrency (computer science)Library (computing)Open sourceFreewareFormal languageStandard deviationHash functionComplete metric spaceSource codeUniform resource locatorMusical ensembleRevision controlKey (cryptography)Denial-of-service attackFunction (mathematics)Data structureThread (computing)Fundamental theorem of algebraGradientComputer virusProjective planeSystem programmingLine (geometry)Type theoryString (computer science)Key (cryptography)Formal languageFunctional programmingQuicksortMultiplicationForcing (mathematics)Level (video gaming)WindowStructured programmingHash functionUniform resource locatorRevision controlCodeSoftware developerData structureBasis <Mathematik>Reverse engineeringElectronic mailing listGoogolInterface (computing)Dynamical systemComplete metric spaceWeb applicationPrimitive (album)Web serviceConcurrency (computer science)Library (computing)ExpressionComputer programmingVariety (linguistics)Thread (computing)Semiconductor memoryArithmetic meanWeb 2.0Semantics (computer science)Instance (computer science)ArmPresentation of a groupMultiplication signIntegerInteractive televisionCombinational logicValidity (statistics)Memory managementMachine codeDebuggerCore dumpEmailMappingProgramming languageExistenceCompilerEndliche ModelltheoriePoint (geometry)Electric generatorOpen sourceInterior (topology)BitCuboidData storage deviceNatural numberProblemorientierte ProgrammierspracheCompilation albumFront and back endsComputer animationLecture/Conference
08:15
Programmable read-only memoryData structureType theorySynchronizationUniform resource locatorString (computer science)Function (mathematics)System callLocal ringLogicDenial-of-service attackConstructor (object-oriented programming)Instance (computer science)Trigonometric functionsPhysical lawWritingWechselseitiger AusschlussMereologyData storage deviceReading (process)Field (computer science)Level (video gaming)QuicksortFunctional programmingStatement (computer science)String (computer science)Data structureResultantData typeKey (cryptography)Variable (mathematics)BlogMultiplication signLetterpress printingUniform resource locatorParameter (computer programming)Pointer (computer programming)Order (biology)SynchronizationStandard deviationLibrary (computing)Type theoryExclusive orProgramming languageMathematicsClient (computing)Cartesian coordinate systemBitSystem callCompact spaceLine (geometry)AlgorithmBlock (periodic table)Set (mathematics)CASE <Informatik>Loop (music)Constructor (object-oriented programming)Form (programming)CodeComputer programmingIdeal (ethics)Web applicationNormal (geometry)Right angleFerry CorstenSlide ruleComputer animationLecture/Conference
16:07
String (computer science)Alpha (investment)CountingUniform resource locatorMenu (computing)Server (computing)Function (mathematics)Instance (computer science)Source codeHand fanTorusGroup actionElectronic visual displayForm (programming)Normed vector spaceDependent and independent variablesStatisticsCodeError messageWeb 2.0Computer programmingRevision controlEmailFunctional programmingCASE <Informatik>Web applicationCountingUtility softwareDependent and independent variablesKey (cryptography)Query languageAlpha (investment)Address spaceSource codeLevel (video gaming)RootNumberInformationLine (geometry)CodeElement (mathematics)LengthData storage deviceOptical disc driveVarianceWebsiteCuboidoutputType theoryParameter (computer programming)Semantics (computer science)Numeral (linguistics)InfinityString (computer science)Reading (process)Server (computing)Letterpress printingImplementationElectronic signatureSet (mathematics)Logical constantStatement (computer science)MereologyComplete metric spaceError messageUniform resource locatorDatabaseIntegerForm (programming)Service (economics)Loop (music)Instance (computer science)Multiplication signUser interfaceBitProper mapInterface (computing)Thread (computing)Front and back endsRoutingMathematicsFerry CorstenCorrespondence (mathematics)Computer animationLecture/Conference
23:59
Data storage deviceCodeMenu (computing)Line (geometry)Disk read-and-write headWebsiteUniform resource locatorLine (geometry)System callProcess (computing)Game controllerLecture/ConferenceComputer animation
24:55
Process (computing)Read-only memoryUniform resource locatorData storage deviceLibrary (computing)Type theoryCodeStandard deviationLinear mapImplementationError messageString (computer science)Interior (topology)Function (mathematics)Parameter (computer programming)VolumenvisualisierungEmailData structureDean numberLine (geometry)Tablet computerConstructor (object-oriented programming)CountingSlide ruleUniform resource locatorCASE <Informatik>Reading (process)Interface (computing)Codierung <Programmierung>Computer programmingComputer fileSoftwareParameter (computer programming)MereologyFunctional programmingDependent and independent variablesString (computer science)Structural loadType theoryBuffer solutionError messageSet (mathematics)Data storage deviceData structureLoop (music)Data typeRow (database)Program slicingNumberProcess (computing)Field (computer science)Pointer (computer programming)System callMultiplication signSystem programmingStreaming mediaCodeLetterpress printingMiniDiscSerial portKey (cryptography)NeuroinformatikMoment (mathematics)Cartesian coordinate systemLevel (video gaming)Object-oriented programmingFerry CorstenTouchscreenStrutInfinityConnected spaceBlogoutputDot productLecture/ConferenceComputer animation
31:21
GradientInstallable File SystemStaff (military)GoogolWebsiteGame controllerUniform resource locatorRow (database)2 (number)Computer programmingKey (cryptography)Data storage deviceComputer animationLecture/Conference
32:12
Data bufferProcess (computing)Uniform resource locatorSystem programmingThread (computing)CodeStatement (computer science)Endliche ModelltheorieRead-only memoryMedianArrow of timeBlock (periodic table)Parameter (computer programming)Execution unit10 (number)SynchronizationString (computer science)MassFunction (mathematics)ParsingWechselseitige InformationUniform resource locatorBuffer solutionMultiplication signForm (programming)CoroutineFunctional programmingRun time (program lifecycle phase)Virtual memoryRight angleLengthFundamental unitInstance (computer science)Structural loadClient (computing)Data storage deviceResultantArrow of timeRow (database)Operating systemError messageQueue (abstract data type)Loop (music)Level (video gaming)MiniDiscMathematicsService (economics)MappingStaff (military)Computer fileStatement (computer science)Network topologyCodierung <Programmierung>Endliche ModelltheorieProgrammschleifeThread (computing)Data miningAddress spaceFlow separationIntegerInterior (topology)Exception handlingTelecommunicationFlagVarianceLocal ringGodPointer (computer programming)Ocean currentSimilarity (geometry)QuicksortSystem callSet (mathematics)Key (cryptography)Resource allocationSystem programmingParameter (computer programming)Shared memoryPrimitive (album)AlgorithmCASE <Informatik>SummierbarkeitOperator (mathematics)Direction (geometry)CommutatorInformationReverse engineeringVariable (mathematics)Message passingConcurrency (computer science)Computer programmingNumberLogical constantField (computer science)CollisionLetterpress printingComputer animationLecture/Conference
40:06
Process (computing)Single-precision floating-point formatVirtual machineUniform resource locatorCache (computing)Maxima and minimaElectronic signatureComputer networkError messageExecution unitServer (computing)Remote procedure callHost Identity ProtocolClient (computing)Proxy serverProgrammable read-only memoryString (computer science)Instance (computer science)Set (mathematics)Local ringKey (cryptography)CodeFlagKey (cryptography)Parameter (computer programming)Uniform resource locatorData storage deviceError messageService (economics)Electronic signatureMechanism designMereologyPointer (computer programming)Polar coordinate systemQuicksortImplementationWeb 2.0Front and back endsScaling (geometry)Computer programmingInterface (computing)Operator (mathematics)MathematicsProduct (business)Functional programmingStandard deviationProxy serverThread (computing)Process (computing)Virtual machineClient (computing)System callHookingServer (computing)Right angleoutputWebsiteType theoryString (computer science)LastteilungData structureUser interfaceFlagSoftwareCache (computing)Field (computer science)Perfect groupConnected spaceINTEGRALLocal ringAddress spaceForm (programming)DialectLibrary (computing)Instance (computer science)Computer fileArithmetic meanAreaMiniDiscCodeLevel (video gaming)DebuggerSemiconductor memory2 (number)Message passingSet (mathematics)Entire functionMultiplication signComputer animation
47:52
Inclusion mapShared memoryComputer programmingUniform resource locatorGraph (mathematics)Data storage deviceSeries (mathematics)2 (number)Cache (computing)BitStructural loadMereologySet (mathematics)Server (computing)NumberVolume (thermodynamics)Instance (computer science)BenchmarkLogic gateComputer animationLecture/Conference
50:54
Session Initiation ProtocolProcess (computing)Thread (computing)Computer programmingInstance (computer science)BefehlsprozessorElectronic mailing listComputer animation
52:01
WikiServer (computing)Complex (psychology)User interfaceDatabaseRead-only memoryUniform resource locatorCodeFormal languageWeb browserHome pageServer (computing)CodeGreen computingFormal languageComputer programmingGroup actionInteractive televisionDatabaseWebsiteBlogUniform resource locatorInterface (computing)Repository (publishing)System programmingWeb pageType theoryConnected spaceBitWeb applicationCuboidMultiplication signRemote procedure callCoroutineComputer animationLecture/Conference
54:13
Primitive (album)Semiconductor memoryVirtual machineWeb pageConcurrency (computer science)CodeQuicksortPointer (computer programming)WritingCoroutineMessage passingString (computer science)IntegerDampingInstance (computer science)Type theorySoftwareFunctional programmingData conversionEndliche ModelltheorieFormal languageArithmetic meanHierarchyBitParallel computingProgram slicingPoint (geometry)Different (Kate Ryan album)Shared memoryJava appletLevel (video gaming)System programmingCore dumpComputer programmingSpeech synthesisData typeControl flowWechselseitiger AusschlussState of matterLecture/Conference
59:15
XMLComputer animation
Transcript: English(auto-generated)
00:07
Hello. Is my microphone on? Yes. Thank you everybody for coming. My name is Andrew Durand and I work at Google in Sydney, Australia and I work on the Go project.
00:23
So what is Go? Who here, first before I start, who here has heard of Go? Okay, who here has read the documentation or looked at the website? What about actually written some Go code yourself? Alright, cool.
00:43
So for those of you who don't know, Go is a general purpose programming language. We built it as a system language when we first announced it, but it's turned out to be useful for a wide variety of things, not just systems programming. I've got a list of its killer features here, the things that sort of make it special
01:02
in combination. So the first of all, it has a strong emphasis on simplicity in its design. It's very easy to learn all of the language features are very simple in and of themselves and they're also orthogonal. The way that the language features interact is very easy to predict and very simple to understand.
01:22
It's a memory managed and syntactically lightweight language like a dynamic language like Python or JavaScript, it has that kind of easy loose feeling where you don't have to worry about memory deallocation and you don't have to do so much worrying about typing things.
01:43
It compiles to native machine code in Intel x86 32-bit and 64-bit and ARM code generation and so in practice its speed is comparable to C. We have two compilers,
02:02
we have GC which is our custom Go compiler and we have GCC Go which is a Go front end for GCC. So the code generation on GCC Go is slightly faster than GC, but GC is the more mature compiler that we devote more attention to. But they're both feature complete.
02:24
One of Go's big feature points is that it has strong concurrency primitives built into the language and it's a CSP style concurrency model like other languages like Limbo if you've heard of them. It is a statically typed language
02:42
so I mentioned that it's kind of like a dynamic language but it's not a dynamic language it's actually very strongly typed but it has a nice system of interfaces that make a kind of compile time duck typing possible. So it's a lot more malleable than your traditional statically typed languages.
03:02
We have a very consistent and small standard library that is really nice to use and all Go code is self-documenting and we have a documentation extraction program called Godock and therefore the project is very well documented and we place a huge emphasis on complete
03:21
and correct documentation in all of our project's code. And finally it's a free and open source project even though the core development of the project is sponsored by Google the core team are all employed by Google but we release everything in the open, it's under a BSD license
03:41
all of our development happens in the public on public mailing lists and we've had over 150 different contributors contribute code to the project directly who aren't from Google and of those people there's probably about 10 to 20 who contribute on a regular basis
04:02
we actually have a Windows port that is almost it's like 95% of the way there and that's being contributed wholly by non-core contributors so we've got a good community growing. Oh and we're one year old as of November so it's pretty exciting
04:21
but what am I going to talk about in this talk? I'm going to go through the complete development of a web application a simple one. There's a lot to cover in this talk and so it's going to move quite quickly. It's going to contain a lot of code examples so if you're not familiar with
04:41
Go syntax there will be things in there that aren't immediately clear if you're the type of person who sort of obsesses over small details I encourage you to kind of force yourself to look past it and focus on the actual sort of semantic meaning of what I'm explaining rather than the precise syntax
05:02
so yeah we'll just get started. If you're going to tweet about Go then use the Go lang hashtag so that I can find it and see what you've got to say. So the program that we're going to write is pretty much the simplest of all possible web applications which is a URL shortener it does two things as a web service
05:21
it takes long URLs and then returns shortened versions of those URLs and then when it receives a request to the shortened version of the URL it will return an HTTP redirect to the long URL. So the best place to start when writing any program is with the data structures
05:42
and our program GoTo will store maps, short URLs to long URLs so the natural data structure that we would reach for is a hash map Go provides a built-in map type that allows you to map values of any type to any other type and to initialise a map we use the built-in make function
06:04
so this is a line of valid Go code, this is the actual type, a map of integers to strings and our variable m is now a map of that type so to put the string 1 in the box identified by int 1, it's just the familiar kind of syntax
06:24
that you would see in many different languages and to pull it out is the exact opposite, so u would now equal the string 1. The first bit of unusual Go syntax we're encountering here is this multiple value return so this expression m2, we can
06:44
actually extract two values from it and the first value would be the value stored at 2 but the second value is a boolean value that indicates whether that key is actually present in the map or not and in this instance there isn't a 2 in that map and so v would be an empty string and present would be false
07:04
but if I put a comma present after the u then present would have been equal to true. So now that we've sort of decided that we want to use a map, let's specify the actual type that we're going to use for our program and we'll call it
07:21
URLStore. So this is a Go type definition I'm basically just giving a name, URLStore, to this concrete type a map of strings to strings and so to instantiate this hash map I just use the make function and then m points to that region of memory at which that hash
07:41
map exists. And so to store a mapping of a to Google.com I just assign Google.com to the key a and to pull it out is the similar reverse. So this is all we need as far as data structures but it
08:01
has a shortcoming. The map type isn't thread safe so if we want to access, if we want to write to the map from multiple threads we need some way of moderating access to that type and because we're going to be serving a web application by definition it has many concurrent requests coming at it at once.
08:26
So to protect the map type from being modified while it is being read we must add a lock to the data structure. So to do this we'll change the type definition. Instead of just being a map of strings to
08:40
strings like that we're going to turn it into a struct. And a struct is just like in C, a concatenation of values of other types and any Go type can fit inside a struct. So in this case we have two fields, URLs which is our map of strings to strings and then the second field is called Mu and it's of type
09:00
sync.mutex. Sync is part of the standard library and rwmutex is a reader-writer mutex. So multiple clients can take the read lock simultaneously but then when somebody tries to take the write lock only one person is permitted to take it at once, or one
09:20
function, and that's at the exclusion of all readers. So in order to access the map and be mediated by the lock we need to add setter and getter methods. So the get method will take the read lock by calling Mu rlock and then returns the URL of the string after
09:44
unlocking the read lock. So this is our function definition. The function name is get. This is our function receiver which is on a pointer, it's a URL store. And then its first argument is a key which is a string and it returns a string. So
10:03
first we call smu rlock because s is our URL store and then we extract the URL from the map and then we unlock the read lock and then we return that URL. And our set method does the reverse but with a slight added complication. The set method will take the write lock and update the map
10:23
but if the key is already present in the map we want to return false and not do anything to the map because we don't want multiple clients to be able to clobber each other in writing the same key to the same spot, the same URL to the same key in the map. And this will come up later and make our
10:42
algorithm simpler. But in this case, if to walk through this particular piece of code, set takes a key and a URL both strings, and then returns a boolean value. So first we take the write lock and then we extract and then we find out whether key is present in the URL map
11:02
and if it is present, we release the lock and we return false and bail out. But if it isn't in the map already we can store the URL under key and then we can unlock the write lock and return true. And this is a little verbose and it could be cleaner. And to demonstrate how it can be cleaner
11:21
I'm going to introduce this concept called defer. In Go you can use a defer statement to push a function call onto a stack. And the stack is local to the enclosing function in which you make the defer call. And you can defer many, many function calls. But then when the surrounding
11:42
function exits, all of those function calls are executed in last in first out order. So for this particular function foo, I'm deferring a call to print line from the string formatting package and I'm deferring a print of world but then I actually just make a normal function call to print hello
12:01
and so foo will, when it executes, will defer that function call, print hello, return, and then this function call to print world will occur, printing hello world. This is pretty much the simplest way to introduce defer. I don't have much time to go into it in detail. But the Go blog, if you search for Golang blog
12:22
there's a really thorough article written about this. But we can use defer now to simplify our setter and getter methods. So our get method from before if we look back a couple of slides, it used a temporary variable URL instead now with defer we can just take the read lock
12:41
defer the release of the lock, and then return the value from the map directly without having to use an intermediary variable. And similarly in set we can take the write lock and then immediately defer the release of the lock so that whatever happens when this function returns, that
13:01
lock will be unlocked. And this means that we don't have to include unlock twice in this function before each of the return statements. And so this is the sort of most common use of defer. But that's our set and get functions. Now to complete the design of our
13:21
URL store type, we need to initialize it whenever we want to use it. Now I mentioned earlier that we need to make map types before we can use them. And so because URL store contains a map type, I need an initializer function that will initialize that map
13:41
and return a new URL store. Now this would be a place where in typical languages you might expect to use a constructor to construct it but Go doesn't have this concept of constructors. Instead we just use functions of the form new foo where foo is the name of the type that you want to construct. And this
14:01
seems like an omission but it actually ends up being quite useful when you're writing more composable code but I'm not going to go into that today. So our function newURLStore returns a pointer to a URL store value. And in the body of this function all we do is we have a URL store literal
14:21
and then inside I specify that URLs is the result of making a map of strings to strings. So this function will return a URL store that is ready to use. The read write mutex doesn't require any initialization. Its zero value is a mutex that is ready to use and that is unlocked. So
14:45
when we're using the URL store, if I want to initialize S I say S colon equals newURLStore and now S is of type pointer to URLStore and then I can call S.set to set a key and if that returns true then we've
15:01
successfully added it to the map. And then to get a URL by key I call S.get. And you notice that this if statement looks a bit strange. In Go if statements can have an initializer in them in the same way that a for loop can in many languages. So in this case the variable URL
15:20
is the result of our get call and then in this if clause and any subsequent else clauses that variable URL will be scoped to that particular set of blocks which makes it very easy to keep things nice and clean and compact. So that's a small bit of Go idiom. So now that we have our
15:40
data type which is thread safe and ideal for storing the data associated with this program we now need, and we already have a get function which will serve us fine we need some way of putting a URL into the map and getting a key back in Exchange. So this method put
16:01
takes a URL, generates a corresponding key and then stores the URL in the map under that key and returns the key. So in our, the way we'll generate the key is by, the way most
16:20
URL shorteners work is that you take some base like base 63 is a common one where you have upper and lower case alpha numeric characters and zero to nine and you just take an integer and then convert it to that base. So I have this function gen key which does exactly that but I'm not going to show the implementation of that here. And then I have added a new method to URL
16:44
store called count which simply returns the number of elements in that map. So len is another built-in function so we're taking the length of the map returning the number of items inside the map in a thread-safe way using the read lock. So I generate the key by generating the corresponding alpha numeric key
17:05
associated with, that corresponds to the count of what's in the map. And I do this inside a for loop, an infinite loop and so what I'll do is generate a key attempt to set it, if the set succeeds then I return
17:20
that key, that's a success, a successful put. If it hasn't set it, it means that somebody in the meantime has already generated that key and set before I have and so I loop back around and generate the next key and by this time count will have incremented because somebody's already inserted something under my feet.
17:44
So then we've talked about how we have get and put which service the backend part of our URL shortener but now we need to look at the user interface. And the user interface is going to be served over HTTP so we'll
18:04
use Go's HTTP package to provide the infrastructure for serving HTTP requests. Now this program here is a complete Go program that serves Hello World over HTTP and just to walk you through it briefly, we have package main which is a package statement. Every Go program begins with a package statement
18:24
and every Go program begins in package main with function main. So I'm in package main, I import the string formatting package and the HTTP package and then in my main function I call HTTP handle func which registers this function hello to handle all
18:44
requests to slash which is our web root. And so when somebody accesses the web root on this server this function hello will be invoked and it has this function signature that takes a response writer and a request.
19:00
The request is the request data of the HTTP request, the incoming request with information such as any query parameters, the path itself, the address of the person making the request, any headers, etc. And the response writer is a type that you can write to which is what I'm doing with fprintf and then it also
19:23
has a set of utility functions to modify headers and other things. But in this case all I want to do is write the string hello world using fprint to w. And then finally I call listen and serve to set up my web server listening on port 8080 and then this function will block forever serving HTTP
19:43
requests. So how can we leverage this to actually implement GoTo's HTTP interface? Well our program needs two HTTP handlers. It needs a redirect handler which will take incoming shortened URLs and provide an HTTP redirect to the longer version of that URL and I have the
20:02
add handler which handles the submission of new URLs and returns the shortened versions. So in our program's main function we'll use handle func to register these two new functions, redirect and add, under the web root and slash add. And
20:22
the semantics of this is that a request to slash add will go to the add handler and then requests to anything else will get sent to redirect. And this is the behavior that we want for this program. So here's the implementation of add. The add function, it reads the URL
20:43
parameter out of the HTTP request, so a form input box named URL for example, and then it puts it into the store and then sends the corresponding URL back to the user. So inside our function we get the request's form value that's called URL and store it in the variable URL
21:03
and we call store.put putting the URL into the store and getting a key in exchange and then we print the new shortened URL which, because I'm running on my machine on port 8080, you'll just get I'm just printing that out with the key appended to the end
21:22
and we're writing that to the response writer, so that's what the user will see in exchange. But what is this store? Well store will be a global variable that I've created and somewhere in my source file I have this at the top level, var store equals new URL store. And so this will be the global instance of store that will be accessed by
21:43
the add and redirect HTTP handlers. But I only showed you one part of the add, the post handler, we have to have the part of the handler that displays the form in the first place and so if URL is empty it means that there's been no submission
22:04
or there's been an empty submission. And so in that case I'll just fprint this constant add form which is a string constant of an HTML form and print that and then bail out. But if URL is not empty it means
22:20
that somebody's actually submitted a URL and I can throw it into the database. I mean this is an overly simplistic UI, obviously if you were going to build a proper web app you would put a bit more effort into it. But at this stage this actually works quite well. And so the redirect
22:41
handler is a function that takes a key out of the HTTP path and then retrieves the corresponding URL, the longer URL from the store and then performs an HTTP redirect to take the user to the site.
23:01
If the URL isn't found we want to show a 404 error. So in our redirect handler we access the request URL's path and we slice off the first character of the path because it's always going to be a forward slash and then we look up the key in the store with get
23:21
and take the corresponding URL and if the URL is empty we know that it's not in the map. So we return a not found error using this not found helper. Otherwise we can redirect the user using the redirect helper and we redirect them to the URL that we retrieved from the map
23:41
and we return a status of found 302 found which is the appropriate HTTP code. So we've written I think it's about 90 something odd lines of code and it's a complete functional working web app and I can demonstrate what we've got working so far. So I run that
24:05
head to localhosts 8080 the conference Wi-Fi okay and then if I want to add
24:23
is it cold in here? So I add Fostem and then I get the resulting URL. When I visit that URL it should take me to the Fostem site. And then I can add another URL girlang.org
24:43
and we'll see that this works as well. So you know that's pretty nice. But when this process ends, when I hit control C those URLs that I've entered have just disappeared into the ether so it's not a particularly
25:01
useful application at the moment unless you have some magical computer that will never switch off in a process that will never end. So we need to modify the URL store so that it actually logs URLs to disk and then restores that data when it starts back up. So before I get into that I'll just introduce another aspect of Go's
25:23
type system which are its interface types. And interface types define a set of methods and then any type that implements those methods can be used where a value of that interface type is expected. So in a way you can think of interface types as defining behaviors and then any object that behaves in that way
25:43
can be used where that interface is expected. So one of the most frequently used interfaces in Go Land is the writer package, the writer type which is specified in the IO package. And so the type writer is an interface which contains the method
26:03
write which takes a slice of bytes or a buffer and then returns the number of bytes written and an error value. There are many, many different types that implement writers. There are compressors, network connections, HTTP connections, files,
26:21
tar archives, you name it, it's in there. And in fact we've already used an IO writer in our HTTP handlers. The w response writer argument to our HTTP handlers it implements the write method. And the fprintf function in the string formatting package expects
26:44
an IO writer. So we can use the response writer as a writer because of the shared interface. So that's interfaces. But we were talking about how to represent data, this URL store data
27:02
on disk. Well the way we'll do that is we'll use Go's gob package and gob is a serialization package which turns Go data structures into streams of bytes and vice versa. The gob package has the functions new encoder and new decoder which wrap IO writer and IO reader
27:21
values respectively and then they return encoders and decoder objects which allow us to call their encode and decode methods to write and read Go data structures. And so when we're writing to disk and we want to store this data on disk we need a data type to describe it.
27:44
And so we'll create this new type record which is a struct that has two string values, the key and the URL. And then our new save method which takes the key and URL as a string will create a new gob encoder which wraps some file and then calls encode on that encoder
28:03
to write our new record. This is a record literal which contains our key and our URL and then that encode will write that record to the underlying file. But what is s.file? s.file is our new field file which is a file handle that I'm putting on the URL store
28:24
and then in our new URL store initializer function we'll add this code which has gone off the screen to open a file with os.open. The file name is a new argument to URL store and then if the file doesn't open correctly then we'll bail
28:43
because if we can't open the data store we probably don't want to continue running the program so log.fader will actually exit the program. But if we do succeed we'll set s.file the struct the structs file field to our file handle f.
29:01
And so then in save when we refer to s.file we're referring to this open file. But now the last part is to load the data back into the program when the program starts and so we define a load method. And this load method returns an error value. It takes no arguments. And first what it does is it seeks to the beginning of the file. And if the seek fails
29:24
it will return an error. And then it creates a gob decoder. And what we do is we set up an error value and then we go into an infinite loop. So for error equals nil it means just continue looping while there isn't an error. And then I set up a record value r and then
29:44
I decode into r by giving the decode function a pointer to r. And so decode will read bytes from the underlying file and try to marshal them into a record struct, into this record struct. And if that succeeds, if error is not nil,
30:03
we can call the set method to set it into our map. And in this way this will loop forever until it reaches the end of the file. And when it reaches the end of the file error will equal os.eof. And if that's the case we want to return nil, we want to return a no error because that's the
30:23
expected failure case. But if there's any other failure case we'll return that error and pass it up the stack. And so the last thing we need to do is stick a call to s.load into our initializer function and if that fails we'll print a benign error because the first time we run this program our store won't
30:42
exist and so it's okay for this to fail on the first run. Oh and finally we need to add a call to save each time somebody successfully makes a put. So in our put function before returning the key we'll call s.save, passing in the key
31:02
and URL. And then if there's an error with that we'll log it. Oh and the last thing is we need to, when we instantiate our store in the top level of our program, we need to provide some file name where the data will be written and read from. So I can show you what we've written so
31:23
far. So if I add google.com and then visit slash one and you notice that I actually already
31:43
had one stored in there. Slash zero is the FOSTM side and so if I control C out of that, well if we look there's this store.gov and if I hex dump it you'll see that there's two records in there. The first one is a key of zero
32:03
pointing to FOSTM.org, the second is a key of one pointing to google.com and it will continue on in this vein. And so if I restart the program and reload this URL it should work, but it's not. Don't worry. There's a lot of
32:24
intermediary programs and they're not there. I've made changes in the last couple of days so it might not be synced up totally. Anyway assuming that that all works and it is all correct, which it is, it is, consider this pathological situation. When we have many
32:44
clients that are adding URLs simultaneously, say I've just launched this service as Google's fancy URL shortener and suddenly a million people try to add URLs at the same time. Even though we can safely update the map concurrently the disk writes might happen simultaneously, like the underlying
33:04
save calls might happen at the same time. And now some operating systems do writes atomically and so this might be okay, but some of them don't and we can't rely on this kind of behavior. And secondly even if the writes don't collide it means that every client has
33:20
to wait for its put to go through to disk before it will return. And we don't mind making clients wait for them in memory map to be updated because that's a very fast thing to do. But if you're talking about disk writes, in a heavily IO loaded system clients will wait way longer than they need to for their requests to go through. So what we want to do ideally is to decouple
33:45
the put function from the save function and make them happen in separate, not at the same time. So this introduces concurrency and the fundamental unit of concurrency in Go is the Go routine.
34:03
And there's a reason why they sound like co-routines because that's essentially what they are. But a Go routine is a lightweight thread that is managed by the Go runtime. They exist in shared memory, like many familiar sharing models, but they're much cheaper to create than an operating system thread. They have a very small stack. Creating
34:21
a new Go routine is under about 4k of allocation so you can create many thousands of them or even hundreds of thousands of them if you'd like without running into too much trouble. So to launch a Go routine we use a Go statement which is a similar sort of syntax to the defer statement.
34:41
It's Go followed by some function call. And when you call Go and some function, the function itself like the function pointer and the function's arguments are evaluated in the current Go routine. A new Go routine is started and then that function executes in the new Go routine. Really? 15 minutes left?
35:02
Oh my god. So we have foo executing in one Go routine and bar executing in the current Go routine. And that's it. That's as simple as Go routines are. But this isn't very helpful. What if you have, you know, what you need to talk between these Go routines? And so this is where channels come in.
35:23
A channel is a conduit like a Unix pipe except it's typed. You create a channel and then give it to two ends of some communication in separate Go routines and then they can send things down the pipe, down the channel to each other. Like maps, channels are initialized with the make function. So to make a channel of integers
35:42
I call make chan int. And so ch in this case is a new channel of integers. And then communication is expressed using the channel operator which is a less than dash like a little arrow. And so to send the integer seven down this channel we do channel
36:00
less than dash seven. And then to receive an int from a channel it's kind of the reverse. And we receive a value from ch and then store it in i. The data always travels in the direction of the arrow. So that makes sense. So the example of communication between Go routines. Here we have
36:23
some function main. We create a channel of integers and then we launch a Go routine calling sum and we pass it some arguments followed by the channel c. And then in the body of sum we add x and y and send that value down c. And then back in our main function we receive the value from c and print it.
36:42
So channel operations, and like they would in this case, they typically block until the other side is ready. So in this case this send to c will wait until this receive from c is ready to happen. Now obviously they're going to be ready immediately in this instance but in other instances this can be a very powerful synchronization primitive and gives way to some very interesting
37:02
algorithmic possibilities. But channels can be buffered or unbuffered and if you make a channel buffered the ascend to that channel will not block until the buffer is full. So to make a channel buffered you simply add a second parameter to make which is the number of, which is the size of the buffer. So this
37:24
channel of integers will store 10 integers before it gets full. So instead of making a function call to save each time we do a put, put can just create a record value and send it down some channel. So I'll add this field to my URL store,
37:43
save, which is a channel of records, and then when I successfully perform a set I will send this record containing our key and URL down the channel save. And then at the other end of that channel we need to have something that's actually receiving those values and writing them to disk. Now we'll
38:01
create this new method called save loop and save loop will run in a separate go routine and what it does is it opens the file identified by file name and bails if it can't open it and then creates a new gob encoder that wraps that file and then loops forever receiving from s.save storing the record in r
38:21
and then encoding r writing it to the file to the underlying file and logging an error if an error occurs. And so the final thing is we need to modify a new URL store so that it kicks off the save loop in another go routine when the URL store is initialized. And so we create some
38:44
constant save queue length which is the the internal buffer length of our save channel and so we can have up to a thousand URLs being saved before puts will have to wait for the buffer to clear. And then when I create the URL store
39:05
value I'll also provide save with the result of making a channel of records of save queue length and then I call s.load passing in the file name parameter and log the error and then I kick off
39:22
with a go routine that's running save loop passing in the file name. So just very briefly in this program already there's a lot of magic numbers a lot of constants I want to replace them with command line flags I can import the flag package and set up a bunch of global variables blah blah blah. The one thing I have to
39:43
be sure to do is call flag.pass at the start of my main function to actually read the command line options and set these global variables to their respective values. And I need to move my initialization of new URL store into the main function so that I can refer to my now passed command line flag and similarly we
40:05
update the listener address and the host name in the ad handler but I'll quickly demonstrate... or will I? No, I won't. One more thing, the last thing that I'm going to do to this program is
40:21
so far we have a program that runs in a single process and while it runs concurrently in that single process and it will spawn multiple threads to handle blocking operations so it will always have something to do. But a single process running on a single machine can only go so far. What I'd like to
40:40
do is turn this into a production ready web scale URL shortener and to do this I will shard this program into master and slaves and the way this will work is users will make requests through some sort of load balancer which I'm not going to worry about now but they'll hit slave frontends via
41:01
HTTP and these slaves will make RPC calls to the master and what I'm going to demonstrate now is the implementation of those RPCs and I'm turning this into a slave so I... just a second... so to introduce
41:23
Go's RPC mechanism it's part of the standard library and it's very simple. The way it works is given a value of some type RPC will expose that value the value the methods of that value that have this form
41:41
and the form is for some type t they need to have two pointers as their arguments and only two and return an error value and so if you have a type which implements methods of that form then RPC can serve them over a network so to make our URL store a RPC service we need to change the
42:04
function signatures of our get and put methods so instead of take instead of get taking a key and returning a URL instead it's going to take pointers to two strings key and URL and then return an error and similarly put is going to is going to take pointers to URLs and keys and
42:22
return an error now and and once you've done that you also need to obviously modify the call sites of those methods in our add and redirect handlers so that they do the right things but i'm not going to show that now because i'm running out of time but um to actually serve the our URL store over the network
42:45
i'll add this uh command line flag RPC enabled and um it's called RPC and it's a boolean value so if i call my go to with dash RPC it will it will serve its get and put methods over RPC over HTTP and the way we do that is
43:04
if the RPC enabled flag is set um i call RPC dot register name to register store under the name store and then i call handle HTTP and that will hook the RPC engine into the HTTP package and later on when i set up my HTTP
43:22
server it will also transparently serve RPC through it using a little bit of magic but not much and so now that we have the URL store available as an RPC service we can build another type which looks like a URL store except all it does is pass its requests through to the master server
43:42
and so we'll call this the proxy store and the proxy store is a struct that simply contains an RPC client handle and so our new proxy store initializer function will take an address and this is the address of the RPC server and it will call dial HTTP from the RPC package to connect to that address over TCP
44:04
and then uh if an error occurs it'll log it but otherwise the client will be put in the proxy store field and that proxy store will be returned to the user and then we define get and put methods on that proxy store
44:20
and what they do is they take keys and URLs and then they pass through those requests by invoking the call method on the RPC client and this will call the store.get method at the other end of the RPC connection which will be another instance of go to passing in the key and URL and so when this returns it returns the error value from that call
44:43
and URL will be updated with the resultant value and similarly the same thing will happen with put and key so this is enough to implement like a pass through but um there's something missing the slave has to cache some of this
45:02
data otherwise it's all it's doing is adding work so we already have the perfect data structure in which to cache this data and that is the URL store so I'll add a URLs field to my proxy store struct call it URLs and then initialize it
45:21
in my uh initializer function by invoking new URL store and the one thing that I'm not showing here is I've just modified your URL store so that if you give it an empty file name it just won't try to read or write it to disk it still does all the in memory stuff it still keeps a map but it won't write anything to disk and
45:40
then I modify my proxy stores get and set methods or get and put methods so that first it tries to get from the local URL store cache and if it finds something in the cache if error is nil I can return and then that value this URL value will be updated with the cached value otherwise I try the RPC
46:03
call and if the RPC call has a problem this will return the error if it doesn't have an error meaning it's successfully found um something it's successfully gotten the URL from the RPC server I'll set that in my URL store cache in the local cache and then return no error
46:23
and then the put method is simpler it attempts to do a put to the master RPC server and if that's successful it updates the cache so now we want to integrate um proxy store with the web front end we want our slaves to serve
46:41
our web interface and for it to pass all its requests through the proxy store instead of the URL store um because they both share the same methods get and put we can define an interface type called store which includes the put and get methods and our global variable store can be of type store
47:01
and then in our main in our main area we can create a master address flag dash master which is the address of the master RPC server and if a master address is provided it means that we must be a slave if we're being told to connect to a master this go-to instance is a slave so we instantiate a proxy store pointing to
47:26
that master address but if master address isn't supplied then we must be the master so we create a URL store which reads and writes from this data file but then the rest of our interface code behaves exactly as before I've just swapped in an entire new set of functionality
47:41
and the interface has been implicitly taken advantage of and I've not had to do any further changes to the front end for this to work so I'll just demonstrate um this ball in flight um
48:04
so I have a a if I go to slash ad on localhost um we'll see we have a URL store running that's already got a few URLs in there but then what I've actually written is
48:22
this little uh little graph graphing program and this this has got its hooks into my go-to instances and we have a little benchmarking program and what the benchmarking program is doing is you can see identified by this blue series at the top the blue series is the benchmarking program
48:43
making gets against my go-to program and the yellow series down here is the benchmarking program making puts and the red series is the the go-to store reporting the number of gets that are being made every second and the green series is reporting the
49:04
number of puts that are being made every second so we can see from this graph that approximately eight to ten um puts are being made per second by the benchmarking program and about 100 um gets are being made and they're being serviced one to one because there's only one master and that's all that's
49:22
happening what I'll do is just increase the volume a little bit um so we're now serving between five and six hundred um get requests a second and about 50 put requests a second so we're increasing the size of our data set substantially with every second that goes past so if I want to introduce a slave we can see what happens
49:42
introducing a slave we notice that immediately the red series the master's get requests have gone to zero because now our slave which is this new purple series is serving all of the get requests it's sitting between the benchmarking program and the master so the benchmark is hitting it hard
50:01
but the number of put requests are still remain constant and they're still being served by the master server because they're being passed straight through if I add a second slave predictably we can see that as the as the number of aggregate gets
50:20
the blue series that are being made remains the same roughly um we can see that each slave the darker blue and the purple series in the middle are sharing the load about 50 50 and now the master's get requests have gone up a little bit because it has to prime the caches in each of the slaves now that each slave is sharing a different part of the load
50:44
adding a third slave um we can see that again this uh it's it's now gone down to about a third and the master is still sitting pretty low and the one thing I should have showed you from the beginning of this demonstration is um if we look at the process
51:12
if we look at the uh that my process list these three instances of go to here are running at just between 15 and 20
51:22
CPU while this well this instance here is running at about five percent and that's our master so the master is being shielded from this onslaught of of requests and the benchmarking program itself is kind of doing most of the work um and you notice that I haven't specifically asked
51:44
for any threads in any of these programs except my slaves are running about 10 threads a piece and my master is running about about 10 threads as well well my benchmarking program actually has 92 threads so it's hammering pretty hard um and this is this is now serving
52:03
like six seven hundred requests a second um on this puny macbook air which is totally underpowered and and pretty much a piece of junk so um you know imagine what it could do on a on a real uh server and this is without having to this is like genuine high performance
52:22
um code here without having to even really think about efficiency it's all just kind of given to us for free by the fact that go is such a lightweight language now so um this while this program works and you can see it's fairly resilient and it's fairly performant
52:40
um you'll notice that there's a bunch of stuff that could be a lot better obviously the aesthetics could be greatly improved it could be a lot prettier um go has a template package that would make this very easy to accomplish and if you visit the go website there's a code lab which goes through creating a web application with the proper interface
53:01
the reliability of this system could be a bit better currently if the master slave rpc connection goes down um it will just die and never be saved and um and you could fix this by having some kind of dialog routine to reconnect every time it goes down and manage the connections
53:21
um as the size of the URL database grows it might not fit in ram anymore so we might have to shard it out and do something a bit more complicated and if you wanted to support deletion of URLs that would complicate the interactions between the masters and the slaves because the master would have to report that URLs have been deleted but these are all exercises for the reader and this code is all available um in my github repository so you can
53:43
check it out um we've got a lot of there's a lot of go resources if you're interested in learning go the first place to go is golang.org um where there's lots of documentation tutorials and there's even if i can demonstrate very briefly there is even a box on the front page
54:02
of the website where you can type go code and have it compiled and run on one of our servers um there's our go blog and there's a bunch of other stuff so does anybody have any questions yeah hi um what i haven't heard in your uh
54:29
speech was what why would i have to use go and not for instance java or c++ for this well if you like the syntax of java and c++
54:43
then have fun but um honestly um once you get to a certain size in substantial like the types of programming approaches that java and c++ encourage is a very uh type laden um hierarchical approach to writing
55:01
software which uh once you get to a certain size it actually starts to really weigh down on you and go is has a much more compositional lightweight approach to writing software so i mean it's a long conversation if you want to get me afterwards any other questions
55:21
uh so the two things that turned me off of go when i first saw it was first that it has null uh implicitly in the type system i.e if you specify a function that takes a string it either takes a string or it takes null and the same thing if you return it that's not actually true um there are certain types in go that
55:43
that can be null and a string is not one of them um only a reference type can be null in go so if you have a string or an integer or a or a float or any sort of concrete data type it can't be null it can only be at zero value which is usually numeric zero or an empty string
56:03
um but if you have a pointer type a pointer can be null or if you have a map type a map can be null if it's not initialized or if you have a an array or a slice type rather a slice can be null so does that mean that i can write uh that i can implement a type that cannot be null
56:20
and or yes and if you have like a struct type you can use a struct value and a struct value cannot be null all right i mean if you need further clarification i can you talk about it after um yeah one of the
56:41
one of the things that that strikes me is that go is apparently a language which is designed uh for concurrency yet when you can compare it with well-designed concurrent languages it becomes clear that go's concurrency primitives are at best half-assed and at worst actively dangerous um you have designed a concurrent language which is based around
57:01
shared mutable state and that's pretty dim-witted if i may say so to begin with um and then you also have channels which block callers you have core primitives which are not type safe i guess my question is how you were able to design a concurrent language which is so terrible at being a concurrent language it would be nice it would be nicer if you could
57:21
have phrased your question in a way that wasn't incredibly insulting because i do think because because i do think that you actually have a point but what you're actually pointing to is a philosophical difference um go is a language that is very close to the machine it has its heritage in c but we don't we allow you to shoot
57:40
yourself in the foot if that's what you want to do um it doesn't it's not a immutable sort of uh fenced uh non-shared memory model for concurrency because that's not the way processes work on um on the machines that we write code for so go is more of a pragmatic language in that sense but go's primitives like go routines
58:02
and channels um they actually actually allow you to do what we say is that you share memory by communicating about memory instead of uh communicating by sharing memory so while we allow you to use things like mutexes to implement like traditional um evil sort of shared memory concurrency channels actually allow you to do
58:24
in a way that is very efficient and easy to understand and predictable like pass around a pointer to a region of memory and when you when you write it's very easy to write code with channels that pass as pointers and pass as references around and that that is safe because you can see there on the page that it's safe when you send that
58:43
pointer down a channel don't use it anymore and then it's safe so yes it is possible to totally uh screw it up but the tools that we give you actually make it very very easy to write safe reliable concurrent code okay uh let's thank the speaker and uh have a lunch
59:05
break