Our experience from building and running a SaaS in the cloud
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 170 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/50541 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
NDC Oslo 201414 / 170
2
3
5
6
8
11
12
16
21
22
23
27
31
35
37
42
43
45
47
48
49
50
52
54
56
57
58
61
65
66
67
71
74
77
80
81
83
84
85
87
88
89
90
91
94
95
96
97
98
100
102
107
108
112
114
115
116
118
120
121
122
123
126
127
128
130
133
135
137
138
139
140
141
142
143
144
145
147
148
149
150
153
155
156
157
158
159
160
161
162
163
166
169
170
00:00
Computer-generated imageryBlock (periodic table)Scaling (geometry)Elasticity (physics)Structural loadVolumeGroup actionTask (computing)Convex hullData recoveryOpen setTime domainMusical ensembleBackupInternetworkingDrill commandsReal numberView (database)Process (computing)Multiplication sign2 (number)MathematicsPressureKey (cryptography)Endliche ModelltheorieThomas BayesCasting (performing arts)Compass (drafting)Server (computing)BitCartesian coordinate systemVideo gameMedical imagingInheritance (object-oriented programming)Software developerPoint cloudTemplate (C++)Search engine (computing)Different (Kate Ryan album)CASE <Informatik>Regulärer Ausdruck <Textverarbeitung>Sign (mathematics)DampingPrototypeField (computer science)BackupOffice suiteBasis <Mathematik>Information technology consultingService (economics)Software frameworkElasticity (physics)Graph coloringDistortion (mathematics)MiniDiscInformation privacyWordOpticsQuicksortMereologySpacetimeEmailOnline helpRight angleExecution unitHuman migrationPattern languageLevel (video gaming)WebsiteOptical disc driveFrequencyDatabaseDemonInternetworkingAxiom of choiceComplete metric spaceGoodness of fitProjective planeCurveProcess (computing)Data typeNumbering schemeUltraviolet photoelectron spectroscopyLibrary (computing)SequelFile formatIterationSource codeParameter (computer programming)MappingMatching (graph theory)CuboidSet (mathematics)CodeClient (computing)Data storage deviceCategory of beingSatelliteSystem callEntire functionDenial-of-service attackSingle sign-onDescriptive statisticsDefault (computer science)Figurate numberFunctional (mathematics)Moment (mathematics)Scaling (geometry)Line (geometry)Content (media)Game controllerScripting languageDensity of statesComputer programmingBoilerplate (text)Uniformer RaumOpen sourceUser interfaceFreewareLoginWeb pageWeb serviceInteractive televisionGame theoryMobile appCycle (graph theory)Natural numberNumberParsingThermal expansionState observerPhysical lawWave packetKeyboard shortcutState of matterTouch typingGraphic designInstance (computer science)Type theoryProgramming languageWeightChemical equationFeedbackRevision controlAnnihilator (ring theory)TrailConfidence intervalDependent and independent variablesInformation securityPoint (geometry)CloningSampling (statistics)Flow separationError messageDivisorTowerCellular automatonSineLie groupReflection (mathematics)Software testingLaptopPlotter1 (number)Poisson-KlammerException handlingAdditionTerm (mathematics)Associative propertyFitness functionPlanningEquivalence relationDevice driverComputer iconPatch (Unix)VirtualizationDesign by contractComputer configurationCodeTorusAuthenticationRange (statistics)Limit (category theory)Natural languageComputer architectureMixed realityAnalytic setToken ringAreaINTEGRALComputer fileDirectory serviceLink (knot theory)Mobile WebDisk read-and-write headWeb 2.0InformationUniform resource locatorBuildingTracing (software)Plug-in (computing)MultilaterationThresholding (image processing)Heegaard splittingTime zoneElectric generator.NET FrameworkLastteilungFront and back endsPresentation of a groupValidity (statistics)Arrow of timeSemiconductor memoryRuby on RailsWeb applicationStructural loadJava appletAlgebraic closureCalculationInterface (computing)Computer animationProgram flowchart
Transcript: English(auto-generated)
00:02
Okay, I'll get started then. I'm Arling, I used to be a consultant and developer. I'm still a developer. But after I decided to do a start-up, I'm also a DevOps architect, front-end developer,
00:26
salesperson, dealing with legal stuff and accounting. So if this presentation is a bit all over the place, it's because that's my daily life. We decided to build a software-as-a-service for businesses or consultancies.
00:49
And we started out obviously very simple, with just a small application, and I want to talk about how that evolved to be kind of a quite big application.
01:08
Just to show you, give you an impression of how this evolved graphically, this is the first ugly prototype, which turned into this.
01:21
This is the first time I got the designer to kind of help me out, or I sent this to him, make it look nice. Next iteration looked a bit like this. And then I got my co-founder, which is an interaction designer, who also has to do lots of front-end and graphic design to help me out. So it started to shine a bit more.
01:45
That's just how it looks. But it kind of illustrates the development also on the technology side, although I wouldn't swear on my grandmother that the tech is as pretty, but we try.
02:03
Last place I worked before I started this was at Forward in London, and we were doing lots of Rails applications, Ruby on Rails. So that was my programming language and framework of choice, and I wanted to create
02:21
something fast, and I felt confident in that and productive. So I kind of just stuck with that and decided to go for it. So my experience from that has been, and it still keeps me quite productive, it's been a few issues.
02:40
Obviously, there's been a lot of security patches the last few years to Ruby and Rails. But they kind of got around that. As long as you keep up to date, just monitor it, it's very easy to apply them, and it usually goes fast.
03:00
They're kind of taking it to another level now, I feel, although you have to pay attention. Another thing is, and that might not just be a Rails thing, but in terms of gems or libraries, it's very easy when you have a problem, you search for a solution.
03:20
So we're using MongoDB, and I'll talk a bit more about that later. But I started using MongoDB, and then I wanted to have some kind of history, so you can track history when it changes to the models. So I just Googled that, found a plugin or library for that, and then we needed
03:42
pictures or images that we wanted to store, so there's a library for that. Then we needed OpenID so we could authenticate with Google, et cetera, and you wanted to store parts of those tokens in the database, so there was a library for that, and you
04:01
go on and on, and if you don't think about it, and just pull in a lot of libraries, you can end up with quite a lot of small libraries that someone just made once, threw out there, and when it comes around to upgrading to a new version of the main
04:20
library, the other ones might not be updated, so I ran into that quite a few times, so, and when I sort of started looking at it, it's actually just one of these libraries for, I don't know, 70 lines of code, and I spent the day to try to figure out how to upgrade it until I just looked at it, okay, I'll just
04:42
write it myself, or fix it, so that's something to pay attention to. We ran this on Heroke. I don't know if you know about Heroke, but it's very simple to get started, just push, git push, and it's running.
05:01
For some kind of technical subtleties, we decided to move it to Amazon web services to run our own server. We started out with one server, and now we have at least 12, probably more.
05:23
We also run it in a secure zone, which is called VPC, or virtual private cloud, which basically gives you the ability to find subnets, so we have two load balancers in the DMC. We have a VPN server that you have to connect to if I want to deploy to the
05:44
servers, and a net server that kind of takes care of the outgoing traffic for, or, I guess, if some of these servers need to download something from the internet, they have to go through this. They are not exposed to the net at all.
06:02
We have one zone with the app server, and then the databases, search engines, et cetera. I will get into details about this later. They run in a third zone.
06:21
Amazon web services, I was a bit familiar with that before I started out, but it's been working quite well. The virtual private cloud stuff takes a while to understand and set up, but once you have it set up, it's very, I don't know, reassuring to know that
06:40
the data is fairly safe, and it's only you that can access these things. There's a few tricks I learned. For example, the default settings for the load balancer, so the load balancer stays on traffic to servers, and you attach a server to a load balancer,
07:04
and the load balancer pings the service on some URL to make sure it's online, and if it's not online, it will take it out of the load balancer and hopefully switch to another, but when you're starting out small, you might have just one server, and if something goes wrong on that server, the load balancer will take it out, and then you fix it,
07:22
and you want it to go as quickly as possible back up again. The default settings for these health checks is something like 30 seconds times 10, so you fix it, and then you have to wait five minutes, 10 minutes, whatever, for the load balancer to actually recognise that it's working again,
07:40
so I just lower that to the lowest possible thing now, and if you ever have 10 servers, maybe I can erase it again so that we have a bit more higher threshold. Feel free to ask questions as we go along, by the way. I would love it if we got interaction here,
08:00
or there's no big conclusion at the end, it's just I'll just go through our stack, so feel free to stop me. Another very important thing is that security-wise, and security has been kind of a main driver for a lot of this,
08:22
so you start out and just prototype something, and you build something, deploy it on a Roku, and you think everything is fine, but then you start selling this stuff to big clients, and they have a lot of security requirements for you, so you have to document your security, and you have to be sure that everything is secure,
08:42
so that's one of the reasons we went for this virtual private cloud thing, which makes it really hard to access the servers. But then Amazon Web Services has a web interface, and you might have a code on GitHub that has a web interface, etc. So we have turned on two-factor authentication on everything,
09:03
so if I need to log in, I always have to use the codes for my mobile. So just think about the weakest link all the time. So security has been one driver, and then privacy has been another one, so when I first started this, I didn't think I would have to answer questions about privacy,
09:24
like where is your data stored, how do you keep it from others, etc. Fortunately, we were able to get away with this as long as we run on Amazon servers in the EU,
09:41
so basically our servers are in Ireland, and that more or less works with the Norwegian law and requirements. So it's important to understand these things and actually take it seriously. And that are kind of non-functional requirements that I can drive your architecture and design and sort of limit your options a bit.
10:06
So, but obviously having 12 servers, whatever, in the cloud, and we're basically two people in the company. We do a lot of different things every day,
10:22
and it's not every day I set up a new server. So I do it, and I leave it running. And I might set up another one, a different type later, but it's very hard to remember how you do these things. So one day I woke up and the server that we had,
10:41
it was just like unreachable. It's not really there anymore. So that can happen when you run servers in the cloud. They get stuck, something happens. It's very easy to fire up a new one, but you need to configure it and you need to deploy to it, right? So how do you do that and how do you remember how to do that and how do you make sure you can do that fast
11:05
when the roof is falling down in a sense? So we decided to use Chef. Have you heard about Chef? It's kind of like puppet and infrastructure automation framework.
11:21
Also written in Ruby, so that's probably why I chose it. But it allows you to define basically programmatically how your servers will be set up. So on the web server, I install Git Nginx, which is kind of routing traffic to the app.
11:44
Unicorn is just wrapping the app in a container. And different custom stuff that I have for running background workers, etc. So you specify this in code, and you run a command on the server that basically goes through all this code
12:02
and just makes sure everything is installed, all the folders are created, the users have the right permissions, etc. And everything is just there. So if I need another server that is identical or I need to just make another server, just run that. That has given me a lot of confidence
12:22
and helped me get some sleep. So that has been very nice, essentially. Some things take a bit longer to install, though. So for example, Ruby, they have a lot of new versions.
12:40
And the Ubuntu repositories, whatever, they don't always keep up to date with the new versions. So if you want the latest, you have to compile it from scratch yourself. And that could take, I don't know, 20 minutes, depending on the server. So I found out that it's most of the things I want to install using Chef, but the things that take a really long time,
13:03
I do it once, I create a snapshot of that image, and I use that as a base image for all the other servers. So whenever I have to change that, I have to create a new base image, but it's not that often. And the good thing is that, you know, how to install all these application servers
13:22
or services, whatever. There are what I call cookbooks. There are already recipes online. People have done this before. So if you're lucky, you can just Google how to like Chef Nginx, and you'll find the cookbook. And it just works out of the box. And you can customize some attributes, most of the times.
13:41
But like all other open source kind of ecosystems, sometimes these are just things that people made in a hurry and just put it out there. I wanted to do like set up MongoDB and a replica set with some properties.
14:01
And I found one that can do a replica set, and I found another cookbook that could set the property, but not one that could do both. I had to write my own or take parts from here and there. So that's something to look out for. And Chef is nice, but it's very far from perfect.
14:21
There's a talk about Docker later on, which I'm hoping is the new silver bullet. So we'll see about that. But it's helped us get quite far. But it's quite a bit of a learning curve, and it's a lot of effort.
14:44
So that was a bit about the infrastructure. Client side-wise, we have, well, it's almost a single-page web app. It's more like a three-page web app. Anyways, we've been using lots of JavaScript or CoffeeScript.
15:02
I really enjoy CoffeeScript. It's probably because it's quite similar to Ruby. I haven't had any issues with it at all. It's easy to debug. You see, even though it's not always the same line number because it expands. So when it takes a CoffeeScript function and turns it into a JavaScript function,
15:21
it automatically sort of gets a bit bloated because you have to have the curly brackets and some other stuff. But it's very easy to debug still, which is what I hear people are a bit afraid of. But that's never been an issue for me. What can be tricky sometimes is that,
15:41
and I can see if I can find an example of that. So the way the syntax works, I can show you.
16:02
JavaScript has this setTimeout function, where the function is the first argument and the milliseconds are the last argument. However, CoffeeScript is very nice when you can do things like this. You pass in an argument and then the second argument, or the third, whatever,
16:21
as long as it's the last one, if that's a function, you can just ident, you know, use an arrow and just ident the contents afterwards. But given the native setTimeout function on JavaScript, it's the other way around. One trick is to just wrap it in another function that just switches the argument and then you can use the CoffeeScript syntax.
16:42
But that's more or less the only challenge I'll have with it. We've also been using Backbone.js. I guess that was very much hyped at the time we started. And it's been working out well enough. If you compare it to other client-side frameworks,
17:06
it doesn't do that much. So you have to bind it to, all the bindings to the, you know, forms, et cetera. You have to do it manually. You have to do it yourself. But the good thing about it, it's not that intrusive.
17:23
And it's fairly flexible. But you do end up writing a lot of boilerplate code. So I'm not sure I want to replace it with something like Angular yet. But I might get there one day.
17:40
I had the same issue here as with just Ruby or Rails, is that at least when you start out with something, you don't know it that well. And you have a problem, you want to solve it. And you Google for the solution and you find the library that does that for you.
18:01
So for example, I had a nested model. So, you know, something like, you would have a model with a name, attributes, and then you might have, like, children. And then an array with things inside here. And I can figure out how to split that.
18:25
Or how to do that with Backbone easily. So you have collections and models in Backbone. So you would have a collection of the top model, and then you would have each of these models would have a children collection, which again would have their own models.
18:41
So to begin with, I can figure out how to do this, but someone made a library called Backbone relational, which is, I think, 2,000 lines of code, which is probably twice the size of Backbone itself. So I pulled that in until one day I figured out how to do it, and it was very simple.
19:02
So I could then delete 2,000 lines of code by adding and run 30 or something. It just saved a lot of code, so I could just delete all this by just using a simple parse method.
19:21
I don't know if you can, maybe you can see it here. Just doing this, basically. And split that out into the separate models myself. So it was another lesson learned.
19:41
The nature of our application is that it's essentially a CV database, which stores all your skills and projects and everything you've done. Sometimes salespeople, for example, they want to search and find skills. And we also want to give people auto-completion,
20:02
auto-suggestions, et cetera. So we needed a search engine. And the obvious choice was Elasticsearch. It's very scalable. It's got a JSON API. And it was really, really easy to get started with. So I just, we already had the data in JSON,
20:22
so I just drew it in there and searched, and things came back. And it seemed to work. Very nice. Until you start fiddling or running to edge cases. So, for example, an example would be
20:40
search for C Sharp or .net or something like that. By default, like a good search engine, Elasticsearch will strip out stop words and punctuations and all this. So you have to specify an analyzer that does not do that.
21:03
So we ended up with for the big text fields. Let's see if I can find that. This is a regular expression pattern that it will split on commas and whitespace, I think.
21:28
And also, periods, but not if they are on the beginning of a word, because of .net. And it took me, this was not standard,
21:42
but they have this, it's very flexible, so you can specify this, but it took me a while to understand how to create this and how to specify that. The same goes for, let me see if I can show you an example use case.
22:01
So you would fill in a customer, and you need suggestions, and you get some more suggestions, and all this comes from the API, and we also have tags here. The problem is when I search for, start typing Java in lower case,
22:22
or if I start typing it in upper case, I want it to hit the same thing. I really mean the same thing. I want it to be case insensitive. So what Elasticsearch does, or usually does by default, is to just lower case everything, so it stores everything lower case,
22:40
and then all the search phrases that comes in, they go through the same kind of filter that also lower cases your search terms. We're always matching lower case against lower case, which is very nice, until you want to use something like facets, which typically gives you the top 10 scales, which are what we use for these kind of recommendations
23:05
or suggestions. The problem is then if you get the top 10, and they're all lower case, you can't use them. So we figured out you could actually use a multi-field mapping,
23:21
which looks something like this. So you have a description, which is stored twice. So one time it's using one analyzer, and then you store it untouched, so basically not analyzed at all. And that means you have to specify when you search, okay, when I search,
23:41
I want to search in this field, but when I want the facets, you use the untouched one, et cetera, et cetera. And it takes a while to get all these things right, and there are still some edge cases, which we're working out. So you end up with quite a huge mapping. Obviously, you can generate this with code, most of it.
24:02
And there are some good frameworks wrapping it, but it takes a while to get everything right. But I still really like Elasticsearch. Another issue is similar to what I had client-side. It's you have a nested document, and you want to search in it. So there are two ways to store this in Elasticsearch,
24:23
or how to map this. You can say, okay, we store it as one document, and we say this document has lots of child documents. The problem is then when you search and you find a match, it returns the whole parent document. It never returns just the children.
24:44
So to be able to do that, you have to store them in separate documents and then specify a parent-child relationship, which is okay, but it takes a while to try out all these things and get it right.
25:01
We also had to make sure the search engine is kind of live and responding, and it turned out it kind of slows down if you don't kind of ping it all the time. So we, like every other thing, just set up something that just pings it. It's got a very good developer community, though,
25:23
and they are actually based in Amsterdam, I think, the headquarters for the company that's sort of behind Elasticsearch, and there's been several Elasticsearch meetups here in Oslo with all these, several of these people who actually build this thing. I think maybe Martin is even here at the conference.
25:42
So that's been a really good thing for me. I met them, I talked to them, they actually fixed some issues that I found really fast. So that's a good thing, I think, and I really look forward to how it will develop. So MongoDB, NoSQL database.
26:02
It kind of made sense that we have CVs, they are documents, store them in JSON, throw them into MongoDB, don't worry about schemas, nothing. It's been very nice, a long time. I haven't had to worry about migrations or data types. I just push data in there,
26:22
and it's probably one of the reasons that we have actually been able to develop this as quickly as we have. However, when you do have to do migrations, like move a field from somewhere, it does take, it's not straightforward. You can write some kind of custom JavaScript thing
26:40
that you run on MongoDB, or you can write it in Ruby, or read out the field from one document and write it with a new key in the other and then another script that kind of cleans it all up afterwards, which is something you get a bit more for free with the usual database migrations
27:00
on a SQL database, I think. And I'm starting to feel the pain about not having joins. So you have lots of documents, and it's nice when you just retrieve the whole document, and this one document is what you ask for, but when you need 200 documents, and you need some information from this, some from this,
27:20
and it just takes a bit longer than it should, I think. So I've been looking a lot into Postgres, which now has JSON support, quite extensively, actually, in 9.4. So at the moment, I'm kind of feeling I should say goodbye to Mongo, and go back.
27:42
So I'm not sure yet, but I think that's where we're heading. MongoDB, it scales okay, but the way you configure this is quite tricky, I think. Yeah, we have three MongoDB servers,
28:02
which is called a replica set. You have to have three because they vote, so if you have two, they'll never agree, so you always have to have the third one. And if you go beyond that, you have to have five, seven, et cetera. The challenge is that when you want to have backups,
28:22
you don't want to take backups from the main master. So you can specify that one of these, they can never be elected master, they have a priority of zero, and they are sort of hidden from the set, but they are part of the voting, so that's where you take the backups to.
28:41
And there are three, four, five different ways of backing up MongoDB, but just as not recommended, not recommended, not recommended. The only recommendation is to snapshot the entire disk, and just store that, and that's what we do. And it's easy with Amazon, but feels a bit so weird, because if you don't have to recreate something
29:02
from the backup, you have to start a new server, attach that backup image, and sort of go from there. Although Amazon does have quite good support for taking snapshots of your disk.
29:20
The other hard part was to actually generate documents. So when we started out, we thought we could get away with just doing PDFs. And there are some frameworks that allows you to generate PDFs from HTML. They're out there, they work like this,
29:42
but that's where it started, until people started demanding doc, like doc and docx, et cetera. So we had to figure out, okay, how do we generate documents? I'm not coming from kind of the Microsoft world, so this was a bit new to me. What we ended up with was using word templates.
30:07
It looks a bit like this. We found a framework or a third-party application that can take,
30:22
these are almost like the, call it built-in merge fields, but they're only just plain text, but these will be replaced by this framework with merge fields, and then they run that through OpenOffice, and out comes a document. So you post some JSON to it, and it turns this into a Word document,
30:43
which it works, and it's quite flexible, but there are lots of edge cases. So for example, images are transparent. They won't be transparent when they come out in the other end. You can't float everything like you want to.
31:02
There's sometimes the differences between docx and PDFs. I said some of these are due to bugs, or missing features in OpenOffice. So it works, and we always figure out how to do this, but it's quite a steep learning curve,
31:21
and we want our customers to be able to do this really fast and easy. So what we're looking at now is, okay, can we do this another way? So what we want to do is to have like a drag-and-drop interface and just generate the documents from there. So we looked into something called Apache, P-O-I, which is,
31:41
I think it's really good for Excel documents. They kind of call it backwards engineer this thing, but, and then read this sentence, which is, we do not have someone taking care of this. We need someone to take this thing
32:02
under his hood as his baby, blah, blah, blah. So this part is I wanted to use, which generate documents is now an orphan child waiting to be adopted. So that was a no-go. What we're looking into now is a framework called S-Pose, which based from our trials is really good
32:23
at abstracting all these different formats. So you can specify something with this library and it will come out exactly the same in PDF word, whatever. I think they have .NET version as well, but we are using the Java one,
32:41
and just because we can, we are wrapping it with closure. This is early-stage prototyping, so I can tell you more about how that worked out later. What I learned before I started this and after I've done this
33:01
is that monitoring is very, very important. You need to make sure that your servers are up and running all the time, and you need to monitor them a lot. So we use on a very low level, I don't know if you know the Unix world at all, but we use something called Monit, which just checks disk space, memory usage, etc.,
33:23
and knows how to restart services when something goes wrong, and they also notify you via email. We've been using that a lot. That is very easy to set up with Chef, so let's see if I can find an example of that.
33:41
I think it's a workers. So with Chef, it's as simple as just saying, okay, for this worker, I want to create a Monit file, and it looks something like this. So check the process with this process ID,
34:00
and start it like this, and you stop it like this, and you can have lots of these different checks, which checks all kinds of things for you. So that makes sure it's up and running. Chef also has a daemon that you can specify, so that will basically run, well, when I restart,
34:21
it runs almost every five minutes, and it just makes sure that everything is up to date, and everything is installed as you told it to. So I once tried to stop. I wanted to do something, a manual change from the server, and I tried to stop Nginx, and it just kept coming up again,
34:40
and it's because of Monit. So I stopped Monit, and now it still comes up again. That's because Chef restarted Monit, and Monit restarts Chef, so it's almost impossible to stop anything on the servers now, but I guess that's a good thing. New Relic is like an online tool. It's very useful for figuring out where in your code
35:03
things are slow, so it tells you something like which controller is being used most, which is the slowest one, is it the database query that is slow, or is it this model or this service, whatever that is slowing you down, and that actually works both client-side and server-side.
35:21
So that's, and they also ping your servers from around the world, and it tells you if something goes down. Same thing with Pingdom. They just ping it. We have a status page that just says, am I connected to a database? Am I connected to this? Blah, blah, blah. If not, return something else than 200,
35:42
and Pingdom will notify me. I learned the lesson the hard way, so when you do this more or less on your own, and something goes wrong, it sends out an email, but you know how you connect your mobile to Wi-Fi, and sometimes you're connected to Wi-Fi,
36:00
but you're not connected to the internet. So I was at home one day, and I just walked out the door in the morning, and it's like, whoa, just got tons of emails saying everything was down, and it's like, why didn't it tell me this before? And it's because I wasn't connected to the internet, so now I have SMS notifications in addition to that,
36:21
so just a lesson learned. I mean, I use something called Airbreak, which is basically hooks into every exception, or uncaught exception, both client-side and server-side. It sends it to the service, and you get an email saying, okay, this thing went wrong on this URL.
36:42
This is the stack trace. These are all the request parameters, et cetera, et cetera. So it's very useful. Instead of having to monitor your log files and look for errors, it just tells you immediately where it is, and then you can fix it fast.
37:03
So I think a few other tools I've used is Intercom, which is, let's see, it's basically just a chat here. So any user just says help me, and you get notified on an app or by email,
37:24
and if you answer fast enough, they'll get the response back in the app. If not, we send them an email, and that all happens automatically, and that's increased a lot of feedback from our users. So it's a very, feel like you're quite close to the user, which is good.
37:42
I've also been using something called Mixpanel for analytics, but it's not been giving us that much value for now. So what has gone wrong? Let's show you, so we have all these pictures, and they are stored on Amazon,
38:03
but we can't just store them open because it's kind of private to the customers, but it's still nice to leverage Amazon's infrastructure in terms of hosting. So the way it works is that when I asked my model for the URL to that image, it gives me a URL with a key that lasts for like an hour.
38:23
So if you have this parameter, you can download that image, and you can cache it for an hour. However, every time I did a new search, it would just generate a new URL, which then lasted for like, well, a second later, it will last for a second later. So I wanted to cache this, and I started caching these URLs
38:44
in something called Redis key value store, very high performance. And that worked fine, until one day someone was like, the app is really slow, and I was searching, and it's like takes forever, but it works. So I wasn't getting any real errors.
39:01
It's just, it was very slow. And I thought I'd set the timeout to one second, which I think that was, that's quite low. Okay, if it times out after a second, it should just still work. But for some reason, it was taking longer, and that's obviously because the timeout was one second per image.
39:22
So it was kind of queuing up always. So I ended up setting the timeout to like 0.0, one second. So it times out immediately if it can't fetch it. So even if that's 100 images, it still probably just totals to a second, and it still works.
39:41
So we have source code on GitHub, which I think is really nice, but there have been, people have at least tried to hack them quite a lot lately. So there has been some distributed Nilo service attacks on them. And that actually means that GitHub
40:00
might be unavailable for an hour, which is okay, but if you're about to deploy or just in the middle of a deployment, and you're trying to fetch code, the new code from GitHub and put it on your server, then that is a problem. So I'm not quite sure how to solve that. You could always mirror it somewhere else, and maybe that's the solution,
40:21
but I'm not sure. So I mentioned these servers on Amazon. It hasn't happened in a while, but it did happen that they became unavailable, where you might get an email saying, we're going to retire this server. You need to get off it within a week.
40:42
And then it might just happen the day after. So that happened to us once as well, because there is always a reason they are retiring them. So I think they find out something is wrong with them, and they try to keep them running. But what I learned is that as soon as you get an email
41:01
saying, oh, we'll retire this in a month, just do it immediately, because it's probably a reason they aren't retiring them. We also do single sign-on integration. So we support Google Apps and active directory federated services integration. So you log in, you're redirected
41:23
to one of our customers' ADFS server that is exposed to the internet. They log in there, and they sign requests and back to us with a valid email address, et cetera, which we then validate. And that worked. That wasn't too hard to set up,
41:40
and it's been working fine until one day I had replaced the server with a new one, but it was exactly the same, because I used Jeff's exactly same stack. It was not a new version of the code. So it's like, what went wrong? Why isn't this working? It worked yesterday. It's the exact same thing.
42:01
So it turned out the clock on the new server was a bit off when compared to our customers' servers. So you need to take that into account. So there are some tools that allows you to synchronize clocks with some official place. That helped. And you can also increase the allowed drift a bit.
42:24
But that was a tricky one. So going forward, we're trying to make the app more interactive. So two people work on the same document. We want them to be able to do that seamlessly without overwriting the other ones' changes
42:41
or just getting a conflict and saying, no, you can't send this. So we've been trying this service called Pusher, which is essentially WebSockets. So if someone does a change on this one, goes to the server, the server tells Pusher, Pusher sends it through WebSockets to the client
43:03
or the other user, and he can fetch the latest data. But that is tricky to get, correct? Because when you open and close your laptops and you reconnect, you move around, everything can happen. And it's hard to know the state on the client side. So that's our experience from that so far.
43:20
We've been using it a little bit, but not extensively yet. So I guess I think that was gone through our stack and kind of how that has evolved and what has worked and what hasn't worked that much.
43:43
Any questions or comments?
44:00
Shared JS. Thanks. So that's interesting. What does it do? Ah, nice.
44:23
Nice, nice. We're looking for that, actually. Any other comments, questions? Something you want me to dive deeper into? I'd really appreciate if you used the comments box downstairs,
44:42
just the red, yellow, green ones. And if you have any suggestions or feedback, please come to me and tell me. I appreciate the honest feedback. So, yes.
45:12
That's a good question if we need it to be as scalable and if we could run a non-cloud solution. So I think obviously we have big ambitions.
45:23
We want to have lots and lots of customers on this. So we're building it to scale, but maybe we're doing it prematurely. The other thing I think is the flexibility, which I like. So if I want to play with, let's say I wanted to use Postgres instead of MongoDB
45:42
or I wanted to try a new library somewhere, it's just a quick command and I have a new server. I can play with it and I can nuke it again if I don't want it. I kind of like that.
46:02
And I can scale them up and down. So most of these servers are small instances or micro-instances for now. So they don't cost that much. But they give us, I mean, it's just a click of a button and you have a new server, which is, I haven't found the equivalent in terms of having an in-house server,
46:22
which you have to. But I see your point. Obviously if we run, maybe we could run all this on one box and it would still work then. Maybe it's a bit over-engineered for now. I think I can agree with that. But we are trying to build something that can scale
46:42
and it will have to scale very soon. But it's obviously hard to know which parts you need to scale up front.
47:02
But doing a startup, start counting hours, I think you get crazy anyway. But no, I agree. It could be worth doing a calculation to figure out if it's actually worth it. But it does give me the confidence
47:22
that I can scale almost overnight if we need to, which is nice. And we can say that to customers.
47:46
So obviously, coming from a developer background, sales marketing is not my strength, really. But I think marketing, we're still pretty bad at. But the sales part, we're getting better at.
48:01
But we do everything from just meeting people here, because our clients are consultancies. So I can do docs here. And I reach my target market that way. And just being part of the community, which I enjoy from before. So for me, it's a win-win. I can meet people.
48:20
I enjoy talking about technical stuff and process stuff, et cetera, with them. And they are potential clients as well. Which is good. But yeah. I think just for us,
48:40
it's just word of mouth. Just being out there, that works. And then that gives you a lead somewhere. And then it's your responsibility to take it from there. But business to business, and being few people, being something new,
49:03
it's, that's what I tried to talk about earlier. There's like the security requirements, the privacy requirements, the backup solutions, all that. You have to make sure you reduce the risk for your clients. So we dump all the CVs in Word documents
49:21
to your preferred storage solution every night, just so you have it. So you can trust that if we disappear or go down, which we don't do. But still, you're safe and you have all your data. But in terms of sales, yeah, it's just talking to people, reaching out, being visible,
49:41
and sometimes cold call. But it's quite long sales cycles, at least for the large clients. There's lots of legal stuff and just getting things to happen takes a while.
50:00
I don't know if that answered anything, but yes, yes, yes, yes, yes.
50:30
Yes.
50:54
Yeah, oh, nice.
51:25
The nice thing about CFX is it converges, so you're saying you change something and put it back again. What you got was Monit and Chef trying to do both of them, doing dynamic provisioning. So if you split it into the static provisioning and then have Monit look at the static stuff
51:41
and do all the dynamic things, it works treatably. You just change the files, replicate, you're done. Yes, that sounds like a good solution, I think. But you will have to go in that template. No, that was just a simple example. We have some bit more complicated ones, but Chef is running,
52:02
but it's mostly just installing. On the third, fourth run, it doesn't do anything really, but obviously if something has changed, it will probably check that most of the services are running, but it's not really its job. It just happens to do that. But I agree separating responsibilities,
52:23
there would be nice. And as I said, Chef is probably, it could be how I use it, but I feel it's not perfect. All these DevOps things, they are quite immature or new,
52:41
and I think people are trying to find the right kind of balance or separation there. So that's why I'm a bit curious about Docker, for example. Cool. I'd love to talk more about that. Anything else?
53:01
Yes? But is that just not another, sorry, so Gitorious instead of? Okay.
53:22
So I love all the features with GitHub, but probably it would be nice to, or advice to keep a clone somewhere, and maybe Gitorious would be a good option. Yes. Cool. Thanks.
53:41
Yes? Thank you.