Thanks for coming out late Thursday, guys. It's been a long couple of days. I'm Nick Merwin, the founder of coveralls.io.

And I hope you guys have seen some of our badges around town. GitHub, ReadMe, et cetera. I don't know. We're kind of all over the place, hopefully, right now. But today, we're going to discuss how it went about figuring out how to get coveralls, a SaaS app, into the hands of customers that wanted to run it within

their corporate networks. It was pretty uncharted territory for me. But I hope this general overview and what we'll see later with just a really basic prototype will help you guys get some confidence that you could also take your app to the next level by offering a hosted version to

a customer base that maybe you never thought was possible. So in that sense, it's kind of going to be partially a SaaS model discussion and some just general sysadminning stuff. And then, of course, all kind of viewed through the lens of Rails since coveralls is a Rails app.

All right, so why would you want to set your app free into the dark world outside of your cushy deployment environment? So currently, let's say your app is hosted on Heroku or a VPS.

Coveralls is on DigitalOcean. And if you have a subscription-based service where users are paying by the month, and maybe there's a few tiers and usage-based add-ons, and it's humming along, accruing users, and it seems like your potential customer base is covered. But you might be neglecting an even more lucrative and even preferable customer base, which would be the

enterprise, quote unquote. And this could be monolithic companies where dev teams have a much harder time getting upper management to embrace cloud-based tools. Or perhaps the issue is just that setting up a $5 a month service fee takes a mountain of vendor approval paperwork.

So let's chat about how it can make sense to offer a hosted version of your app in addition to your cloud service. So there are three kind of tenets that I ran across in figuring this out as to why you would want to do this.

And the first one is security. And it's pretty obvious that we all worry that when you use a cloud app, you essentially give up control of your data, and you assume that the devs on the other side hopefully use bcrypt on your password. But you never really know for sure and just hope there's never a real data breach. But beyond passwords, it could be any sort of data that your

company's security department has forbidden from leaving their networks. And in fact, it could preclude the possibility of them ever using your app. This could be maybe a medical company that wants to use your data mining tool but can't upload anything to the cloud due to HIPAA compliance issues. Or maybe it's a dev team that wants to use your source code analysis tools but can't risk sharing or leaking

proprietary code. There's also just general business apps like CRMs, internal tasking, chatting, scheduling, that would have plenty of info that those companies wouldn't want to share with a competitor. But by running a hosted version of your app within

their networks, they can just be more confident that they won't ever have their accounts hacked or their data compromised out on the open internet. So the second one is a bit more esoteric. Reliability, and this is just kind of a worst case thing, if your app that you're giving them is a DevOps tool that's part of the build or deployment pipeline, then

they're not going to want there to be any unplanned maintenance downtime. So for instance, with GitHub or Travis, if something goes down, something goes wrong, you don't have control of it, and it blocks your pushes or your builds, that could screw up your deployment. And so if you're running it internally, then your dev teams

and your sysadmins will always be able to coordinate to make sure everything's good to go when it comes to crunch time. So lastly is cost. And the traditional subscription model where it's just pay by the month doesn't really work when you've given over your app to a customer.

So oftentimes, those potential customers are also less price sensitive and could be willing to pay a premium for your hosted service and some sort of extra white glove support service. But once you deliver the app to them, you have to assume that it won't be able to phone home to check the

subscription status. So instead of a time-based subscription, like a monthly fee, you probably are going to need to do some sort of seat-based usage or something else that's beyond just the monthly or perhaps yearly. And then once you're thinking about seats, then you can do

seat packs as tiers, as in the number of active users that can access the app as it's living within their network. You could also build some sort of self-destruct or shutdown mechanism that will force the customer to purchase or renew the license after a set amount of time, like six months or a year, just to make sure that they

come back and they keep a valid license. So there's a lot of different opportunity to work out business model kinks beyond just a traditional SaaS model. So before we go into the how we figured this out, let me

just mention coveralls and what it does and why we took it to the enterprise. So coveralls, really quick, coveralls is a tool for code coverage tracking and notifications. We can annotate your pull requests with success or failure statuses based on changes in your coverage percentage, or just send messages to your chat or email

with updates. This helps dev teams make sure they don't deliver untested code, especially in a production. And on site, you can see line-by-line coverage of the code base, the cloud version for open source is free, but we charge for private repos. And the rub comes because while we don't actually store

source on our servers, we do store private scoped OAuth tokens for GitHub. And so for some companies, that's just a non-starter. They can't allow us to see within their private repos. And then some of them are already using GitHub Enterprise, so they're not using Cloud GitHub to start

out with, and the two are not interoperable. So what we did is gathered interest from a potential customer base over a matter of months, and it became clear that there was enough of an interest that we should try

to do this and try to make it happen. And so that's what brought us here. So there are some pretty big main hurdles that I found in converting your general cloud Rails app to a Fog one where it's living below the cloud layer.

So delivery and installation. Let's talk about delivery first. So the best user experience for your customers would be to have the least amount of work to get your app up and running in their environment, obviously. So three download files would be, I think, is the most

consolidated way that we can achieve this. And that's first the virtual machine that the app is going to run on, and that could just be a Linux box, as I'll show you, a virtual Linux box. And that's typically between 800 and a gigabyte size download. The next one would be the packaged up app files itself,

and that's a decently high number, sometimes between 50 and 100 or more, depending on how many gems you have, because all the gems need to be bundled and vendored into the app package itself, as we'll see. And then lastly, a license file that's specific to the

customer and is generated for every customer. And so out of those three, your customers would probably only need to download the VM once in a blue moon when you do big upgrades to the underlying system or have new dependencies. The app package is something that they would download

whenever there's updates, and we'll get to that. And the license file would be around the same amount quicker, like when they go to trial to active or what. So actually, let's talk about those incremental updates for a sec. With a standard deployment, obviously, when your code is

hosted up on Heroku, getting features and fixes out to your users is easy as Git push. Not so in this case when your app is running completely out somewhere in the wild where you have no access to it. So that's where the app packages come in. It's a smaller download that has just the most up-to-date

bug fixes, features, et cetera, that is a much quicker download than having to re-download and configure an entire virtual machine. So also, for the license files, one last thing, it makes sense that this is the part where you'll probably need to

build a secondary app in addition to your main cloud one that will just service those customers and charge them subscription fees and have trial accounts and all that stuff. So that's where they'll download the license file and

then download the other two in the secondary app. All right. So installation-wise, the networking config is the first thing that they're going to see. And since you don't want them to have to log into the VM themselves, the best way to get around this is to

build a tiny little menu-driven Ruby app that gets presented on boot. And I'll show you guys how we did this. And that way, the customer won't need to know any of the specifics about Linux networking. In addition to that, external access just has to

be assumed as not going to be a possibility. So hopefully, the reason why they're purchasing this hosted version is so they can be completely confident in the security behind it. They won't give it any external access. It'll be totally walled off in a sandbox in their network,

hopefully. So the assumption of there not being any way to access it from the outside is probably valid. Last one is just process management. And that's how are we going to get background jobs and the server itself to stay alive and start at boot.

All right, a couple more of these hurdles. So product support, generally, you can see immediately when a user hits a 500 error, you're going to get an email, and you're going to be able to go on to your error tracking service, Airbreak, whatever, and take a look.

In this case, there's no way to know when that happens, there's no way to get pinged. So you're just going to get an email from your customer saying something's not right, what's going on. So you need to provide them a way to be able to send you details about the exception. And so we're going to look at how to just keep those exceptions somewhere that we can then let your customer

send to you and attach to an email or something. So that goes same for logging. If something's really funky on it, you're going to want to be able to see if there's any weird parameters coming through the routes, especially when they're not causing an exception, per se.

Lastly, there's resource management. If it seems like things are running slowly on their site, there needs to be some way to address that. I'm not going to go over that in this talk, because I feel like that's more of a sysadminning issue. But that's something for homework, I guess.

So lastly is the intellectual property question, and this is a big one. And because you're giving them a VM, they can unpack the VM and mount the disk and look at everything you put there. So even though it comes in one nice file, it's still extractable, and you can just load up another VM in VirtualBox or VMware and then mount the disk that you

had delivered them. So that means they're going to be able to look at everything as though they were a root user. So I feel like database access, unless, I don't know, there might be some way to get around this, but you should just assume that they're going to have access to the database itself.

And it doesn't really matter. I don't know. That's the bigger question here. Along with code obfuscation, since Ruby, there's no way to really, really fully protect your source code once it's being executed in somebody else's environment, and it has to actually work itself, you can write a code

obfuscator as a deterrent. And that's what we've done for coveralls Enterprise. But I think that prevents people from immediately reading it if they just mount your disk and look through your source file tree. But it's probably safest to just cover this with your

license and say something like, the license you're purchasing from us covers only the use of the software. Modification and redistribution are not permitted. So it just kind of comes down to legalese as being your last line of defense here.

And if this keeps you up at night, then maybe the best thing to remember is that your customers for your app should be most interested in getting updates, getting bug fixes, new features, and support rather than breaking into your source code, reading it, copying it, spreading it out on the internet.

So that could be its own talk about how to actually achieve some level of code obfuscation. But we're not going to go over that really today. OK, so let's get into the nitty gritty here. So this is a pretty hectic diagram.

But it's basically showing the general architecture of how the app is running within our VM, which is Ubuntu. And we chose Ubuntu because it's just widely used and relatively easy to configure.

So the main components that live within it are the network config app, which is the first thing that customers will see when you boot it up. And that lets you do things like select static networking or DHCP, and then set the name servers, and then reboot, shut down, et cetera.

So it's just kind of a simple little starter app. And then behind our web server, we'll go into why we chose passenger for it. There's going to be a pre-installed app that I just made in Sinatra. It's a really simple app that facilitates the unpacking of

the package file that contains the actual app or the app updates and the license file. And so within that, behind the web server will live your Rails app also. So the Rails app itself, some of the things that these

three components, I think, are the main ones that set it apart from your standard cloud hosted app or where you started from to begin with to get it to this point. So the license file reader is going to read and cache your license at the boot when your Rails app boots.

And then it's checked on every page view. And then you can do things like enable or disable features or lock the app down. The second one is data import and export, because when you need to download a brand new upgraded virtual machine, say it's got new dependencies for new features, then you're

going to need to give your customers a way to get their data out of the old one and get it into the new one. So that's things like dumping the database. If your app has uploads, then you're going to need to pull those together if the uploads are living on the VM. And then anything else that the users might have uploaded or

changed that has a state on the VM. And the third one is support package generation and downloads. So I'll show how we can use the Rails rescue from in the most general of terms just to collect all the exceptions that happen in controllers, at least. And they can be archived into a temp directory.

And then when an admin for the app, as it's living on the VM, is ready or needs some support, they can hit a link that generates an archived file that we can encrypt also and then email it or Dropbox it over to us.

OK, so setting up the environment first. For development, I think it's probably easiest to use VirtualBox. It's free and pretty simple. And it's really easy to export an appliance from your

machine called an OVA file. And that's just a nice little, it's a tar or a zip or something with a .ova extension. And that's just something easy you can link to and serve from your CDN. Also, Ubuntu, just because of how widely it's used, hopefully

we can trust its built-in security settings and standards. So we kind of get that out of the box, and our customers get that confidence out of the box. So when you provision it for your Rails app, you obviously want to use the minimum amount of dependencies and touches to it so that you can just keep the download

size smallest. If you don't need to install something like Image Magic, then just don't do it. There's no point. And it's pretty easy to end up with a two gigabyte or more virtual machine, whereas you probably could have kept it slimmed completely down to a one gigabyte.

OK. So also, when you're doing development on the virtual machine, it helps to have two separate versions of it, one that you're going to use just for

packaging up your app. Because you're going to need to have the full app sitting somewhere on the virtual machine so that you can run a bundle to vendor the gems into the app itself to get ready to be completely packaged up. It's not something you should do in your local

development environment. Say if you're developing on OS X and you vendor your gems in that environment, they're not going to run on Linux or once you're in Linux. So all right. Next, let's talk about networking configuration.

That's just a little screenshot of what a user will be prompted with when they, or a customer, when they load up the virtual machine the first time, or every time, really. You can still SSH into it. It's not like completely blocking everything off. But this is just sitting within a file in Ubuntu called

the TTY, the teletype. And it's pretty damn simple to set up. And really, you're just, to make this work, the bare minimum is just having some system calls that write to the file, the interface file, or the DNS file.

So there's also shut down and reboot options. So here's some of that. Can you guys read that? It's pretty small. I have it on a text editor, too. So yeah, like I was saying, it's a menu-driven app.

And it's just basically collecting information from system calls, and then displaying it, and then letting you input things just using getS as a very simple Ruby app, and then letting you just reboot the entire computer. So it's pretty standard, simple, and no rocket science

here. So the next part of the puzzle is the server itself. And I chose Passenger for it because it just seemed the most dead simple to put your app in a directory and have

Passenger just start serving it. Once you hit Touch Temp Restart, then Passenger will load the new code. And it doesn't really care if the code isn't there to begin with. It's not going to completely blow up. It's just going to 404. So that means the Loader app that we'll talk about in the

next slide can easily just extract the packaged up Rails app into a directory that's already been pre-configured for Passenger to serve the app from. And as you can see, sort of on line 14, that's where we're running the Loader app.

So as soon as you boot up the VM for the first time, those error pages will actually, if the Rails app is not found, it'll redirect you to the setup page. The setup page is where you can upload your package file and your license file, and it'll be extracted into its eventual living place for good.

So the Installer app itself is just a simple Sinatra app that takes the package and the license file, puts them in the place where they'll live.

And this one, you want to have pre-installed on the VM for distribution, because it's going to be doing the work to get Rails up and running. So we can go over a bit of what is actually happening in

the Sinatra app. I'm using a simple encryption library called gibberish that just is a nice abstraction over, I believe, OpenSSL, and then this horribly insecure shared key up there, asdf, asdf. And if you do have your code obfuscated, then it doesn't

really matter that the key is right there. And it's just best to assume that if people are going to be looking into your code, they're going to figure out everything about it to begin with, so you don't really need a huge key.

OK. So what it does is it first will decrypt the license file to make sure that it's valid, or it will just use that license file. Let's say if you're upgrading your license, it can

just put that in your Rails app, and your Rails app will start picking up the new license. Maybe it's gone from trial to live. So if the package itself is present, the package file, then it'll decrypt it and place it in the directory

where passenger is pre-configured to read it from or to load it from. And then it'll just touch the restart file and redirect you straight to root, and so you'll be good to go. So right, that index is just how simple it can be.

I mean, this is like the bare minimum of what it takes. So tweaks to the app itself that we discussed. The license file in a regular customer subscription app, there's all the customer specifics in the

database, and that's checked on every page load. But we can't do that here because there's no way for the subscription side that you're selling them on your sales app to have any effect to the Rails app that's running on their side.

So when they get a new license file, it contains within it, it's just an encrypted JSON file, and it'll live in the temp directory and get loaded on boot every time. So we also want to add some of the Rails secrets, well, all of them. And for the simple example app, I'm running Devise also.

So the secret token in Devise secret can be read from the license file on boot. And you want to do that because you don't want one customer to be able to tamper with another customer's cookies. It's a really kind of extreme use case or possibility, but still just best practices.

All right, let's see. So this is a really basic module that demonstrates how to read and write the license file. It's using the same encryption key as the loader. And this would obviously be obfuscated, hopefully somewhat, just for a deterrent.

But it gives you an idea of how simple it is. It's the first thing to get loaded in the initializer so that the rest of Rails can use it when it's booting up. And from here we can have calls out to it all over the app, checking the trial period, disabling functionality,

displaying messages to link out to the web if they need to upgrade their license. And another big one here is the seat limit. And the seat limit is something that can be checked, say, in a user validator, like validate seats and just do a count against the license seat number.

In that case also, for administrators, you'd also want to provide a way for them to manage the users, of course, as though they were using your cloud app. So they would be able to deactivate or delete users so as to free up seats if they needed to.

All right, so next is the support package. And this is the really simple implementation of an exception tracker. And it's going to put exceptions in little encrypted files with a clean back trace into your temp directory to be ready for download by the admin when

the time comes when they start hitting 500s. And that just encrypts it back up and tarz it and just is able to be emailed to you or Dropboxed to you, depending on how big it is.

All right, lastly is the data import and export management mechanism. So this is just a dead simple PG restore and PG dump. These are Postgres commands as this would be in a controller. So the first one is purely just taking whatever you spit

at it and attempting to shove it into the database. And export is just giving you an entire dump. So nothing really tricky here, but it's definitely an important part of being able to upgrade

between virtual machines. For background jobs, I'm not going to get into it too much, but we used Form and Export, which allows you to take your proc file and generate upstart processes. And those can be used by Ubuntu, and so that'll let

them be executed on boot and stay running, hopefully. I think the next step here would be for your app itself to show the background job status within the app itself in some sort of dashboard. And of course, that wouldn't be something you would show on your cloud app.

But just something to think about. So lastly, when all of your pieces are in place, what's the best way to make it actually distributable? Hopefully, you can pare it down to a single script because it's just like a git push Heroku. This would be something that deploys to your CDN and maybe

updates your users that there's a new version of it ready for download. So some basic things to do here. This is super basic rake task that just grabs the version, does a bundle, a vendored bundle that throws everything

in your vendor directory, and then tar's everything up that's pertinent, excludes what's not, and then encrypts it all using your shared key. And that's it. So some of the things that it could do are obfuscation at

this step. If you needed to run a post processor on it, then it would happen here. And then the last one would be actually uploading it to your CDN and then notifying your users. And I keep a version file in the root just so that all

these scripts and random things can use it. Because each download of the package is going to have some sort of version attached to it that your users are going to be able to want to go back and forth or be able to identify. So that was pretty much it for what you can get up and

running as a prototype. And beyond that, once you actually have it in customers' hands, if they're already using AWS, they might want to see it as an Amazon image which can be spun up without actually having to download anything.

It's pretty trivial to convert an OVA file to an AMI. Amazon provides some command line tools to make this simple, so you could also put that into your packaging script. Actually, no, this would be a different script that would run purely on your VM when you're ready to cut a new version. And then once you have an AMI listed on Amazon, it's pretty

trivial to just hit the public button, and then it's searchable in the public registry. So some more things that are going to be taking the next step beyond what we've covered. Resource management, there should be an easy way for

your admin to see how your VM is performing. And this could be part of the support package also. Clustering, perhaps you could set up a mechanism to run multiple VMs at once for better performance.

Mail server, which is not that tricky, but could be important since in your cloud app, you're probably using SendGrid or Mailgun or whatever. Those aren't going to be accessible from the VM. So your admin should be able to specify a mail server. Incremental VM updates, you could also include bash

scripts to change pieces of the VM around when you import a new package file. So when that gets exported or untarred and migrations are ran, those migrations could also include, say, new dependencies that were stored inside the

package because you can't just run an apt-get install from there, you have to assume that your VM doesn't have any access to the outside world. So we talked a little about the SaaS app that would run

in the cloud in addition to your main one to sell license files to facilitate that. And last one is enterprise sales, but that's just a joke, because who knows about that? That's the big question mark, which I have no idea about either and is definitely uncharted territory.

So I think we're running out of time, so maybe we should just do any questions? Yeah, the question was, what are we doing for obfuscation? That's still kind of a work in progress. Right now, it's a Ruby C extension.

And the way that GitHub does it is they've compiled their own Ruby to do it. So it's at a lower level than just a gem extension. And so figuring out how to do that right now, but it's not something that's widely discussed. Yeah, well, the question is, how do we reduce the

amount of patches that we need to ship? Because there could be maybe a bug that's specific to one customer. But we don't want to have to have everybody else come and download it all at once. Or maybe you just did a black wall customer. Yeah, there's really not a good way to tighten up the

feedback loop beyond having to package up and then release brand new versions for every little bug piss. I mean, I think that's more of a customer relations question, where if it's a small bug, maybe it can wait into a bigger point release if it's being fixed just for

their environment. Or if it's something that's business critical and it just doesn't work for them and they're a paying customer, then, of course, no matter how small the change is, a new version has to be released and everybody will see the new version out there. Yeah, it could be incremental like that, where the package

in the migration will actually do the updates. But that would be for smaller things. Maybe you can include a .deb that can be loaded in that's not too huge. But for OS level, that would necessitate cutting a brand new VM and asking your customers to go download a new

one, then do the whole import, data export dance. Scaling support teams. Well, we haven't really had to scale ours yet, because we haven't had so many customers that it's become overwhelming. But I feel like we're going to just need to scale in

traditional ways. But it's yet to be seen what extra sort of hurdles we're going to have with supporting multiple versions of the package and the VM and having this kind of asynchronous support flow, where we get emails or

dropboxes of just packages of logs and 500 errors. So it's definitely uncharted territory also. So there's two pretty simple little apps up there that are just prototypes.

Entercom, calling Enterprise Communications, just lets you post a simple little blog app. But it shows off some of the license file reading. And pretty much all the screen grabs from the presentation were from that app, except for the app loader, which is the little Sinatra app.

Anything else? All right. Thanks for coming, guys. Thank you very much.