The journey from Powershell to Grunt for Build and Deploy
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 170 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/50627 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
AngleBuildingPoisson-KlammerDirectory serviceMultiplication signPartial derivativeComputer fileDifferent (Kate Ryan album)Projective planeCodeHand fanQuicksortCASE <Informatik>Repository (publishing)Poisson-KlammerWeb 2.0Group actionOpen setMathematicsBuildingPhysical systemCartesian coordinate systemScripting languageServer (computing)Product (business)SoftwareTable (information)Software developerSystem programmingForcing (mathematics)Open sourceTask (computing)Unit testingOrder (biology)Revision controlWindowMechatronicsData structureCellular automatonType theoryVotingInformation technology consultingGastropod shellPower (physics)Configuration spaceAngleBitBasis <Mathematik>Transformation (genetics)Logic gateMetropolitan area networkMereologyMaizeSubsetComputing platformNumberProgramming languageWebsiteOffice suiteSpeech synthesisProcess (computing)TouchscreenCategory of beingWeb pageReal numberBlogVideo game consoleWrapper (data mining)Descriptive statisticsApplication service providerC sharpAnalytic continuationVisualization (computer graphics)Complete metric spaceComputer animation
08:27
Graphic designFunction (mathematics)Configuration spaceError messageCompilerDefault (computer science)Active contour modelModule (mathematics)Local ringBuildingMathematicsExecution unitTask (computing)Scripting languageGamma functionMenu (computing)WikiProgrammable read-only memorySummierbarkeitMassReading (process)Link (knot theory)BackupSpecial unitary groupService (economics)Graph (mathematics)Computer fileComputer iconException handlingBlogRevision controlSoftware frameworkWeightData miningRollback (data management)Coma BerenicesConvex hullDirectory serviceAsynchronous Transfer ModeWitt algebraAnnulus (mathematics)Server (computing)Online helpHydraulic jumpMereologyPoint (geometry)Scripting languageElectronic mailing listSoftware developerServer (computing)Computer fileConfiguration spaceCartesian coordinate systemParameter (computer programming)Repository (publishing)MereologyOnline helpMultiplication signInternetworkingMessage passingQuicksortSoftware testingBuildingExecution unitResultantProgramming languageSoftwareWebsiteGoodness of fitIntegrated development environmentShared memoryShape (magazine)Directory serviceType theoryUltraviolet photoelectron spectroscopyCASE <Informatik>BackupForm (programming)BitPlanningSubsetLimit (category theory)Factory (trading post)Medical imagingDifferent (Kate Ryan album)Positional notationPhysical systemString (computer science)Product (business)Unit testingSlide rulePhase transitionWordMultilaterationOrder (biology)Series (mathematics)CodeSimilarity (geometry)Function (mathematics)Core dumpGreatest elementLastteilungAnalytic continuationSystem programmingFerry CorstenGame controllerWrapper (data mining)Cross-platformApplication service providerComputer animation
16:53
Right angleMobile appPhysical systemParameter (computer programming)BuildingScripting languageStatement (computer science)Configuration spaceWindowCASE <Informatik>MassMathematicsComputer animation
17:34
Configuration spaceParameter (computer programming)Demo (music)Normed vector spaceDrill commandsTime domainAsynchronous Transfer ModeMaxima and minimaPhysical systemSystem programmingVariable (mathematics)Arithmetic meanServer (computing)Electronic mailing listTask (computing)Service (economics)Branch (computer science)Absolute valuePhysical systemSoftware maintenanceQuicksortUniform resource locatorNetwork topologyProjective planeParameter (computer programming)Web 2.0IP addressTask (computing)Different (Kate Ryan album)Peg solitaireCartesian coordinate systemScripting languageSurface of revolutionInstallation artProgramming languageBuildingPlug-in (computing)Mobile appBit1 (number)Shared memoryMultilaterationCalculus of variationsRewritingMassSoftwareTouchscreenRight angleSet (mathematics)Presentation of a groupGame theoryComputer animation
20:08
Object (grammar)Task (computing)Computer fileConfiguration spaceTime domainNeighbourhood (graph theory)AreaExistenceMacro (computer science)Correlation and dependenceModule (mathematics)Link (knot theory)Asynchronous Transfer ModeSoftware testingTask (computing)Point (geometry)BlogPhysical systemRight angleType theoryUnit testingIn-Memory-DatenbankIntegrated development environmentCartesian coordinate systemLine (geometry)Electronic mailing listDirectory serviceDatabaseException handlingTablet computerSubsetConfiguration spaceShared memoryCASE <Informatik>Interactive televisionScripting languageInstallation artBuildingTouchscreenComputer fileFunctional (mathematics)Metropolitan area networkSimulationProcess (computing)Reading (process)Peg solitaireSoftware developerWebsiteDefault (computer science)MereologyComputer animation
24:04
Physical lawDivision (mathematics)Menu (computing)Read-only memoryAvatar (2009 film)Civil engineeringLine (geometry)Computer fileContent (media)Product (business)Task (computing)LogicSoftware developerConfiguration spaceParameter (computer programming)Computer animation
25:14
Task (computing)Stack (abstract data type)Open setTable (information)Software testingModule (mathematics)Computer configurationVertex (graph theory)Reading (process)Library (computing)Revision controlInternet service providerVirtual machineCloningDefault (computer science)Installation artLocal ringBlogDependent and independent variablesOperating systemMobile appComputer chessVirtual machineCartesian coordinate systemSubsetError messageSoftware developerSoftware testingSI-EinheitenProjective planeOpen source2 (number)Service (economics)Order (biology)Exception handlingTask (computing)Physical systemPoint (geometry)Directory serviceScripting languageCodeType theoryRepository (publishing)CloningPeg solitaireProduct (business)Line (geometry)Web pageSystem callBitGoodness of fitCuboidComputer fileSystem programmingResultantSource codeComputer animation
28:53
EmailReading (process)Mobile appContent (media)Boilerplate (text)Line (geometry)Execution unitQuadrilateralDisk read-and-write headSubject indexingRule of inferenceScripting languageMaxima and minimaSpeech synthesisRepository (publishing)Point (geometry)Scripting languageTable (information)DatabaseCross-platformProgramming languageInstallation artMobile appSoftware developerCartesian coordinate systemComputer fileFactory (trading post)Open setSoftware testingDeterminismBuildingSelf-organizationCodeLocal ringRight angleMultiplication signComputer animation
31:26
Message passingExecution unitDynamic random-access memoryMaxima and minimaConvex hullTask (computing)Stack (abstract data type)Directory serviceLibrary (computing)Default (computer science)Software frameworkRevision controlComputer fileError messageSystem programmingPlastikkarteExecution unitSingle-precision floating-point formatSequelVirtual machineStorage area networkScripting languageIntegrated development environmentMultiplication signPoint (geometry)Mobile appNumberPeg solitaireTask (computing)MathematicsTransformation (genetics)Host Identity ProtocolState of matterTraffic reportingEndliche ModelltheorieGreatest elementMessage passingBuildingKey (cryptography)Computer animationDiagram
33:44
Scripting languageType theoryProgramming languageGroup actionSpeech synthesisMultiplication signSoftware frameworkBlogScripting languageArithmetic meanPoint (geometry)Mobile appBitWeightDampingWordRewritingBuildingCartesian coordinate systemProcess (computing)Physical systemSystem programmingCalculus of variationsExterior algebraUsabilityCASE <Informatik>Java appletPerturbation theoryAxiom of choiceApplication service providerCausalityXML
37:20
Revision controlSoftware frameworkDirectory serviceComputer fileTask (computing)Library (computing)Module (mathematics)Computer iconComputer-generated imageryGastropod shellDrum memoryCross-site scriptingAutomationSource codeCloud computingMassPlug-in (computing)WebsiteMultiplication signArithmetic meanCore dumpInstallation artComputer-assisted translationCompass (drafting)Computer animation
38:13
Type theoryContinuous integrationCASE <Informatik>Scripting languageDependent and independent variablesCartesian coordinate systemGoodness of fitQueue (abstract data type)Self-organizationParameter (computer programming)Stress (mechanics)BuildingMathematicsProgramming languageMultiplication signProcess (computing)Software developerPhysical systemInstallation artWordTwitterFeedbackStructural loadMereologyEmailElectronic mailing listDifferent (Kate Ryan album)Integrated development environmentReading (process)Order (biology)View (database)Analytic continuationData conversionWeightVirtual machineOpen sourceExterior algebraBit rateMusical ensembleRoundness (object)INTEGRALPartial derivativeRight anglePersonal area networkStorage area networkGame theoryArea1 (number)Computer animation
Transcript: English(auto-generated)
00:01
Okay, thank you all for coming to my talk. Are you sure you want to be here? Douglas Crockford speaking. Mark Seaman are speaking. You know, some great speakers out there. Do you want to really be here? So this is a talk about the journey from PowerShell to Grunt for our build and deploy. Okay, my name's Paul Stack.
00:22
I have some contact details on the screen. If anybody takes offense with what I say, or wants to discuss it further, then please do get in contact with me. I'm an infrastructure engineer for OpenTable. After the last session, someone said, what does an infrastructure engineer do?
00:41
I don't know. I seem to be away from the office an awful lot and I have an amazing team who are building very cool tools for our infrastructure based around using Puppet for orchestration of changes to our production systems. I'm a DevOps extremist. I, over the past six, seven years,
01:02
I loved CI. After CI, I thought, there's got to be something else and then I really get into continuous delivery and in order to understand continuous delivery better, I started looking more in towards DevOps and getting into the DevOps world and now I just believe that it's the only way that companies can truly succeed
01:20
in delivering amazing software in the best manner. Okay, I'm a conference junkie. This is my eighth conference. I feel as though, of this year, okay, it's only June, I feel as though I should go to a support group so that I can go, my name's Paul. It's been two weeks since my last conference. But I enjoy going and meeting lots of new people.
01:44
So, this is a description of where we were, where we got to, and where we went and where we're at now and we're gonna be showing lots of code examples so people can see them and there's a ton of our code that's already open sourced so you'll actually be able to see a lot of what we've already written.
02:01
Okay, so in the beginning, when I started at OpenTable just short of three years ago, we had a custom written build tool. Oh, okay, it was a proprietary piece of software that was a little console application that required to be installed onto a build server. Okay, it was written in C Sharp and ASP.NET 2. It's probably not the best thing
02:20
if we look at it now for a build tool. Okay, back then it was really good because it was interacting with our SVN repository back then. Who still uses SVN? It was interacting with our SVN repository and being able to pull tags and it was being able to create tags for us and so on.
02:41
It was extremely versatile because it took care of things like our own internal web config transformation. So, our website, our main website has got nine different variants of it, different languages across the world and this little tool managed to swap
03:00
all the different web config pieces into the right place when it required. So, it was actually very good. It became extremely difficult to maintain. Extremely difficult to maintain. What we did is we had a build server, one build server, okay? And on that build server was this console application. And also on that build server was Visual Studio, okay?
03:24
And it became very difficult to maintain. So, we thought, let's start moving away from that. But this was a central agent for all the builds across our web platform, okay? This was quite a big deal. We had a lot of builds going on.
03:42
And when that server went down, that agent, we were in real trouble, real trouble. So, we had the rise of PowerShell, okay? I'm a huge fan of PowerShell. I always have been a huge fan of PowerShell. If people can understand what I'm actually saying, that's Microsoft Power Shell. It's just my silly accent.
04:00
And over the next couple of months, after I joined the company, my team really, we became responsible for building a new web application, okay? The application was ASP.NET MVC app, and the old tool didn't build that app, okay? That's when we realized that things had to change.
04:21
We needed to create new scripts, okay? And I had just come from a previous company that I had just gone and implemented an entire new build system, actually an MSBuild. I don't know, looking back, I don't know why I'd ever choose MSBuild. I guess, any MSBuild developers? Anybody like MSBuild?
04:41
By need. Who likes the brackets? Everyone loves a bracket, right? So, I started to realize how amazing PowerShell was, and how versatile it was for Windows infrastructure. So, on top of PowerShell, there is a DSL, the main specific language,
05:00
called Saki. It is not pronounced P-sake, as everybody that I meet does, okay? It's a silent P, okay? So, and it's got a very similar style to Rake. You describe things in tasks, okay? But the best thing about it is,
05:21
it avoids the angle bracket tacks of MSBuild. Does anybody know Saki? I know a couple over there do, okay? It's a pretty awesome tool. It was originally written by a guy called James Kovacs, as of version 1.0, and then it was re-released by Georges Matos as of version 2, okay?
05:41
As I said, I was creating things in MSBuild for a long time, and I spent many, many sleepless nights trying to work out why my build scripts didn't work, and it was because of brackets. Brackets and escaping of characters. Anyway, the DSLs are a lot easier to use,
06:00
and they're a little bit more fluid, okay? They just make things a little bit better, okay? So, why Saki and not Rake? I'm a reformed Microsoft fanboy, okay? I used to be a huge fanboy, and it was all about Microsoft tools for me, okay? And because of that, I hated, I detested Ruby, okay?
06:25
I write Ruby on a regular basis now. I'm like changed completely. But PowerShell had to be the way forward. If it wasn't MSBuild, and it had to be Microsoft, it had to be PowerShell. Now, I know somebody in the crowd has written their own, like, build script tool, but sort of.
06:43
But this was going back like three years ago, and I didn't know many other tools existed then, okay? With all the power that PowerShell actually brings us, this is extremely versatile at what it can do for builds and deploys, okay? So PowerShell remoting for your deployments.
07:02
You can start jobs. You can set process weights, et cetera, on your build. So it becomes really good. So the script looked as follows. So, we have a repository.
07:22
And the last time, somebody is actually maintaining this, but originally it was created two years ago, okay? And we did all sorts of cool stuff, but our build script effectively looked like this. Okay, so we passed in some properties. We had some configuration, which was like the type of configuration. So if it's debug or release or so on and so forth.
07:43
Solution directories, solution names, project names, whether it's actually, whether it's gonna be packaged or published or whether this is a part build. The build number. And it just started to grow exponentially, okay? And if you look at the git blame of this file, you'll see that probably five or six different people
08:01
have been touching pieces of it and breaking everything. And we'll discuss why that's the problem later. And it was really easy, okay? So we had a task default, which depends on build unit tests and acceptance tests. Okay, and the build basically said run ms build. So we had a wrappers file, which I'll actually show you the wrappers.
08:25
This is the wrappers is where all the nastiness is. There we go, run ms build. Okay, and again, this has got horrendous over time. And we would switch based on configuration. And we would just, because you're inside PowerShell and you can connect to all the pieces of the system,
08:42
we could just use exec ms build. And we could actually trigger from PowerShell and ms build script. Okay, and then you pass in all the parameters and then we actually use the output parameters to TeamCity to tell us when the build was ready and so on, but we'll come back to that later. And after it finished the build,
09:02
it would then go through a series of steps based on the crap that people added to the scripts. And I say that in the light hardest sense of the word crap. It just so happened that these scripts other people wanted to use. I made the biggest cardinal mistake. I made these a separate repository of their own.
09:23
They were not part of my application. So therefore anybody in the company who wanted to could go and change those scripts. That was my worst mistake. And we'll discuss why that was a bad mistake. And then at the end, if you want to package it, it would go off and package the build, it would do some other DLLs
09:40
and it would check some of the package. We want to make sure that the package that... When we package that up for the first time, we want to record a checksum and then when we deploy, we used to actually check that that checksum was the same because we're a SOX controlled company and you have to prove that what you built and test is exactly what is released.
10:01
I don't know. And then we ran some unit tests. We had another wrapper called run unit tests. Maybe not. They're in there somewhere. And we just basically passed in some configuration values
10:21
of where the end unit exe is because we used to use end unit. And we would pass in the directory that you would run test against. And then we actually just passed in a couple of extra parameters, where we said unit test results.xml. So we want to get the result back so that we can feed that into TeamCity. And then we passed in the type of tests, unit tests.
10:44
Looking back, that can 100% be refactored to the same method, just passing, having a wrapper and pass the types in rather than copying and pasting. But we wanted to separate the fact that there were unit tests and there were acceptance tests and we wanted people to see that. We wanted to see that there was a distinct difference.
11:01
So the build script in itself was extremely simple. And that's what PowerShell gave us. And the SAKI notation gave us that simplicity. Now, when it came to deployment,
11:23
it became even more simple. So we originally just started deploying this in a very small scale. And we would just basically say, get the site directory, get the package directory, the deploy folder, okay? You would just change some, this is just like configuration,
11:41
swapping the correct configuration into place. And then we just copy the folder across the network share onto the server. And then we would remove all the temporary deploy and then we would actually just do an IS reset. So we would swap the system into place.
12:02
As soon as we had swapped into place, we'd done an IS reset and then cleaned up. That is not a good deployment plan. It's really not a good deployment plan. And that was like a V1. And that one was added two years ago. And then we started to get a little bit more involved with deployment. We had pre-deployment.
12:21
We had setting load balancer statuses. We had warmups. We had going off and getting our artifacts. And a pre-deployment was, see, we started to become much more professional with our PowerShell scripts right here. Passing in and help messages so that people could see what they needed to pass in. And it was go off to get your system from Artifactory.
12:45
We use Artifactory for our packages. And when you get, in fact, let me just go to the bottom. So deploy site. Deploy site would basically say, for each server in the list, because we could pass in a common delimited string of servers, for each one in the list,
13:00
go off, get the files, extract the files to the server. Does that make sense? Really simple. We're not doing anything bad, okay? And after deploy site, we would then call the deploy configuration, which would go off and it would swap in the correct configuration based on if it was production and America,
13:21
or production in Japan. It would go and get the correct configuration values and put those in place as well. So that was our pre-deployment. And then we would take the server off the load balancer. So we would go off and we would interact with SQL Server to take our server completely out of the load balancer. Now when it went off the load balancer
13:41
and we got a good exit code, a zero exit code that it's worked, we would then do the final step, which would be the actual deployment in itself. And all we did was call final deploy, PS1, which effectively swapped the server, swapped the folder systems around. We would take the old one,
14:01
we would rename it to be old or backup or temp, and we would take the new one and rename it to be the actual folder itself. And we would do an IS reset on that. Again, that is not a good deployment plan. Not a good deployment plan. And so this is going back like two years ago. And we were very much as a company
14:22
at the beginning of our continuous integration and continuous delivery start. And it's something that takes an awful long time to implement because each environment has got its own different way of doing things. And each environment has its own stakeholders
14:40
that you have to consult in order to build that pipeline. And this was the first phase of it. So we actually moved from a company that did manual deploys of these packages that this old C sharp application created across and started now having a real sort of a pipeline. And I call it a sort of a pipeline.
15:01
Anybody any questions on those scripts? Does anybody want those scripts? You want them. We were 100% Microsoft. So that is exactly why there was no need for anything else.
15:20
It was MSBuild and PowerShell or nothing. Okay, and that was okay. That was okay. We hired ASP.NET developers. Our developers all had core C sharp knowledge. And we didn't have anybody inside the company who was really pushing the bonds of other languages anyway. But yes, it wasn't cross platform in any way, shape or form.
15:41
If you do want these scripts, let me know. You can have them. I won't even charge you for them. And so we could do all other sorts of things in here. So if I show you some of our warm up scripts are actually sending requests across,
16:01
making sure that a get request returns a 200 and that the application warms itself up as part of the deployment. So our pipeline actually became quite good and it looked something similar to, it would just run a bit, sorry, that's the wrong one.
16:29
So we would pre deploy the site and then we would deploy the site. Okay, and based on parameters that we passed in, we could then say and also warm the site up. Okay, run that part of the script as well.
16:42
But very quickly, because of so many people, there we go, because of so many people touching this, let's go back to the actual, so the script look okay. Fundamentally, there's nothing wrong with them.
17:01
It's probably not the best way of doing it. But people started, as I said, people started to adapt them. So one person would go, but my app needs to do this. So let's add a little if statement in there. And if it's my app, then I'll do it this way. But my app has this build configuration parameter.
17:20
So let's add that into the case statement. All solid principles are going out the window right there. And very quickly, we get into a massive spaghetti mess, huge spaghetti mess. And we ended up with a build in our system that takes 17 build parameters.
17:44
They'd have all sorts of rubbish going in there. We'd have deploy directories, we'd have branches, because people were using branches for their things. You'd have project names, you'd have some package location, some artifactory. You'd have some hard coded IPs. You'd have like, whether it's a service or a web app,
18:02
it just became an absolute maintenance nightmare. But rewrites completely suck. Anybody been involved in a software rewrite? Has it been fun?
18:20
Have you wanted to pull your teeth out? Yeah, they're not fun. They're really, really not fun. I'd already written one set of build scripts. Why was I going to put us through writing another set of build scripts? Do we really want another? Then we had like a bit of a revolution in the company. We were no longer a Microsoft house.
18:40
We were actually able to start writing applications in other languages. And in the summer of 2013, we had an engineer joined us called Andy Royal. And luckily I managed to get him on loan to my team for the first couple of months. And we had just started to write a Node.js app. Somebody told me I could write a new application. I was going to choose a new language. I was like unleashed.
19:02
And Andy wanted to learn Grump. So Andy then decided that, why don't we start writing the build system in Grump? Okay, because it's a new shiny thing, right? And this was like a year ago. None of the other ones had really come out yet. And we'll look at some of the other ones a bit later. But for anybody who doesn't know, Grump is a JavaScript based task runner, okay?
19:23
So you register tasks and then you can call those tasks in place, okay? It's got a massive community ecosystem. When I rewrote the talk this morning, there were 2,954 Grump plugins available, okay?
19:41
There's new ones appearing every day. Now you go and search for a screenshot or share or something like that. You'll probably get 20 variations of different packages, but there's a huge community present. But of course, Grump works really well with node. Okay, so you do npm install Grump
20:00
and you'll actually be able to start interacting with your node application from it. So what do the scripts look like? Right, so everything has a Grump file, okay?
20:27
And we skip that top part. That's not really relevant, okay? So we go off and we go to a directory and we can say, for every JavaScript file in that directory, register that function,
20:43
register that file itself that you find as a function, okay, as a task that's available. At that point, you can start to, you don't have to register everything explicitly. You can let recursion happen and let it register all its own tasks, okay?
21:02
And then we can say, but register a task of default. So if somebody just is in my directory and types Grump, it'll do the build, okay? But if somebody runs Grump build, it'll do a clean build, a JS hint, it'll run my mocha unit test and it'll copy the build across for packaging, okay? If somebody just wants to run mocha,
21:21
it'll actually, now what our mocha tests are actually doing here is it's using an in-memory database. So it's actually filling the database with known data and running acceptance tests against that data, okay? And it will clear the database down and refill it, okay? So if you want to retest your code, you want the process to start again.
21:42
And then the last one is, if you're just in development mode, you can just run JS hint to mocha, okay? Now let's have a look. Everybody read the screen? Awesome. So Grunt, simple Grunt, and it's gone off and it's done everything, okay?
22:01
And you can see. What it's done is it's JS hinted everything, okay? And it's filled my database and it's ran all my acceptance tests against the database. This, we have a read me that basically says you need to do a brew install of Mongo, have that running and then this interacts with Mongo in the background.
22:21
And at the end it comes back and says 100 tests have passed in 121 milliseconds, okay? And we've copied the build. Okay, now if I do Grunt dev, we just do a JS hint and we clear the database. I can't connect to it at the minute. And so at that point, we can run different pieces of the system, okay?
22:42
So then Andy wrote a blog post after, so his blog post was like Grunt your deployments too, okay? So, and we were like, why do we get things built by Grunt and then deployed by another system? Why can't we just get Grunt to deploy our system for us?
23:01
And he registered a task called deploy and deploy called a list of other tasks, okay? So it's gone off and got the artifacts. It took the app offline. It stops the application. It makes the release directory. It updates all the new symlinks for the new deployment. We do an SFTP deploy. We do an NPM update.
23:20
We set some configuration. We start, we wait for the application to start. We run our warm up scripts. We wait, we verify, we put it back online and we have cleanup afterwards, okay? This is a much better pipeline straight away. And it's much more explicit about exactly the tasks that it does. We're not hiding this away in team city.
23:41
This is not a slating of build tools by any way. They can just be misused in a very big way. But we have the same task locally that we can run against a test database or a test acceptance site that team city would run in our CI environment or our pre-prod environment or our production environment.
24:02
And how it looks in team city is this. So for build, we have a team city step that says grunt. The grunt task is build. We're not hiding anything away. It's as simple as it possibly gets.
24:22
There's no extra configuration parameters. What we like to use our CI tool as is just somewhere to orchestrate a job. Everything else must be encapsulated, okay? And we can go and we can have a look at the deploy step. Oh, and there we go.
24:49
The task is just deploy API. Again, all the logic is held inside the grunt file. So there was no hidden surprises when a developer decided to run a command locally
25:02
that didn't happen in production. So if we have a look at some of the contents of those, so we have an artifactory task, we have a clean,
25:22
we have a clear DB, we have copy, we have get artifacts, we have HTTP. So in fact, if we look at HTTP, okay, it's just going off and just running service status pages. So we have a page on our internal systems that we can hit, that's like service status
25:42
just to make sure that the application is running our internal applications only. Don't go and try and DDoS our website. I'll note you. And we just make sure that we can actually warm it up and we collect any errors if it returns an incorrect response code. It doesn't do anything too bad at all.
26:00
And if we have a look at something like JSHint, we're basically just running a simple JSHint across all JavaScript files in the solution, okay? So developers who are good at code were actually starting to use code in their build system and in their scripting system.
26:22
And all, you know, these are not complex scripts. Those seven lines right there, that's all available, open source. You Google how did JSHint with Grunt and that's almost the first result you'll get. No, don't try it. Maybe it's not the first result. But the point here is that why reinvent the wheel?
26:45
We had a second chance at creating a good build system, a good deploy system and we could use that. We could use that second chance in order to make it better. And we could be a lot more interested in things. So then Andy was like, okay,
27:01
so we do our builds and we do our deploys, how can we run our acceptance tests? We don't want every day, every developer to have a copy of these five systems on their machine, okay? We don't want to be running our tests against Mac OS X when we deploy into Ubuntu in production. So what do we do? So he hooked into a tool called Vagrant, okay?
27:22
And we have another task called Grunt Acceptance. And what Grunt Acceptance does is at this point is it does a Vagrant up, actually spins up a physical machine using Vagrant, sorry, a VM on my physical machine using Vagrant,
27:43
it will run some tests against that VM and it'll destroy it. So if I'm in the directory, I can say Grunt Acceptance, and it's running the Vagrant up task. And it goes off and it clones a VMware and Ubuntu VMware box from another repository
28:03
inside my machine. It will install Mongo, it will install Nginx, it will install NPM and node and all these bits and pieces and it will run all my tests against them. Now it takes like 10 minutes to run. But before a developer checks in some code that they take 10 minutes just to go Grunt Acceptance, this works against the known operating system.
28:23
That was huge, absolutely massive. And he wrote a blog post about this called Grunt Your Acceptance 2, Grunt Your Acceptance Test 2. So it's, what we're trying to show is that we're not doing anything, we took this opportunity when we had a new app
28:42
and we were like, right, what does our app need to do? And we continually added new pieces to the pipeline when we needed them. But the main thing, I'm just gonna kill that because I don't want that spinning up. The main thing is that the build files themselves
29:03
were packaged with the app. So when a developer checked out the code from GitHub, they had not only the actual application code itself, but they also had the build code. It meant that everything could be done on the same checkout. Developer could test everything locally. They had the Vagrant file that they could test everything with.
29:20
So they used Redis now in there as well. So we're really starting to move forward. Any questions? Awesome. Anybody just want me to shut up so we can go for lunch? Yeah, okay. We're almost there, I promise.
29:41
Now this was for a Node.js app. So can the scripts be reused? Yes and no. Okay. I said before that we didn't want these to be, we fell into the trap before of having central scripts. We did not want to fall into that same trap. So if another team who wanted to build and package things
30:03
with a Node.js app in the same way wanted to use the scripts, they would have to go and clone our repository and they would have to see what we're doing with it. Now that's a good thing and a bad thing because sometimes it creates silos and sometimes it creates a non-similar way of doing things at all times.
30:20
But in an organization of 150 engineers, you're never all gonna be doing things in the same way. Especially if you have lots of languages and you're doing cross-platform development work. But there were pieces of the puzzle that could be reused. So things like pushing the artifactory, clearing down a database,
30:41
getting things from artifactory, HTTP warm-up scripts. All of these are actually reusable scripts. And if we really wanted to, we could create an NPM repository inside our company and then the engineers could go NPM install, open table HTTP, and that would bring those down. But we're not at that point.
31:01
We're not at that point. In fact, what I can show you now is, speaking about cross-platform, this promoted offer application is actually a C-sharp application. And in here, we have a grunt file right there.
31:24
And we have a build folder right there. And if we go into the build folder, we can see that we've got some other things in here. We've got N unit. Damn it.
31:41
Thank you. Okay, so we have N unit. We have SQL. We have other pieces of the puzzle in there. Now, as I said, this is a C-sharp app. And if we go inside our grunt tasks, we can say here that we have build SQL. So this goes off and this will build
32:01
a change set from SQL server, and it will store that with the artifact, the build artifact, okay? And then we have SQL command dot JS. We have package GitHub data, hip chat notifier, get last pinned build number from team city. This is another app that's gone through a transformation.
32:23
So new pieces of the pipeline were put in place because it was a more complex app to build. It wasn't just a C-sharp app, okay? And there's another one, feature dashboard, which is an NVC app that builds a mono.
32:41
So we were like, if it builds a mono, I can build up my local machine using Mac. So let's do that. And then we have grunt tasks for that. So we've one at the bottom called X build because we want to execute X build itself rather than MS build. So all I have to do as a developer,
33:02
if I just do a grunt build, it cleans the build first, it realizes it's actually doing a C-sharp build. So we're in a situation now where our packaging and our builds are in the same state locally as they are in our environment.
33:20
And the same for deploys. If I really wanted to, right here, I could run a grunt deploy and pass in some environment data and it would go off and it would connect to those environments and do the deploy for me. It would just take a little longer because I'm going across the VPN to San Francisco. But we can run any of these scripts at all times. And this was a very key thing for us.
33:49
So are there alternatives? Okay, it's really good to see some people here from script CS. Cause I actually put script CS in there and I told, let's see, thank you.
34:00
I told Justin that I would put it in there. It is actually an alternative. He's going to take a picture of it cause I troll these guys all the time. They're awesome about being trolled. Which is really funny. But script CS is actually there. Okay, you can script stuff on there. I'm sorry Adam. I should have put your tool on here as well. So you've got one called bow. Is it bow you pronounce it?
34:20
The new hotness right now is Gulp. Anybody using Gulp? Anybody know what Gulp is? It's another JavaScript. They claim to be a streaming JavaScript framework, build framework. But there's always alternatives out there. This is not saying you must go and use Grunt.
34:40
Grunt may not fit what you're doing. Okay, it may not. There's Gradle. If you're in the Java world you can go and rewrite stuff in Gradle. You can continue to use PowerShell. Just be very good at what you do with it, okay. Or you can use Make. Does anybody use Make still? So Hadi Hariri wrote a blog post
35:02
on we're basically cluttering the ecosystem with systems that are written as alternatives to Make. And why don't we just stick with Make itself? A lot of people are starting to begin to feel like that again. Scripting frameworks are the new JavaScript frameworks.
35:21
If you really want to, you can go and create one in a very simple amount of time. It would be very immature and it probably wouldn't do an awful lot but you can do it. You can write one from scratch. Why reinvent the wheel though, right? If somebody's already written quite a successful one there's no point in rewriting the wheel. There's really not. So do I think that PowerShell is bad for scripting?
35:44
100% not. We've come from a company where one or two people were responsible for the build scripts. It was like me and one other person cared. When that was the case and everyone was doing things in exactly the same way, building the same style applications in the same language,
36:02
it worked fantastically. As soon as you moved away from central manage, as soon as teams were given autonomy and teams were able to go off and do bits and pieces in their own fashion then there was variations very quickly appeared into the system. What we had is a per design choice.
36:23
Centrally managed build scripts are not a good thing. Does anybody have centrally managed build scripts? They're tough, they're really tough and they can break an awful lot. We also don't write applications solely in ASP.NET MVC or in ASP.NET anymore.
36:42
So PowerShell for node apps, you can do it, but just because you can do it it doesn't mean we should, right? The future for us is we just don't know. We don't know what type of apps we're gonna be building next. Maybe there's a new app that we have to build next week
37:00
and we'll use Gulp for it. Maybe there's a new C sharp app and we'll use script CS for it, that would it. I'm just joking. But the point here is that there's so much resources available out there. We can go and we can have a look
37:20
at the Grunt plugins website. Everybody see that? So you've got Contrib, you've got Watch, UglaFly, Concat, CSS Min, Less, Karma, Coffee, Compass. There's just no, they're all out there.
37:43
And like some of these have been downloaded a huge amount of times, 400,000 times nearly for a JS hint. Anything with Contrib in front means it's a core package maintained by the Grunt team. If it doesn't have Contrib in front then it's actually outside of Grunt themselves.
38:04
And there's, this comes with its own problems, right? Because if I go and I, in fact let's go here. If I say npm install,
38:23
it goes off and it downloads the word. Pseudo npm install. It goes off and it downloads the word. So if you depend on those external packages
38:40
people do make breaking changes to them. So be wary if you do go and use an open source tool like this in order to build stuff. There are changes that happen all the time. But sometimes that's okay. Just be wary, be yardful about that's the case. Okay, let's just cancel that.
39:08
So I have like probably 20 minutes left, okay? We're definitely not gonna take 20 minutes. We're gonna take like five and then I'll take a couple of minutes for questions. And we can run and get in the lunch queue first.
39:22
And I have some tips. The tips here are think about, be mindful about what might happen in the future, okay? So you may change completely what you're doing. So don't throw all your resource at creating these amazing build scripts, okay?
39:41
Now have good build scripts and reliable build scripts but evolve them over time. As your application grows and your company needs change and your team size grows, evolve your build scripts to evolve with the application itself. 100% do not centrally manage your build scripts.
40:01
I can't stress this enough. I've been bitten by this where I've had people calling me at like midnight saying, the builds don't work. I'm like, well, what do you mean it doesn't work? Well, it doesn't build. What did you do to it? Oh, well, I just added this new parameter offer, you know. And we would just have this never ending list of problems.
40:21
I would get in in the morning and I'll change something before I go home. And the San Francisco team are then blocked because I've broken their part of the build scripts. I didn't know what they were using it for. So centrally managed is tough. It's really tough. Get lots of feedback. These build scripts, is anybody not using a continuous integration tool?
40:45
Who uses TeamCity, Jenkins, Bamboo? Okay, cool. So if you're not using a continuous integration tool, get a continuous integration tool, even TFS. Even TFS.
41:01
Just listen, if you're using a tool, that's a good thing. We'll talk about the tool in another time, but that's something different. Use TeamCity, there you go. That's what you said. But just make it simple, okay? Make your script simple that the way that it executes inside your continuous integration tool happens on your developer machine
41:21
and Jeff's developer machine, okay? They have to be consistent, okay? And the last one is have empathy for your other developers. Don't just go and make lots of changes and not tell anyone because that makes you look like a douchebag. And when you have to get phoned
41:41
because you've changed their build scripts, you're the one that has to suffer. You're the one that has to apologise to the team for stopping their application deployment. Does anybody have any questions? You must have questions. You're a CI and CD guy. How do you feel about this? Of sole packaged with your application
42:04
or centrally managed build scripts? Okay, okay. And the more you get into continuous integration and continuous delivery work, the more that you'll see that having a good build system is extremely important. You must be able to rely on it. You really must.
42:22
Any other questions? Yeah. Saki.
42:54
100%. 100%. And that's why I said, depending on the type of organisation you're in or what type of application you're in, there are all alternatives.
43:01
This is not a you must use grunt. Grunt is the best. If you need to and you're a 100% Microsoft shop but you don't want to use PowerShell, then use script. It's good for that, right? Is it? Is it good? Okay, well, there you go. So there you go. Bow is even better, right? This is more of a choose what tool you're comfortable with
43:24
and choose what tool is simple, okay? But it's very effective in what's getting the job done. If I was still building C-sharp and .NET apps, I would 100% still be using PowerShell because I find it phenomenal.
43:41
Every time they make a new release of PowerShell, it adds so much more to the language. But I don't understand the C-sharp building, C-sharp thing, but hey, I'm a Ruby developer.
44:02
I don't understand these things. But go off and try some things. Go off and try some tools. Go and get, if you're a .NET developer, go and try Gradle. There's like 26 chapters of documentation on Gradle. That'll be a fun reading before you go to bed one evening. But there's lots of tools out there.
44:21
Choose a tool that, get advice. Twitter's a great place of, I have this application to be built. What technology should I build it in? You'll probably get 100 responses saying, gulp, gulp. And then you'll get a load of responses from the C-sharp guys going, scrape their bio or PowerShell. But speak to people, get some insight into how they've evolved
44:43
their build script and how they've evolved their environment going forward. If anybody wants to send me any rants or tells me I'm wrong, please do. I haven't put my email on there because I'm awful at email. I have a backlog of emails to reply to. So tweet me because I am addicted to Twitter.
45:00
I'm actually always on Twitter. So challenge me. Tell me why I shouldn't be building something in Grunt. And I'll happily, happily have a conversation with people about it. Continuous integration, continuous delivery is something I'm very passionate about. And being able to communicate with other people on their experiences always helps me as well.
45:21
There's always other people's views on this. I'm not the right, I'm not always right on this. You quoted. But this is true, right? We all have our own ways of doing things. So right, let's go and get some lunch. Thanks guys.