Software Testing for Failed Projects
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 96 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/51845 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
NDC Oslo 201675 / 96
2
7
8
9
12
14
19
20
26
28
31
33
38
40
43
45
48
50
51
61
63
65
76
79
80
83
87
88
90
93
94
96
00:00
SoftwareProjective planeSoftware testingDifferent (Kate Ryan album)Event horizonUMLComputer animation
00:51
Software testingSoftwareMultiplication signSingle-precision floating-point formatProjective planeRule of inferenceRight angleSoftware testingData miningUniverse (mathematics)WritingShared memoryInternetworkingTwitterComputer animationXML
02:12
Physical systemProcess (computing)TwitterProjective planeProduct (business)System callRight angleCustomer relationship managementQuicksortSoftwareSoftware developerServer (computing)Hydraulic jumpGroup actionCoefficient of determinationComputer animation
03:32
Software developerForcing (mathematics)Inheritance (object-oriented programming)Projective planeComputerRule of inferenceCustomer relationship managementInternet forumPhysical lawComputer animation
04:18
CodeSoftware developerCodeSoftware testingProduct (business)Order (biology)Unit testingInternet forumLevel (video gaming)Forcing (mathematics)Instance (computer science)FreewareGradientFilm editingSystem callComputer animation
05:08
OvalFreewareSoftware testingProjective planeSpreadsheetRoboticsProduct (business)Message passingCodeSoftware testingFunctional (mathematics)Single-precision floating-point formatMultiplication signMobile appExecution unitMaxima and minimaTape driveRight angleDenial-of-service attackOpen sourceComputer animation
06:53
CodeGoodness of fitComputing platformVideoconferencingCryptographyExecution unitMathematical analysisVideoconferencingSoftware testingBlogCondition numberGastropod shellProjective planeCoefficient of determination1 (number)BitUnit testingMobile appMetric systemCodeMathematical analysisCASE <Informatik>Physical systemReal numberLine (geometry)Single-precision floating-point formatComputing platformTraffic reportingExecution unitWave packetMultiplication signDifferent (Kate Ryan album)SoftwareForcing (mathematics)NumberState of matterArithmetic meanCore dumpXMLUMLComputer animation
10:20
SpreadsheetMobile appMultiplication signSoftware bugSoftware developerProjective planeLine (geometry)RotationHypermediaComputer animation
11:05
Computing platformVideoconferencingCryptographyCodeExecution unitProjective planeSoftware bugElectronic mailing listIterationMultiplication signDisk read-and-write headExecution unitSoftware developerDecision theoryCore dumpVideoconferencingDiagramJSONUML
11:44
CodeSoftware testingMetric systemNumberCodeMobile appPhysical systemTraffic reportingLine (geometry)Arithmetic meanSpectrum (functional analysis)Level (video gaming)Metric systemScaling (geometry)Projective planeDiagram
13:01
Software testingExecution unitVideoconferencingData conversionLine (geometry)Software developerSeries (mathematics)SoftwareMereologyBlogComputerInternetworkingPoint (geometry)CodeWhiteboardService (economics)Rational numberPhysical systemSoftware testingSystem callPattern languageMessage passingMultiplication signLevel (video gaming)Open sourceOrder (biology)DatabaseMobile appPerspective (visual)Projective planeRevision controlObject (grammar)View (database)Component-based software engineeringReal numberMathematicsAnalogyExpected valueTwitterState observerDifferent (Kate Ryan album)DivisorVideo gameCartesian coordinate systemSoftware bugShape (magazine)Perfect groupYouTubeTest-driven developmentUnit testingComputer animationMeeting/Interview
18:16
Computing platformSoftware developerComputing platformPoint cloudProjective planeCollaborationismPlastikkarteWordGroup actionMereologyStaff (military)Service (economics)Disk read-and-write headInheritance (object-oriented programming)Lattice (order)XML
19:36
Software developerSoftware testingVapor barrierSoftware developerProjective planeProduct (business)Event horizonRight anglePhysical systemPoint (geometry)Shared memoryCondition numberBuildingExistenceSource codeComputer animation
21:28
Module (mathematics)Software testingDesign by contractExecution unitPhysical systemService (economics)DisintegrationIntelComputing platformCollaborationismGreatest elementDifferent (Kate Ryan album)CollaborationismBuildingSoftware testingMultiplication signProjective planeINTEGRALPhysical systemWebsiteInheritance (object-oriented programming)Execution unitField (computer science)Unit testingPerspective (visual)Set (mathematics)Object (grammar)Pattern languageTest-driven developmentLevel (video gaming)TheoryWordResultantProduct (business)Service (economics)Computing platformData conversionModule (mathematics)Computer animation
23:36
Execution unitSoftware testingPhysical systemDisintegrationInterface (computing)Functional (mathematics)SoftwareReflection (mathematics)MereologySoftware testingPhysical systemCausalityTerm (mathematics)Projective planeType theoryGreatest elementMobile appWordUnit testingCASE <Informatik>Interface (computing)Level (video gaming)Online helpDatabaseComputing platformComplex analysisFinite-state machineCollaborationismDifferent (Kate Ryan album)Condition numberService (economics)Complex (psychology)INTEGRALState of matterAddress spaceForm (programming)PlastikkarteRight angleBuildingProduct (business)FrustrationWeb pageExpected valueQuicksortComputer animation
26:41
Execution unitSoftware testingFunctional (mathematics)DisintegrationPhysical systemInterface (computing)Multiplication signExecution unitReflection (mathematics)Projective planeUnit testingSoftware testingDataflowScaling (geometry)Right angleComputer animation
27:21
Interface (computing)Functional (mathematics)Scale (map)Software testingDisintegrationThermal expansionGUI widgetProduct (business)Computing platformPhysical systemExecution unitVirtual machineLetterpress printingComputing platformSoftware testingProduct (business)GUI widgetProcess (computing)Projective planeDisk read-and-write head1 (number)Data conversionPairwise comparisonVirtual machineInterface (computing)Variety (linguistics)Physical systemUnit testingWeb-DesignerSoftware developerInheritance (object-oriented programming)INTEGRALProfil (magazine)Scaling (geometry)Order (biology)Tape driveDifferent (Kate Ryan album)Mechanism designMultiplication signExtension (kinesiology)CollaborationismMobile appRight angleAddress spaceGoodness of fitCASE <Informatik>Knowledge-based configurationConfiguration spaceRevision controlIntegrated development environmentDynamical systemGroup actionTerm (mathematics)Content (media)Graph coloringPoint (geometry)Execution unitComputerTelecommunicationStatement (computer science)Cartesian coordinate systemBitComputer programDigital rights managementNatural numberDatabaseSequelSystem callEndliche ModelltheorieDressing (medical)PRINCE2OracleVideo gameComputer animation
34:10
CollaborationismRight angleProjective planeEndliche ModelltheorieOracleCycle (graph theory)Physical systemInformationArithmetic meanCollaborationismProduct (business)Multiplication signSoftware testingDrawingJSON
35:21
FrequencyConfidence intervalPhysical systemData structureComputerComputer chessWeb 2.0Physical systemConfidence intervalTrailSoftwareSoftware bugScaling (geometry)Process (computing)Product (business)Web applicationError messageDifferent (Kate Ryan album)
36:24
Interface (computing)Functional (mathematics)Software testingScale (map)PlastikkarteBitOnline service providerEmailInternetworkingProduct (business)Error messageTerm (mathematics)Software testingNumberInterface (computing)Interactive televisionINTEGRALDatabase transactionComplex (psychology)CoprocessorComputing platformBuilding
38:06
Software testingPhysical systemInterface (computing)Functional (mathematics)Data structureBeta functionService (economics)Execution unitData modelDisintegrationBlogComputer architectureMultiplication signService (economics)State of matterEmailProduct (business)Asynchronous Transfer ModeINTEGRALComputer hardwareSoftware testingUnit testingSubsetPlastikkarteCodeMagnetic stripe cardLevel (video gaming)WritingSoftwareInheritance (object-oriented programming)Function (mathematics)outputPlanningData structureGreatest elementBuildingConfidence intervalComplex (psychology)MathematicsExecution unitIntegrated development environmentSign (mathematics)Error messageForcing (mathematics)Endliche ModelltheoriePoint (geometry)Basis <Mathematik>Finite-state machinePhase transitionDesign by contractMeasurementCAN busSystem callAnalytic setCodeSuite (music)Expandierender GraphInternet service providerStress (mechanics)Address spaceReal numberFeedbackComputer animation
43:48
Hill differential equationSoftware bugGodSoftware testingControl flowPatch (Unix)Web pageSoftware developerMultiplication signService (economics)CodeSource codeComputer animation
45:05
SoftwareSoftware testingMetric systemControl flowVideoconferencingComputing platformCollaborationismGUI widgetError messageVideo trackingSubstitute goodTrailProjective plane1 (number)Data conversionDatabaseMobile appBuildingProfil (magazine)Substitute goodSoftware testingMultiplication signSoftware developerUnit testingPhysical systemCustomer relationship managementMetric systemGoodness of fitCollaborationismEndliche ModelltheorieForm (programming)ComputerSingle-precision floating-point formatMereologyDifferent (Kate Ryan album)NumberArithmetic meanSelf-organizationControl flowCartesian coordinate systemFamilyException handlingComputer animation
47:37
Scripting languageJava appletError messageProjective planeEmailSoftware testingVotingUMLComputer animation
Transcript: English(auto-generated)
00:04
Awesome. Started a minute early. That's fine. It's the last one of the day. Last one of the conference. How's everybody doing? A little tired, maybe? It's been a long event. It's a great one. This is my third NDC Oslo, and it's amazing.
00:22
So this talk is software testing for failed projects. And it's one of the favorite talks I've ever put together, because I am ranting, basically, and I hope that's OK. This talk is where I'm talking about a bunch of different projects that I've been on, and how each of them approach testing differently,
00:41
and how a lot of them failed because of how they approach testing, and what I learned from it along the way. And so what I was trying to answer when I was putting together the idea for this talk is, are we testing the right things? When we go into the project, the things that we're spending our time and our effort writing tests
01:01
or evaluating, are those the right things that we should be focusing on? Is that what matters? Is that valuable activity for us to do? And as I was reflecting over the other projects that I'd been on and talking with friends of mine and the projects that they'd been on, the only thing I could really come up with is that I have no idea with whether we're
01:21
testing the right things. There's no one rule that I could think of that uniformly applied to every single situation. Somebody would say, oh, well, you should always write tests like this. But we could always find an example when that just wasn't appropriate. It was a waste of time.
01:40
And so what I want to talk about instead, because I can't prescribe a universal rule for you, is instead I want to talk about five projects and what each of them taught me about software testing and how I evolved from that. And I hope that's OK, that I'm just going to share some personal stories. And if that's not OK, I promise I won't be offended
02:03
if you want to get up and go see Mark Rundell instead, because I'm sure his talk is going to be really, really funny. So these are five projects and what they taught me. And so my name is Todd H. Gardner. I'm Todd H. Gardner on Twitter. You can feel free to follow me and heckle me or tell me I'm wrong there.
02:21
I won't mind at all. I tend to look like that on the internet, usually. So let's jump right in and start talking about different projects. Now, I don't actually want to tell you the names of the companies and products that these were. And so instead I've obfuscated all of the names to protect the guilty, including myself.
02:43
And so this first project I call Big Todd's Awesome CRM. And so who's worked on a CRM project before? Yeah, they're all kind of the same. So we're going to deliver a customer relationship management tool, dealing with sales calls and remembering people's
03:03
birthdays and all those sort of things that you need to do to have a sales cycle. And we're going to replace a bunch of ad hoc processes that this company had with all their internal sales teams. So this happens to be one of the first projects that I was involved in.
03:21
I used to work actually on the infrastructure side of IT, dealing with servers and schlepping network equipment around and running cables and ceilings and all that kind of stuff. And so when I moved over to the software development side, this was one of the first projects I was on. And we decided to build this CRM using the Salesforce technologies. Who's used salesforce.com?
03:43
You are some very lucky, lucky people. It is not a super enjoyable development experience. But Salesforce has some interesting rules about how you write software. And they're very well intentioned. I had absolutely no idea what I was doing.
04:02
I had never run Salesforce before. As I said, this is one of the first software development projects that I'd been on after I switched from infrastructure. So I'm just kind of making it up as I go along and reading a lot of their docs and their forums and trying to figure it out. And so inside of Salesforce, they
04:20
had some opinions about testing and opinions on how much you should test. Specifically, they wanted you to unit test all of your code. And in order to deploy to their production instance, you had to have a 70% level of code coverage. Like the act of hitting deploy would run all of your tests.
04:40
And unless it's hit 70%, deploy failed, can't go to production today, nope, sorry. It's great. I mean, their intention was developers should test their code, which I totally agree with that intention. However, the Salesforce ecosystem tended to be a lot of analysts who learned just enough code
05:01
to get by. And so what most of the forums and the official documentation from Salesforce had things like this that I call assertion-free testing. And so it executed all the code. We actually hit 100% code coverage
05:23
called every single function here. But we didn't actually learn anything from it, right? Like, there's no assertion. I don't know what any of the methods do. It was awesome. But Salesforce thought they were fine. We hit the code coverage, and we could deploy to production.
05:41
And so we had this app, and it was a bunch of noobs. None of us really knew what we were doing. And that was the style of testing that we approached. And when we deployed to production, there were some interesting things that happened, like that. That GIF, by the way, is from an awesome movie from the 80s called Robot Jocks.
06:02
So the next time you're interested in a really bad movie, you should go to wherever you find old VHS tapes or DVDs now and see if you can find Robot Jocks. It's about all of the countries of the world decide to build giant mechs to settle all of their wars. And so if you want to invade Alaska, you build a giant mech, and you try and fight the United States
06:22
mech. It was amazing. It was the best movie ever. No, it wasn't. It was terrible. But anyway, Salesforce project canceled. You know what? We spent a year trying to build this thing, and we were tripping over ourselves constantly. None of our tests would pass. Well, I mean, they all passed. None of them did anything.
06:41
And so we couldn't actually catch anything before it landed in production. And our customer, the sales units, ended up canceling the project out from under us and saying, you know what? We'd rather stick with our spreadsheets. At least that works. So that project got canceled. And we all moved on, and nothing ever shipped. And so what did I learn from that?
07:03
So I came away from that project, and that was one of my first ones. And I'm trying to understand more about testing coming off of that, because it was pretty obvious that that was our failure condition. We didn't know how things were working. And so I went out, and I read a bunch of blogs and watched a bunch of videos about the gurus and the pundits of testing.
07:22
And there's a lot of dogma out there about it. There's a lot of people who talk about, like, everybody should do this. Thou shalt test this way, and you should do it this way. There's a lot of people who are very prescriptive about testing. And frankly, I got this kind of intention from most of the blogs,
07:41
is that my code was better the more I tested it. There's a lot of stuff, especially I saw a lot in the Ruby community, maybe six years ago-ish, that was basically like the higher your ratio of lines of test code to lines of real code, the better your app was.
08:02
It had nothing to do with the actual app. It was just about how much time and effort you'd spent testing it. And I mean, looking back, I totally bought into this, and it was totally wrong. But that was the intention I had coming out of this. And a lot of people have said this, and I said it too.
08:21
And I kind of regret it now. But that was the intention I came coming off of that project, is we have to test everything. We have to test this thing. We can't ship any code that doesn't have any tests. It's not allowed. And so now I move on to another project. And so this is BigCorp's Video Academy.
08:42
And BigCorp was trying to do something kind of different. BigCorp was building a video training platform for cryptozoologists. It wasn't. I mean, it was building something way more boring than that, but I can't really tell you what it was, because then it might actually reveal who they were. So let's pretend they were cryptozoologies.
09:03
And so BigCorp had this idea that they were going to do unit code test coverage, unit test code coverage. And just like the Salesforce project had mandated 70%, they wanted to do better. They wanted to hit 100%.
09:21
We were going to write a unit test for every single line of code all the time. Every single piece of this piece of software was going to be covered. So we started writing this. If you've ever tried to hit these arbitrarily high code coverage numbers, you know that you can't entirely do it without a lot of mocks, especially when you get down
09:42
into the niggly bits that talk to other systems or the weird edge cases around your app. You have to end up mocking a bunch of weird things just to test these lines of code to hit that metric. And so we had a lot of them. We also had a bunch of code analysis metrics
10:00
that were brought in. And we had to hit these arbitrary guidelines on all of them. And it was measuring how well our code performed. And so on paper, we had a lot of managerial reports that said, we're doing better. Every week, we were doing better. Every week, the code was getting cleaner. Everything was getting better. Things were getting easier to understand. Test coverage was staying at 100%. We were good.
10:22
And yet the project was fraught with bugs. Even though we had this metric, this was a team of 12. Two of those 12 were what they called SWAT, which meant full time they were on bug duty. And there was a giant spreadsheet
10:41
that the business would send to us. It's like, here's the SWAT list. We've already prioritized it. It was never empty. It was always like hundreds and hundreds of lines long. So two people on rotation, they couldn't give it to one person because that person would quit immediately. So they had to rotate your SWAT duty through the team. But it was two full time developers
11:02
fixing bugs on this app that everything had to be 100%. How do we have so many bugs if we have 100% coverage on unit tests? And so the project was drowning in it. We couldn't get ahead. Every time we'd add something, the bug list would get deeper. We'd sometimes take whole iterations off
11:22
just to get rid of things on the bug list. And we were going nowhere. So as far as I know, BigQuery Video Academy is still in development and still slowly drowning. I made the decision to leave that project after a couple of duties on bug duty
11:42
because nobody was really interested in fixing the core issues. But I thought a lot about that code coverage number learning or leaving that project. And code coverage is an interesting idea. So we have this scale. Sometimes I'll talk to people and they'll talk about what their code coverage level is.
12:01
And I think that's kind of weird because if you say a number in this code coverage spectrum, what does it mean? And I feel like there's really only two numbers in this entire spectrum that mean anything. I think if you're at zero, it means you're not testing.
12:22
It doesn't mean you have a bad app. It means you're not testing it because there's zero lines of code coverage. If you have 100%, I still don't know, I think, about the quality of your app. But I know you have a metrics-driven incentive system inside of your business. Somebody cares about that managerial report. Somebody is measuring this for some arbitrary reason.
12:43
But based on my experience, I can't actually tell you what the quality of the system is. Because I've been on apps that had 100% code coverage. And it was crap. It didn't work at all. Anywhere in between, I don't learn anything at all. The number in isolation doesn't teach me anything about the quality of your app.
13:02
We also spent a lot of time where I don't feel we were actually testing anything. We were testing the mocks. So this is a whole lot of code that I don't know what any of it does. It's 15 lines of setting up mock objects of other services that had nothing to do with the thing I wanted to change.
13:21
Most of the tests in the system ended up looking like this, where I would change something that I felt was little and unimportant and six tests in different parts of the system would break due to faulty mock expectations. And I felt like this was very confusing, that you couldn't look at a test and understand anything about the behavior of the system because it was very convoluted.
13:44
I ran across this on Twitter. I didn't build it. But I love it. I love this. This is the perfect. This is the perfect analogy for unit tests. So we have each component.
14:00
And we can test each one working together. And it illustrates that it's obviously not right. This isn't a functional human being. But all of the unit tests are passing. And that was the problem on Big Corp Video Academy, is that we had all of these tests.
14:21
Everything passed. But when we actually turned the system on and worked, nothing quite fit together. You put the pieces on, and the app wouldn't quite talk to that other service the way we thought it did. And it didn't read from the database quite right. And the data wasn't in the shape we expected it. And so all of these other things
14:41
that we absolutely depended on for this app to function, we didn't have any tests for at all. And it was the source of all of our bugs. Did anybody see this? This is maybe a year old at this point. Anybody watch the RailsConf keynote with this
15:02
and then read the blog? Couple people. So this is DHH, for those who don't know. He's kind of like Fox News of the development world. He's very bombastic and says some outrageous things. But every once in a while, I think he makes some really interesting points. And so at RailsConf a couple years ago, he came out
15:21
and he did this keynote where he talks about test-induced design damage. And TDD is dead. It was very strongly opinionated. Like, it's dead. We shouldn't do it anymore. But I think he had a more subtle point to make that was interesting. It was that he would build up a system like he would in Rails.
15:43
And if he had just naively followed the TDD practice of like, I'm going to write a test for this, and then I'm going to build something, the code that he would end up with in order to test everything would be very confusing and unnecessarily abstract for the problem that he was actually trying to solve.
16:00
And so the point that he was making was that following this TDD pattern was leading him to designs that were not as good as if he hadn't followed TDD. And so this followed up with a very, very engaging conversation where obviously there was a whole bunch of people who rebelled against this and said, DHH is the devil.
16:21
How dare he claim any of these things. That is horrible. You're teaching people the wrong thing. But what followed was a really great conversation where DHH, Martin Fowler, and Kent Beck got on a video call together and actually had a rational conversation. Like, didn't yell at each other over the internet like what most of this happens across. Like, had a reasonable conversation.
16:42
It's like a four-part series out on YouTube. I think ThoughtWorks put it together. It's really great. Where they had this rational conversation about how do they approach development? How do they approach when do they get stuck? And how do you build a system that is both tested with what you're scared about
17:01
and designed in such a way that it makes sense? And my perspective on it is that Martin Fowler and Kent Beck are incredibly smart people. But DHH is really smart too in a different way in that Martin Fowler and Kent Beck are luminaries.
17:23
And they've devised some amazing things. But as an independent observer that I don't know any of them personally, I can't imagine, I don't know what Martin Fowler or Kent Beck have built. Like, I can't point it like, oh yeah, they did this awesome system here that did this thing.
17:40
No, they write books and they do consulting and they build stuff on whiteboards. And that's all great. But DHH built Rails and then built Basecamp. And these are hugely successful software projects that millions and millions of people have used and have created billions of dollars in value. And so his perspective is valuable.
18:01
Like, he might not be at the same level of software luminary as the other two, but he has some very practical advice about building applications that are actually valuable, that have created wealth and shipped a lot of awesome capability. So let's move on to another project.
18:21
This is Acme's collaboration computing platform. And I've intentionally put some buzzwords in this because this company was really big on the buzzwords. Anybody ever heard of Buzzword Bingo? Yeah, we've played Buzzword Bingo a lot, a lot in these companies' meetings.
18:40
You build a Bingo card and you put things like IoT and Cloud and Platform and things like that. And then you just slowly, as the managerial staff would say things, you'd cross them off. And then occasionally, you'd elbow the developer next to you and say, Bingo, I got it. Super fun. It was terrible.
19:01
All right, so this project was we were going to develop a platform for spies, the clandestine services, to upload confidential data and collaborate on their missions. Again, it wasn't this cool. I wish I was part of that project. That would have been awesome to be a part of that project. This was way more boring.
19:21
But again, I can't really tell you what it really was because then you might know what the company is and I don't want to get sued. All right, so we were going to build this platform for people to upload a bunch of data about these confidential things and then collaborate with other spies. And so this project, I look back on my career and I'm incredibly fond of it.
19:41
Some of the best software development experiences I've ever had have been on this project. We were doing some amazing things. We were talking about both market and technology frontiers. So we were building a product that didn't exist. There wasn't like, oh yeah, we're just going to build
20:02
SharePoint but this, or Uber but for that. There wasn't anything to compare this to. This was a brand new thing and it was also for a market that we had never worked with. We'd never talked to these customers. They didn't have a product like this and so we were going to build this new piece of thing or this new product for this group.
20:23
And this project had over 200 of the best people that I've ever worked with were on this project. Some of the sharpest designers, brilliant developers and testers. I met some testers there that could find all kinds of bizarre loophole conditions
20:41
and it was amazing. It made me a better developer working with these people. Building this project was costing the company two million dollars a month in burn rate. So that's like six beers here, seven, something like that.
21:01
It was a lot of beers. Anyway, so this project was burning a lot of money to build this project. And we had anything we wanted. The company had put a lot of money down on this. If we needed a license to something, we got it. If we wanted to build a new event hub system for that,
21:20
we could do it. Anything we needed, the developers kind of ran with this to make sure whatever needed to get done got done. We had no barriers. And we followed this thing called the testing pyramid, which a lot of people are familiar with. Has anybody seen this before? Yeah, just about everybody. For those who haven't, it's kind of like a metaphor
21:41
a lot of people use to talk about different kinds of tests in your system. And so unit tests are on the bottom because they're cheap, you can have a lot of them. Integration tests, you should have fewer. System tests, even fewer yet. That's kind of the metaphor that they're going for here. And so we were following it on this project. And so at the unit level, we did TDD.
22:00
We followed that pattern. We had a lot of conversation-based tests, so like object A, we talked to object B and we test all that through. We were doing microservices. Before that was really a word. We had a lot of different modules, and so each one had a set of integration tests that would call that module and tell me how it went. And then we had system tests.
22:20
We had a bunch of Selenium UI that would drive the UI through the system and make sure that everything kind of worked together. And so our testing was like spot on from this technical perspective. And so we built this thing. It took 18 months. This is Target Field. This is where, I'm from Minneapolis, and so this is our baseball field. We built this thing.
22:41
We launched it. 18 months, so much money, super proud. Bam, it's out. To my knowledge, they have not sold a license of this product. It is no longer on their website for sale.
23:02
I believe that this has officially become Shelfware, for those familiar with that term. So I actually took some time off of work after this project because I was actually really upset
23:22
about the whole situation. I feel like I learned quite a few things. From the results, from both building, I learned a tremendous amount of technology from the building of the collaboration platform, but also from its failure, because I was quite upset by it.
23:41
Like, I did not really want to work in software anymore for a little while after this. And I think part of the reason was actually due to this metaphor, and maybe a misunderstanding of this metaphor. Because, so the pyramid talks about like, it has this reflection that you should have more unit tests than you have system tests. It's bigger, it's on the bottom.
24:02
But instead, I like to think that this, that the pyramid, or these types of tests should address risk in your system. I like to think of tests are a way to mitigate risk. Think about your project in terms of risk. What can go wrong? What is going to break? What is gonna cause the project to fail?
24:23
So at the unit test level, unit tests are really good at mitigating functional risk. Do you have like, some complex functions in your app? Do you have a state machine that mutates? Do you need to like, deal with how do you move from state A to state B, or what does a complex conditional do?
24:41
Unit tests are great for helping you understand and make sure that that works exactly as you expect it to in all those conditions. Or maybe you have a system that orchestrates lots of different systems together, in which case you have some orchestration risk. Is your app going to be able to talk to the database the same way? Maybe you should have a test where like, your app calls the database
25:01
and it makes some expectations, or calls that third party service. I think integration tests are a great way to mitigate orchestration risk. System tests help, pardon me. System tests help mitigate interface risk. So you have a lot of,
25:21
if your UI is dependent on a lot of lower level services, and you have to flow through it, you can build system tests to help make sure that everything kind of comes together the way you expect and presents the right UI. But I think there's something on top of this that isn't always reflected.
25:41
And I think that's user tests or market tests. Maybe that's a better word for it. And so the collaboration platform had a huge piece of market risk. Like we were building something that had no comparable for a market we didn't really understand. There was a huge amount of risk in that project
26:00
that was kind of ignored. Like we spent, we're spending two million dollars a month for 18 months on something that we didn't fundamentally know was going to work. So maybe we should have done a test on that. Maybe we should have like, instead of building the super awesomely constructed
26:20
micro-serviced out MacDaddy approach, maybe we should have whipped together a quick and dirty Rails app in a week and shipped it instead of anybody would buy it. Like credit card form on the page, here's like one feature, will anybody buy it? Maybe that would have helped us like address the problem of when they didn't, maybe we shouldn't do the larger project at all.
26:42
The other thing is that each of these kind of risks, it doesn't really flow that you will have more functional risk in your project than you will market risk. The reflection is somewhat true of cost. Like it would cost you more to do user testing
27:03
than it would cost you to do unit testing. But it doesn't necessarily follow if that means you should do more unit testing. If you don't have any functional risk in your project, you don't need a lot of unit tests, right? It's just a waste of time if you don't need them, if you're not, if there's no risk there to address.
27:21
And so in terms of like how many do you have and not of cost, I think you need to look at it as a scale, is think about your project in terms of the kinds of risks that you have. Are you building for a group that you know? Is this just version 21 of a product that you've been iterating on forever?
27:40
In which case you probably don't have a lot of market risk. Or if this is V1, if nobody is using this before, you have a tremendous amount of market risk. And so what do you address here? So different kinds of apps will have different risk profiles. And I think that's up to us as the developers and testers and project managers and whatever to like think critically about our projects
28:02
and like what can actually go wrong and not just blindly follow a testing pattern that just happens to be like easy or fun or convenient or that's what we know. So look at the risk profile of your project and then build a testing program to like address that.
28:21
So if your app is fundamentally transforming SQL statements into HTML and back, you probably don't actually have that much functional risk. There's not really that much going on. Maybe you just need a handful of integration tests that like actually like call the database and make sure and that's all you really need. But if you've never launched it before,
28:42
maybe you should run a couple of market tests to make sure somebody's going to buy it before you spend a lot of time anywhere else. So it's kind of with that in mind. After the disaster of the computing of the collaboration project, I went on and I took a different kind of role.
29:01
I took a role where I was actually an agile coach for a monolith widget e-commerce Selamajig project which is a way better name than what it actually was. The actual name was like, it was like how Microsoft names their products. It was very boring. I wish they had called it the Selamajig instead. And so this was an e-commerce platform.
29:22
Has anybody worked with like e-commerce platforms before? Yeah? And so this was an established platform. It had already exist. We were going to do a few things to it. We were going to add like a product configurator so you could choose the color of your widget and stuff like that. And we were going to add a bunch more products to it and do a little skin redesign and stuff like that.
29:40
And there were some interesting dynamics on this project. Yet another one. The technology was kind of opposite of the last project I was on. The last project, the developers had free rein to anything they want. We had big machines. We had licenses to whatever we needed. Things happened.
30:01
The technology here was not so good. I think our, when a developer would open up their IDE on their computer, it would take several minutes for it to load. The technology that we were based on was an old Oracle product that didn't have a very good extension mechanism.
30:21
In order to extend their thing, you actually had to inherit off of their base classes and then do stuff to it. It was very difficult to work with. The thing that they did have going for them was lots and lots of captive users. So Monolith, they have this little market niche
30:40
for themself where the particular variety of widgets that they sell, they are the only ones that sell it. The people who need their widgets can only buy it from them. In a way, they're just printing money. If I apply the idea of scales to what was the risk profile of this project was,
31:02
you know what, there was almost no market risk. They're just gonna ship another thing to a bunch of people that had to buy from them. There was almost no functional risk because the application that they were building on handled all of that stuff for them. Really, they were just integrating a handful of other systems together. So they needed to test to make sure
31:21
the interface was gonna be pleasing and that we were able to call the third-party systems that we needed to call. However, this particular project didn't do what I thought they should do and write some system tests and write some integration tests. They just decided to write a whole lot of user tests
31:41
instead to not really address that problem. And in the way of user tests, they just made their users use it. Was there a user test? They just shipped it. That was their test. A little side note on unit tests is when I was first starting there, so I said I was an agile coach at this project,
32:01
and I was talking to the lead developer. And going in, I was like, so tell me about testing. Do you guys write unit tests? He's like, oh yeah, we write unit tests. And our conversation kind of drifted away. But a couple weeks later, after we'd started, I came back to him, because there'd been quite a few quality problems coming up. And I'd ask him again about,
32:21
so tell me about your unit test flow. How do you test your code? Tell me, walk me through your process. And the gentleman had a bit of an accent. I imagine I have a bit of an accent to you, but he had a difficult-to-understand accent with me. And he walked me through very slowly that, well first, a developer will take a ticket from the ticket system,
32:40
and then they will code the thing, and then they will open the app and make sure that it works. And that's what we call unit testing. And I'm pretty sure that wasn't just a communication breakdown. He called that unit testing. I think that that's just being a responsible adult.
33:02
But that was a point of contention throughout the entirety of my time there about what unit testing actually meant. But this product shipped. We got the configurator done, we rebuilt the UI, and we launched it. And the crazy thing was, everybody kept using it. Like, we did everything right.
33:22
Or we did everything wrong. We did everything wrong on this project. We didn't test anything. We just kinda like, duct tape and chicken wired something together and kicked it out the door. And it worked. And like, in comparison to the last project, this is amazing. And this is what I call
33:41
one of those dirty old duct taped machines that prints money. That you probably all have in your companies today. It's that really old system that all the developers say, oh, it's terrible, we gotta rebuild it. And the business is like, no, no, this is where all our money comes from. You can't, you might break it. And this system, it just couldn't fail for them.
34:02
And so what did I, I came away from this. I was only, this was a fairly short project, and it happened immediately after the other one. And so I was drawing a lot of comparisons in my head between the two. I just couldn't understand how, in one project, we did everything right from a technical aspect, and it failed.
34:20
And then next, we did everything wrong. And it was a huge success. Like, what, I don't even know what that means. And so if I compare the two, like, the model of e-commerce, it was an information technology department that was running it, which traditionally are cost-sensitive. They budget long cycles, and you can't really get anything new approved.
34:42
It was not really interesting. It was very fragile technology that Oracle system would fall over all the time. We didn't really test it, but it passed. Like, everything, everything worked. Customers bought it. But on Acme, we had an engineering department where, like, we could get money for anything if we thought it would grow the product. It was advancing. We were doing new stuff. We were using the latest technology.
35:01
We tested everything, and it failed. And it was, I think it comes down to these two projects were high risk versus low risk. You would have had to do even more wrong to make monolith e-commerce fail. And there was probably no way that Acme collaboration could have possibly succeeded.
35:23
And so I think, when I'm thinking about systems and what do I need to test, I need to think about what level of confidence do I actually need to have in that system. So, I don't know what I want to say about that.
35:43
Never mind. I'm gonna move on to the next one. This is the last story I want to tell. I think I have enough time for it. Yeah. This is the last story I want to tell, and this is kind of different. So, about three years ago, I started my own company called TrackJS. It's a JavaScript error monitoring tool. We have a booth out there.
36:00
You should come talk to us later, maybe. Although, everybody's probably leaving, so you're probably not. Anyway, so what we're trying to do, because I didn't obfuscate this because it's me, and I'm not gonna sue me. I like me. We find and fix JavaScript bugs in modern web apps. So, you launch a piece of software in the web. We help you understand when your end customers run into problems.
36:21
And that's TrackJS. It's on trackjs.com if you want to check it out more. I'm gonna skip over that. Testing scales. So, if I look at what was the risk when we started TrackJS, there's a huge amount of market risk. This is a brand new company. Any startup, any first product is gonna have a huge amount of market risk
36:42
because nobody knows who you are, nobody cares who you are. You have to do a lot to convince people that you have a problem that I can solve. It's really hard to convince people of that. So, you have a huge amount of market risk that you need to overcome, regardless of the technology you approach. And then, with our specific platform,
37:01
our interface was relatively simple. We didn't have a whole lot of complex interaction that happened there. Not so, we had a fairly low risk there. Building a company today, you have a huge number of online services that you can depend on for email sending
37:20
and financial transactions and social integrations and all kinds of stuff. And so, a huge amount of our work was actually just wiring together bits from the internet and calling out into other services. But that became a risk for us because I had to make sure with how do I talk to Stripe, which is a payment processor. When I talk to Stripe, I need to be pretty darn sure
37:40
that that credit card transaction's gonna go through or I'm not gonna get paid. So, I test that a lot. But in terms of our actual code, there's a few interesting bits of how do I take an error from your site and correlate that with a bunch of other data and give you something meaningful. There's a little bit there. But I'd say it's about a fourth of our market risk.
38:03
It's not huge. There's a few interesting bits, but not a lot. The interesting thing going into testing this is when you're working for another company, you come in in the morning, you work eight hours, and you get a paycheck, and it's fine. Like, if you get some stuff done that's awesome.
38:20
If you don't get as much stuff done, that's fine. It takes a long time before you would stop getting your paycheck. When it's your money, it's literally my credit card that is paying for pieces of hardware and services, and so your brain changes around about what is really important. Do I really need to do this,
38:41
or can I just ship this and see if it makes me some money so that I can pay off my credit card that I've been charging up for the last seven weeks? When it's your money, it became very important to us to launch fast and then measure what happens.
39:04
So rather than spending a lot of time testing all of the backend pieces, we would get something out that, frankly, we were embarrassed about. I would not show any of you some of the code that I have released into production because I'm quite ashamed of it. However, what you get out of it is you get feedback from real users.
39:23
So when you put something out there, I measure how well does it work. So testing can lead very well into monitoring. If you don't test it at dev time, you can push it out into prod and see, well, does it actually break like the way I thought it might, or is a user actually going to use this
39:41
in the way I thought they would? And so applying our test plan back to the actual risk that we saw, here was the kind of test we wrote. We spend a lot of time on user tests. We spend a lot of time where we ship little pieces of code out and see if it works,
40:01
see if anybody uses it. If I ship a new feature or write a blog post about a particular thing, does my sign-up rate increase? Do more people sign up for my service if I talk about this thing? And with that, I'm constantly testing. I see that as testing. You might not see that as testing. I think of it as I am testing what features do my customers want?
40:22
What are the next things that we should build? And that addresses the market risk, which is our biggest thing. In orchestration, we spend a lot of time writing integration tests, which some people will say, oh, those are too slow, they spend too much time, you shouldn't do that. Integration tests actually help us. We really call into Stripe and try and run a transaction and see if it works.
40:42
And Stripe has test modes to support that. When we write tests, we really call into our third-party providers and make sure that we call them the way that they expect to be and it was the way that we get confidence on it. We do have unit tests, but we don't have a lot. We have way fewer unit tests than we have integration tests because we only use the unit tests
41:01
when we're actually doing a state change. I'll show you some more of that in a minute. So here is like the actual kind of, this is the closest thing to code or architecture I'm going to show. The structure that we approach our service. So when we write something new, we want to put a new feature out. We write a beta service.
41:22
Our code is kind of structured in a metaphor of services. We write a beta service. This is going to be a thing that we're going to do. And we launch it, but we launch it with tons of monitoring around it. I use a service called Mixpanel. I don't work for Mixpanel, but they're pretty cool. Mixpanel's kind of like an analytics tool that you can track what users do
41:41
for marketing purposes. And I use that to tell me how are users using the service? And then I have a subset of my customers that I call canary users. The canaries don't know that they're canaries. I just know that they're canaries. So I launch a piece of software and a subset of my users will see that new capability.
42:03
And I measure how many people click on it. If nobody clicks on it, I delete the beta service. I'm like, you know what, we're not going to do this. At this point, I didn't really write any tests for it. I just kind of got it together as fast as I possibly could and then I show it to 2% of my users.
42:23
And I see, did they like it? Did they click on it? Did it blow up when they clicked on it? I want to see what are they going to do with it. I got beta service out as fast as I possibly could because I don't really know if it's going to work. If it works, if I feel like, you know what, there's something valuable here, there's something that can go, then we move it into a real thing
42:41
and we actually break our service apart. And so here's kind of how we think about testing. If there is something that I really want a unit test for, like I have a state machine, I have some complex conditional, I'll build something as a model. I'm trying to get something down to where I have a method that takes inputs and sends out outputs
43:01
and that's all it does. And those are super easy to unit test. Super easy, because you can just bang them through all day, you're not sitting up any mocks, it's good to go. Anything else is an integration test. We're like, we have a bottom level text fixture which basically just switches which account we're talking to, like, am I talking to production stripe or test stripe?
43:22
Am I talking to production mail chimp or test mail chimp or whatever? And so when I write an integration test, it really goes out and creates an error or signs up a user or does those things. It just does it in a test environment instead of the real environment. But it's really doing those things. And because of that, I have a huge level of confidence
43:41
that when I take my service and I put it into prod, it's gonna work. I've been calling this thing all day, it's gonna work. I'm not concerned about it. Now I'm not saying we don't have bugs, because we do. We've had bugs from time to time. But the thing that's really interesting is that sometimes
44:00
being fast to fix a bug is almost as good as it never being broken at all. And sometimes it's way better. And so one of the things, so we are a service for developers. And so sometimes a developer will be like, hey, there's a bug on this page. This thing should be doing this. You know what? Sometimes it's true. And we find it.
44:20
We write a little fix for it. Our code is super simple so that we can get on it. There's not a lot of test break because of how we approach this. And we can usually patch it within two to eight hours. And then we email them back and say, hey, you're right. That was a bug. Sorry, we got a fix out. And you know what? Those developers love us. The fact that they found a bug and we responded to them
44:42
and we fixed it in eight hours, they're like, oh my god. It takes us like six months to fix bugs. How do you do that? Those customers, and we've tracked this, those customers are some of the most loyal customers that we have. They love us. And so showing somebody that you are incredibly
45:01
responsive to them is almost as good as it never been broken in the first place. All right, so I'm going to wrap this up. It sounds like some of the other ones are finishing too. So here is the five projects and what each taught me. I'm going to condense the whole thing down into five things that they've taught me. One, Big Todd's awesome CRM. Don't test to hit metrics.
45:21
The metric doesn't matter. The metric is just an arbitrary projection. And as soon as that metric becomes what you're trying to hit, you're losing sight of the real goal, which is to build a system that works and produces value. Big Corp, test what breaks. In that project, we kept unit testing. We were chasing this number that wasn't what was causing any of our problems.
45:41
What was actually causing our problems was a weird database that had no tests against it. And seven different projects were all hitting the same database in different ways. And nobody spent any time testing that. Take the thing in your project that is breaking, that is causing pain, and spend time focusing on that. Acme's collaboration computing should test the scariest thing
46:03
first. And this one's hard. This one's really hard to overcome your mental model. They should have built a quick and dirty Rails app and shipped it and tried to sell it. But that was really scary. It was a lot easier for them to throw a bunch of developers at it and say, hey, let's go build this thing.
46:21
Because then you didn't have to have any scary conversations or rejections. You didn't have to face the things that you don't know, the things that you're not good at. We always want to do the thing that we're good at. But sometimes you've got to look at the project and be like, you know what? There's that dark corner there. And I know we're going to have to do it someday. You should really do it right now. Because that's the part that's going to blow the project up later.
46:44
Tests cannot substitute for customer use. So it doesn't matter what your tests say. If the customers use it and it works, there's no problem. If the test all passed and the customers can't use it, it's still broken. Ultimately, that's what matters.
47:01
And so you can sometimes substitute giving it to customers for spending time testing. And finally, monitoring can substitute for tests. If you don't have time to test everything, if you can't get every single edge case, testing can move to production as in the form of monitoring. Where you're monitoring what are the unhandled exceptions,
47:21
what are the problems that are happening, what are the things that are actually going wrong, versus what the things that the developers imagine can go wrong. And so depending on the risk profile of your application, you can move some of the time you spend testing to time spending building and watching monitoring tools. If you need monitoring tools for your client-side
47:44
application, you should check out TrackJS. And this is me. This was software testing for failed projects. I'm Todd H. Gardner. You can get me at Todd at TrackJS if you want to email me and tell me how wrong I am. I'm fine to talk about that. If you give me a vote on the way out, if you hit the green one, you can take a TrackJS sticker. You can still take a TrackJS sticker
48:01
if you don't push green, but I'll be sad about it. That's all I got. Thank you very much. I hope to see you next year.