Continuous Integration and Delivery
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 96 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/51817 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
NDC Oslo 201616 / 96
2
7
8
9
12
14
19
20
26
28
31
33
38
40
43
45
48
50
51
61
63
65
76
79
80
83
87
88
90
93
94
96
00:00
Continuous integrationGoodness of fitWeightNumbering schemeEnterprise architectureSelf-organizationOrder (biology)BitData managementMereologyTerm (mathematics)Projective planeInformation securityMemory managementPattern recognitionLattice (order)Disk read-and-write headProgrammer (hardware)CircleCore dumpApplication service providerInterior (topology)CodeComputer programmingCodecEstimatorSystem callHidden Markov modelContinuous integrationComputer animation
03:36
YouTubeWeb pageUniqueness quantificationWordGoodness of fitGroup actionMeasurementMobile appWebsiteGame theoryProduct (business)Traffic reportingFormal languageComputer animation
04:45
Product (business)WebsiteTerm (mathematics)WeightCodePhysical systemComputing platformCycle (graph theory)Dependent and independent variablesInstance (computer science)Fluid staticsMathematical analysisData managementCanadian Mathematical SocietyProduct (business)Mobile appFrictionLevel (video gaming)Vertex (graph theory)MereologyExpert systemConstraint (mathematics)Order (biology)Information securityAuthorizationEnterprise architectureSelf-organizationSoftware frameworkProcess (computing)Core dumpWeb pageComputer programmingSoftware testingNumberWave packetActive contour modelPattern languageSoftwareCone penetration testPoint (geometry)Task (computing)MappingTrailProteinComputer animation
12:20
Service (economics)Server (computing)Active DirectoryVisual systemPoint cloudWeightDependent and independent variablesMathematicsInformation securityInternet service providerExecution unitMeasurementBit rateComputer clusterBitAgreeablenessTerm (mathematics)Parameter (computer programming)MultiplicationCircleEnterprise architectureMultiplication signCuboidInheritance (object-oriented programming)Product (business)Task (computing)Category of beingInstance (computer science)Focus (optics)Self-organizationData Encryption StandardMereologyPhysical systemWage labourBuildingRight angleError messageProgramming paradigmOrder of magnitudeExpected valueDisk read-and-write headTowerScaling (geometry)ScalabilityPoint (geometry)Figurate numberIP addressDigitizing
19:55
DatabaseLattice (order)Staff (military)Term (mathematics)Latent heatFood energyMultiplication signEncryptionFormal languageComputer animationMeeting/Interview
21:08
Lattice (order)Exterior algebraInformation securityProjective planeData managementExpert systemNatural languageEncryptionPresentation of a group
22:09
Formal languageSoftware developerBuildingTerm (mathematics)Medical imagingImage resolutionSlide ruleMereologyProjective planeInformationContext awareness1 (number)Domain nameInformation securityFormal languageInstance (computer science)SatelliteWordEvent horizonComputer animation
25:15
Information securityBuildingMetreMedical imagingImage resolutionData miningInformation securityBuildingTerm (mathematics)Instance (computer science)Computer animation
26:35
Process (computing)Hacker (term)Source codeInformation securityBuildingTraffic reportingNumberMedical imagingBuildingProcedural programmingSelf-organizationMoment (mathematics)Information securityInstance (computer science)Computer animation
28:36
Point cloudService (economics)WeightServer (computing)Active DirectoryVisual systemQuicksortContext awarenessMedical imagingMereologyBuildingProjective planeInformation securityOrder (biology)Nichtlineares GleichungssystemTerm (mathematics)Slide ruleLevel (video gaming)Formal languageComputer animationDiagram
29:30
CodeRule of inferenceProjective planeMedical imagingData managementTerm (mathematics)MereologyDeclarative programmingLattice (order)ScalabilityBitPoint (geometry)Food energyMultiplication signInformation securityInstance (computer science)PlanningUsability
31:44
PasswordString (computer science)Source codeInclusion mapRepository (publishing)Data storage deviceComputer fileServer (computing)Source codeProduct (business)Data storage deviceRevision controlScaling (geometry)Filesharing-SystemInternet service providerComputer filePasswordCartesian coordinate systemQuicksortConfiguration spaceGroup actionPoint (geometry)Repository (publishing)Information securityOrder (biology)LeakFundamental theorem of algebraTerm (mathematics)Connected spaceString (computer science)Regular graphNumberProjective planeUsabilityInstance (computer science)Context awarenessFile formatKey (cryptography)EncryptionServer (computing)Right angleArithmetic meanINTEGRALDegree (graph theory)Perspective (visual)CASE <Informatik>Disk read-and-write headCodeLine (geometry)Vector spaceFunctional (mathematics)GodProcess (computing)Continuous integrationAnalytic continuationShared memoryUniform resource locatorComputer animation
39:01
Software developerSystem administratorInformation securityIntegrated development environmentDatabasePasswordMalwareBuildingLoginCodeScripting languageDirectory serviceIntegrated development environmentMedical imagingAuthorizationData storage deviceInternet service providerDivisorOrder (biology)String (computer science)DatabaseConnected spaceConfiguration spaceComputer fileMoving averageTerm (mathematics)Cartesian coordinate systemProcedural programmingScheduling (computing)Vector spacePasswordCodeDirection (geometry)MereologyWebsiteFigurate numberFunctional (mathematics)Hacker (term)LoginBuildingInformation securityDifferent (Kate Ryan album)CASE <Informatik>Slide ruleLine (geometry)InternetworkingAuthenticationMultiplication signInstance (computer science)Wave packetSystem administratorGraphics tabletPhysical systemRoyal NavyComputer animation
46:17
BuildingInformation privacyPasswordMusical ensembleVolumeOrder (biology)Twin primeSummierbarkeitArtificial neural networkTunisChainScripting languagePublic key certificateBoom (sailing)MereologyLink (knot theory)Key (cryptography)Data storage devicePattern languageGodTerm (mathematics)Source codePasswordPhysical systemComputer fileStandard deviationRevision controlParameter (computer programming)Order (biology)Functional (mathematics)System administratorDesign by contractReading (process)Instance (computer science)Level (video gaming)Rule of inferenceInformation securityData loggerGroup actionError messageWebsiteGame controllerNichtlineares GleichungssystemArrow of timeCodeSelf-organizationDifferent (Kate Ryan album)DigitizingTemplate (C++)String (computer science)Electronic mailing listObject (grammar)BuildingIntegrated development environmentMedical imagingSource codeXML
53:33
EncryptionKnotMountain passPhysical systemWritingPasswordTrailPressureFormal languageInformation securitySkeleton (computer programming)HypothesisMultiplication signMedical imagingInformation securityPublic key certificatePhysical systemProof theoryBuildingSoftware testingPerspective (visual)Point (geometry)Symbol tableError messageVector spaceInstance (computer science)FeedbackPasswordPressureScripting languageSkeleton (computer programming)LoginFormal languageCanonical ensemble2 (number)Term (mathematics)Order (biology)Visualization (computer graphics)MereologyCategory of beingRight angleHidden Markov modelDoubling the cubeFunctional (mathematics)Interactive televisionModal logicGreen's functionField (computer science)Cartesian coordinate systemNumeral (linguistics)Computer animationEngineering drawing
Transcript: English(auto-generated)
00:04
All right. And good morning, everybody. Nice you could be here. Hope you had a pleasant night at Oslo. I did, for sure. This talk is going to be about continuous integration and delivery.
00:23
In respect of my work, I work currently at Lego, which probably all of you know. I have a background as a .NET developer. I've been working with various companies. I started out at Just Eat, which was actually founded here in Oslo, if you didn't know.
00:45
I didn't earn any money on it. Others did, but not me. It was quite a fun journey, anyway. Then I've been at eBay, a non-profit organization, before I came at Lego. I'm not the best programmer out there, and I'll come back to that, because I attended
01:05
a session yesterday where two guys from Microsoft, from the ASP.NET core team, were talking about how they were optimizing the inner workings and memory allocation and stuff like that, in terms of getting the .NET core to perform better.
01:24
I'm not that kind of programmer. That's not me. That's not what I do. I'm much better at pattern recognition and trying to understand the what in things. And I'll come back to that later on.
01:41
I like automating things, and I tend to have a pragmatic approach towards things. This session will cover a little bit of background in respect of what I do at Lego and what Lego is all about. How do I fit in in the larger scheme of things in Lego, because it's an enterprise
02:03
organization with 17,000 employees. I think you need that background in order to understand the problems that me and the team I'm part of are trying to solve. So I'll just touch upon that. Then a bit upon security. What is security?
02:22
When you talk security, people have their own subjective feeling on what security is. But I think we need to elaborate a bit on what security is in terms of understanding how to implement security in the solutions that we're about to provide. And then, finally, from the trenches.
02:42
I've collected a few examples of what we've done. My experiences from the past, failures I've made, others made. All of them probably had something to do with providing cake in the end and the retrospective. And there will be code.
03:03
This is a developer conference, and if you were project managers and some people tend to fill up their entire day's work with meetings and writing agendas and stuff like that and running around in circles and nothing gets produced, the shoulders kind of
03:25
meet on top of their head when they see that there will be code. So it's kind of funny to see from up here when that happens. Your developers, shoulders down, please. There will be code. And that's okay. So, Lego. What is Lego, in essence?
03:40
We have behind, if you go to www.lego.com, you'll end up seeing this, the front page of our website. It's actually around 180 websites, and we support 25 languages. In our last annual report, we stated that we have almost 20 million unique visitors per month.
04:03
We have a YouTube channel with 1.3 billion visitors or actions or however you measure it in 2015. We measure things in millions and billions, so it's quite a large website. We make apps, lots of them, apps for kids. We have some very clever people who are able to write and produce apps for kids who hardly can speak.
04:27
They can't spell. They don't know the words. They don't know the letters. Even so, they're actually able to navigate and play a game and have a good experience in it. I don't know how they do it, the guys who make the apps and stuff, but they're very clever, and we do that a lot.
04:43
And games, of course. Each website is a product, and a product could be the Ninjago site or the city site or the Star Wars site or whatever site. There's a few examples of them here. And we have, in terms of how they are produced, we have product teams in India responsible for giving sites.
05:08
We may have a city product, and a team in India and Denmark are responsible for maintaining and implementing features on the city site, as well as the Angry Birds site, as well as the Friends site and elves sites, etc.
05:24
Behind the product teams, which are very product focused, they just churn out code in terms of the product that they're responsible for. We have a platform team, and they work on a global level. They provide NuGet packages for our Sitecore CMS platform, for instance.
05:42
These sites are built on Sitecore. And we have platform teams responsible for the Sitecore. We have a team, which are entirely focused on tracking, for instance. And we have a team responsible for, well, they produce things and gimmicks for app development as well.
06:02
They're kind of more horizontally focused, where product teams are strictly vertical. They can have several verticals that they're responsible for. And then we have a system team, which is me, with Friends. We manage all this by using SAVE.
06:21
By all means, I'm no expert in SAVE. You're welcome to come ask me about SAVE, if you're interested. SAVE is Scaled Agile Framework for Enterprises. The guy who invented Scrum, in a sense. He wrote a book about it, and became quite a de facto author of what Scrum is.
06:44
Mike Cohen, I think it was. He wrote a book, and held a lot of courses, and earned a bucket of money. And then enterprises started to pick up. And 10 years had passed. And he wrote another book about Scrum for Enterprises, and made a bucket of money.
07:00
And that's where we are today. That's called SAVE. It's essentially how you scale Scrum teams in enterprise organizations. That's what we do. I think it works for us. It has drawbacks, of course. Everything does. But we have the numbers to prove that people's work satisfaction, and the visibility when 20 teams are working together,
07:24
the pendency mappings, and how risks are being escalated. It works much better than before SAVE was introduced into our organization. So come ask me, please. But no detailed questions on release trains and stuff like that.
07:41
This is a system team. There's me. I have no idea why I ended up being the two-headed eagle. Or two-headed snake. I don't know. Then we have a guy named Alberto. He's Italian. He knows about embedded programming. Then we have Michael, who's actually sitting right over there.
08:01
Watch it. Hi, Michael. Who's Barbosa, I think. From Paris over the Caribbean. Michael knows about SQL and stuff like that. Kasper knows about test automation, testing things. He knows about .NET development as well, just like I do.
08:22
I see myself as, regard myself as being the developer who knows how developers think, how they act, what their core values and stuff like that. Not that the others don't know, but that's my background. Then we have a guy named Jonas, who has an architectural background and is able to read 2,000 pages in a book
08:44
and actually remember those 2,000 pages. I can't do that either. We have a product owner with swords and knives fighting the rest of the business, fighting them off, I think. If something needs to be prioritized, go to him. If business wants something, needs something, grooming needs to be done,
09:04
go to your neck. Then we have our self-proclaimed snowplow who is trying to mingle things with the organization and he is our manager. He proclaims himself to be a snowplow and tries to remove impediments and stuff like that.
09:22
What we do in the system team, we are responsible for everything below the platform teams, from the platform teams and down. We are responsible for our soft repositories, for instance. We have the responsibility for our static code analysis tools, everything that all the other teams are using,
09:41
both platform teams and product teams. We are also responsible for the build and deployment pipeline because that is something all the other teams are using as well, including platform teams. We are the foundation of the mother of everything, actually. It's a fun place to be in because you get to know a lot of things and that's where I fit in because when you have so much knowledge
10:06
about how the business is working on a horizontal level, you start to see patterns emerging. You can't see them when you are in a product team and you're just churning out code and you have a cycle website and you have very strict deadlines.
10:20
You just need to get things done. But in terms of being able to change things, you need insight from all the product teams and the platform teams in order to be able to see where the friction is and how to improve the build pipeline. You can't do that by being in a product team.
10:42
So I fit in very nicely where I belong right now. That's what we do in the system team. We do automation. Actually, you can say that we are just maintaining and providing features for an assembly line. People stuff in code at one end and magic happens
11:05
and you end up having an artifact which you can put in a package, put it in stock and deliver somewhere else at the end of the pipeline. That's what we do. A large part of what we do is also to provide APIs for the developers.
11:24
We try to enable them to be creative, to do what they think is right for the task at hand. We try not to be a constraint. At some point, we are a constraint because in terms governance contradicts total freedom to do what you want.
11:45
And we need governance in terms of security. I'll come back to that. So sometimes we are perceived as being a constraint, but we try not to be. We want to enable our developers in all the teams to be self-serviced, for instance, and be able to do whatever they want.
12:02
And we've come quite far, but we're not there yet. Because we are located where we are, bearing the foundation of everything, there's also a support process where if you have a problem and you don't know where to go, come to us.
12:20
And because when I started a year and a half back, we had people who were running around in circles because they didn't know where to go if they had a problem with which nobody owned, with a product that nobody owned, or how do I get access to this system that nobody really wants to touch?
12:41
Every enterprise had them. So people started running off to Michael. He almost never did anything else than supporting people and didn't have time for features because being a supporter is a rewarding place to be and everybody likes a fireman. You come out and put out the fire and people are happy,
13:03
they warm up again with a good experience, but it doesn't help the business very much. You don't have time to implement features and help the business in the long run. You just put out fires all the time. And in the system team, we decided to swim lane all the task switching involved
13:23
and call it the DES developer support, DES for Digital Solutions, which is the department I'm part of. So one week at a time, we have support and there's one guy doing all the task switching and all the other guys are working on the assembly line implementing features.
13:41
Come ask me about how to do support. I think we have good reasoning behind the things that we're doing, but that's not the focus for now. That'll be a totally different talk then. These are just on the top of my head, the technologies that we need to know a little bit about all of us.
14:01
Some of us need to know the how about technology. Some of us need to know how to set up Git, how Git works, or HTML or Amazon Web Services, Mac, Pearl, whatever. But we all need in system team need to know where each of these technologies fit in
14:21
and technologies emerge every day. When you enable developers to think independently, they start to be creative. They are uncontrollable. I know I'm one of them. So you need to keep an open mind, essentially, to be able to work in the system team.
14:41
It's not for everybody. And you have to be quite clever, I think, in terms of being able to keep up. I don't know a lot about Node.js, for instance, but I know it's something we use heavily. I don't know how you do it, but I have quite a good feeling of what it is and how it fits in, just for an example.
15:04
Lego has grown dramatically during the past years. And we have growing pains right now. We're transitioning into a global organization. And in terms of scalability, that causes growing pains. The solutions we implement
15:20
and the examples I provide to you afterwards is an example of how we try to provide solutions that scale. If we take technical debt, for instance, when implementing a solution, we might implement a solution or implement a feature with a certain amount of technical debt
15:41
or manual labor involved. And that's okay, but we have to consider that Lego in two or three or five years will maybe be twice as big. So the manual labor we introduce today might be a magnitude of two or four times in just a few years from now if we grow in the expected growth rate.
16:02
How do we utilize offline resources? How do we scale the organization and implement IT for the organization, which is also scaling? That is one of the major pain points, to nail that one right. And errors do happen, but we try to...
16:23
And the paradigm of giving people responsibility is a major cultural change because before, when you have maybe 8 or 10 or 12 teams, and you know every team that are working for you,
16:40
you can be a dictator in many senses. That doesn't work anymore when you have 25 or 35 teams or maybe 40 or 50 teams in a few years from now. I think now we have about 25 teams and we're growing rapidly. So we have to provide infrastructure
17:01
and organizational units to enable them to work effectively today as well as in two years from now. So, a little bit about LEGO.
17:32
On security. Security is immensely important. I don't think anybody disagrees.
17:42
But what is security in itself? How do we define security? In terms of LEGO, LEGO is one of the most powerful brands out there in terms of trustworthiness, loyalty. There's a company called Brand Finance, I think it is,
18:01
that measures and rates all large companies throughout the world, rate them against each other, and LEGO has been in the top 10 for 8 or 10 years, I think. In terms of loyalty, trustworthiness, etc. etc. They have lots and lots of parameters to measure upon. And that comes from somewhere.
18:25
In terms of what we do and in terms of preventing data leakage, we need to ensure that people trust LEGO because we're not selling LEGO boxes to kids, we're selling LEGO boxes to their parents. And if they don't trust LEGO to be able to
18:44
ensure the safety of their child's data, they won't buy anything from us. Or that we are unable to earn as much money as we do on selling LEGO boxes as we do.
19:00
We also have multiple IPs, intellectual property owners, that enable us to build Star Wars LEGO. Star Wars is owned by Disney, for instance. And we have a strategic partnership with Disney that enables us to build Star Wars LEGO. If we have data leakage, if we somehow compromise data,
19:23
that falls back on Disney somehow, we might not be able to build Star Wars LEGO anymore, we might not be able to do that. And that will have a, shall we say, huge impact on what LEGO is able to do. Also, we can't trust all our employees, we have 17,000 of them.
19:41
So, what do you do? How do you... That sense of security is important to us, but how do we implement it in our everyday? How do we begin doing that? And back to basics, in our everyday, you're in a meeting.
20:02
You're in a meeting with, might be, grooming a new feature, you have your stakeholders, and you have non-technical staff there. And you start talking, you start, we want this and we want that, and I would like to have, et cetera, et cetera, et cetera. And somebody starts questioning, are we safe? How do we ensure that this solution is safe?
20:23
In terms of how do we protect our data? Can we tell our users that, how do our marketing tell our users that we are safe? And somebody stands up and says, what if I told you that we have used a specific encryption algorithm? And everybody dies a little in that room.
20:43
It just doesn't work very well. Energy leaves, just evaporates. Because you have a technical guy who's not talking at eyes' height with a non-technical stakeholder that you have, and you need to protect your stakeholders from guys like that.
21:00
I'm one of those guys. It happens all the time. But nevertheless, it's important to try to talk in language that your stakeholders can understand. So you don't say Rindell and AES and stuff like that to your stakeholders. They don't understand you.
21:21
And because you're the technical expert, nobody dares to question or say, can you explain in human language what you actually mean? It's very rare to experience project managers and stakeholders who are just marketing people, whatever, questioning a developer on whatever the developer says,
21:40
because he's the expert. He makes things happen. I'm sure you can recognize meetings like that. If you have technically savvy people present at the meeting, of course it's okay to talk tech stuff. Of course it is. But we need to propose an alternative for the people who are responsible,
22:01
and we want to be accountable for security, but don't know about encryption algorithms. We need to propose an alternative. So instead, he should have said this, we have secrets, and we have a happy stakeholder, because everybody knows the concept of a secret.
22:22
It's just wording, but it's immensely important, I think, trying to get people to understand what is happening, what is the inner workings of the features that we are providing. If we have secrets, a secret is a secret.
22:42
That's what it is. You don't have to elaborate on it. You don't have to talk anymore about it. It's just a concept. People are smiling. Why is this funny?
23:02
It's because I have a son. He's 10 years old. He was looking over my shoulder when I was preparing for this. And he sat for a while, and we came to this slide, and he looked, and he said, that's just plain stupid. And he's 10 years old.
23:22
And so he understands. He's 10 years old. He understands. He doesn't even know what I'm doing and what I'm working on and why I'm going to Norway. But he understands this image, because the absurdity is so obvious. Wouldn't it be rewarding in a fantastic, sensational way
23:44
if we could talk about security with our stakeholders in terms of images like that? If they could understand that what they are asking for is essentially this? A solution that doesn't solve anything?
24:01
Security by obfuscation, for instance, is just exactly that image. But people don't get it if they even if they don't know tech stuff, they might not get it. And we're responsible for providing them with the information in a context they know and can take into account.
24:23
We're the only ones who can do it. And the end goal should be for them to take ownership of the security features within their solution. So I have a proposal for you on the next slide. Oh, yeah. The part that I wanted to elaborate on
24:42
is that there are terms from domain-driven design. It's been eons since I read the book. It's from Eric Evans, who wrote about domain-driven design, coined the term. He said that you need to build a ubiquitous language, which is a shared language between the developers and stakeholders.
25:01
And everybody onboarded on a project to use the same terms within the domain of whatever they're trying to accomplish. What have we used in terms of security?
25:21
That security is wall building. You have an asset on one side. You have a threat on the other side. You want to protect the asset, so you build a wall. It's a concept easily understood. Everybody knows what a wall is. Just look at the image. They have barbarians on one side.
25:41
They kept flooding the whatever Ming Dynasty was 3,000 years ago. So they built a wall 3,000 kilometers long. And problem solved. And we need to be able to explain to our stakeholders and to ourselves that what we are trying to do is wall building.
26:02
And then you can start having a discussion where to build those walls. How high should they be? Do they need to have barbed wire on the top, for instance? Do we need to have crocodiles in a tent outside and land mines and whatever? Or do we just need a fence maybe a meter high because it's not that big a threat?
26:21
Or it's just very, very rare that we're even going to be exposed and a wall that high made out of wood would be perfectly fine? There's a gotcha. There's a catch-22, I think it's called.
26:43
I found some numbers in a report from 2007. If you look at this image, you have assets, you have threats, you build a wall. It's very clear where the threat is located. People tend to forget that threats very often emerge from the inside.
27:02
So you can build all the walls you want, but you need to build walls and establish procedures within your castle as well. You can't just say that you've locked everybody out officially and now you're safe.
27:21
That's one of the most common misconceptions I could think of. The Panama Papers, for instance, I think they had security in their organization. I don't know if they still exist. That Mossad Fonseca had the proper procedures They needed to log in every morning when they went out to work,
27:42
but still they were compromised. They had a data leakage for the benefit of everybody in this room, I think. But they had customers and they, I don't think their trustworthiness is very high at the moment at their customers. Edward Snowden is another example. And as they do have procedures and they say they know stuff,
28:03
but data was leaked anyway. So take this into account when discussing where to build the walls. It's very important that you don't underestimate where the threats are located and how security breaches happen.
28:20
I haven't read the entire report. I have to do it, even though it's ten years old. I think those numbers still apply for today. What we have to keep in mind when discussing security in order to enable stakeholders to take ownership
28:40
is to keep technology out of the equation. Once you start talking security in terms of the technologies on this slide, and you are trying to explain something to people who are not aware of technologies, they might not know what S.net is and Git and how Perl and Puppet fit into the image,
29:01
you've lost your stakeholder and you've essentially lost a very important part of being in a project and team building and all sorts of things. You need to keep technology out of the equation on a high level in order to establish that ubiquitous language,
29:21
which is so important when you have technical and non-technical people trying to gain a common understanding of the problem at hand. There's a funny story to this one. It was actually Michael. We were sitting at a planning meeting. We have bimonthly planning sessions as part of SAFE.
29:42
We were bored, I think, just a bit. We started discussing how you could explain what security is and how you do not apply security. You do not build a solution like baking a cake.
30:02
You can't just provide it as a topping afterwards. You need to bake it into the cake. I think the meeting stopped a bit early because of lack of energy and whatever. I went down and shopped and we built this the day after. You would be surprised how many project managers,
30:22
managers in general, everybody, both developers and non-developers have commented on that exact image. All the ilities that we know of, scalability, usability, security, but it's hard to explain why they're important to plan for.
30:41
That image, I think we actually got to somebody just by building that. It took maybe an hour. I brought it as an example today because it just worked. It was immensely important in terms of getting people to understand, providing them with an image that they can use
31:04
when stories needed to be prioritized, for instance. A developer asked me if this rules out declarative tags and code. Of course it doesn't. It was a good question but it does not because you just need to plan for scalability and security.
31:22
All the ilities, you need to plan for them ahead. You can apply them in different fashions. That's not really the point here. You just need to plan for them and not forget them and take them into account every time. That's what that image is trying to tell from the trenches.
32:03
Usually user groups, if I'm giving talks there, tend to take questions but come to me afterwards. I'd be happy to answer everything and if I can I'll just point to Michael and run away. All right. Down to basics. You have a secret.
32:21
This is an example of a secrets file. I just made them up. Homer Simpson doesn't work for Lego, by the way. This could be a file with secrets. It could be any configuration file out there. I think how many of you have checked in files
32:40
into source repositories with secrets in it, like usernames, passwords, et cetera? I think everybody has, yes. And it's wrong because then it's not a secret anymore. And you probably don't know who had access to those secrets and did they take them with them?
33:03
Did they take a copy of that file? Well, your secret's today. So in order to protect secrets like that, we always say everything belongs in a source repository. It's a fundamental term in everything that we do. Infrastructure has code and everything has code
33:21
and code, code, code, code. But if you want to protect your secrets, Scott Hansman wrote about it and he said that essentially you need, in order to protect them, you need to keep them out of your source control and out of your source repository. And I think he's right. Because you can't protect anything that you've checked in
33:43
and shared, source repositories contradict the concept of keeping secrets a secret because source repositories are essentially sharing text files and so you don't want them in there. Stop doing it. I still do it, but I shouldn't do it.
34:06
So I'll try to provide an example that how you, how you avoid checking in secrets into your source repositories. And this is a very low-key example.
34:21
I was thinking about what is the minimal viable product, what is the absolutely least you can do in order to get your secrets, protect them, get them out of source repository, but still have a means of deploying them in a fairly decent fashion.
34:41
And this will work in continuous integration and continuous delivery scenarios as well. Of course that drawbacks, but essentially if you do nothing today, if you just check it in and everything works, you should do something or you should try to convince your fellow colleagues that you need to do something.
35:02
What you can do, and I'm turning 180 degrees, if you can create a source repository which you can lock down and lock down good. That means you know exactly who has access to it and you can prevent anybody from peeking into it.
35:20
You can create a source repository. Otherwise you don't. And create your secrets in whatever format that you like. Plain text is fine. You can encrypt them if you want, but how do you protect the keys used for encryption? If they are available on the server, if you have an encrypted file and the key right next to it
35:41
with a plain text password, why encrypt it? But if you have a source repository, that's fine. If you don't, then don't. You can just have a shared folder and add permissions on that share
36:00
and then you know who has access to it. So you do that, you create your secrets in a file share somewhere. You might want to invent a convention for your files. This is just out of my head. Something like that. And then from that, you push your files to your application servers.
36:20
This is very low key. This will work for startups, small companies. It's not enterpricy. It has drawbacks. How do your copy function? Does it copy over a secure line, for instance? Do you open up for man-in-the-middle attacks, etc., etc., etc.
36:40
But in terms of governance, this is much, much better than having nothing at all. And then you need to refactor your source code to look for and expect to find a secret file at your desired location.
37:00
And for the love of God, do not put all secrets in one file because you will, when you copy that file and if you break the schema or something happens and your code is unable to read that file and every secret is in one file, you just obliterated all your applications. I've been there.
37:21
It's not fun. So don't do that. By creating and maintaining some sort of convention towards the products and the things that you want to store secrets for, you're in a much better place. And it scales immensely well as well. Why do we want to provide secrets for applications who don't need them, for instance?
37:43
Why do they need to get access and be aware of secrets they are never going to use? So split it up, separate files. This is a connection string for just an example.
38:01
There's a user ID and a password in there. It's a regular SQL user account. And it is eternally... There are a number of drawbacks that I'd like to show you because when you see that at first,
38:22
it's not a problem, you think. But what happens if you have a data leak? If you leak data and you want to roll your passwords? What happens is that you are unable to roll your password. You have locked yourself down entirely. Because that username and that password has been distributed
38:43
across 50 applications. So you need to change 50 applications in order to change your password. And that's hideous and not very fun from a security perspective. You essentially rule out these two use cases. You are unable to solve them if you provide your secrets to your applications
39:03
in terms of connection strings like that. You are unable to set up automation in your database, for instance, because your configuration file needs secrets. And you are unable to roll passwords for applications and databases. If you have security procedures and audits,
39:23
stuff like that, they might propose that you roll passwords on a schedule every three months, perhaps, 90 days. And that's a good practice. And that enforces better code. Better in terms of narrowing down the attack vectors
39:41
and your ability to handle accordingly if you leak data. So, how to solve it? You need an API. Some users are sitting to the left. They talk and interact with an API
40:02
in order to gain access to their secrets. And you have audibility and everything. You are in a database somewhere. This applies to many systems. There are third-party vendors out there. We have one at Lego, where we store our secrets. Active Directory fits nicely into this image as well.
40:23
You can interact with Active Directory through PowerShell. That's your API. So you can create users and get users and roll passwords and whatever. One thing you have to keep in mind, this is the very abstract image of what you need to do. You need to interact with an API.
40:41
What you need to be sure of is that if you have an API and you provide your secrets, you want to make sure that the secrets and audit database is sure. Because if you are compromised, what would prevent a hacker from changing your audit logs as well if your database is not secure?
41:01
I would do that if I was a hacker. I would take all your secrets and delete every trace of me ever being there. But that's just me. That's what I would do. So you need to have an audibility on the API and on the data that you're trying to store.
41:21
Please don't roll your own unless you're working for a company who are providing that exact functionality. Hackers are so clever. Unless you're really, really, really good at security, this is Bioware, basically. That's what I'm trying to say.
41:40
Don't roll your own. And most of you, this is a, you're mostly founded in a Microsoft.net world, so you have Active Directory. It works very well, very well. Go in that direction before going somewhere else.
42:01
So we have a developer. He wants to create a database. Oh, he's part of a build and deployment pipeline. Maybe he's deploying a website for the very first time. So he built it, the build is green, a deployment kicks off,
42:22
and somewhere along the line, the build pipeline figures that it has no database. So we need to create a database. How do we do that? How do we shield off secrets from the developer? He doesn't have to provide, he shouldn't provide any secrets at all, to create a database and provide credentials for login,
42:44
authentication, authorization, for that database without the developer being aware. How do we do that with an API? Well, first we create some secrets. The build and deployment pipeline needs to interact with the API and it needs to have proper authorization
43:04
to be able to create secrets as well. How your API gets access to creating secrets, that's entirely up to you. But it's an important factor to keep in mind that somewhere along the line, you need to build in trust in your build and deployment pipeline
43:21
in order for it to interact very intimately with your secret store. That's a feature in itself. How do you narrow down the attack vectors? How do you not open up your build and deployment pipeline for intrusions and, how to explain it,
43:44
how do you ensure that your build and deployment pipeline is not open for attacks because it is able to interact with your secret store? Keep that in mind. So, oh yeah, I just put up post
44:04
and there's a convention based and I'm creating an administrative user. That's the example I'm trying to provide. So it's created. A database is created from a vanilla database
44:21
or an image or something like that. And then the build pipeline applies authorization for that admin user. It gets the user just being created. The developer doesn't do anything. And then a script, you can provide,
44:42
the build pipeline provides a script, fires a script against the new database being created, the DB foo, with a user and a password that the developer is completely unaware of. That's how you do it. This is an SQL user. If it was an Active Directory user, the code would be different, bear in mind.
45:03
But this is just to give you an example, a hint on how you shield off your developer from getting to know secrets he should be totally unaware of and how to automate your pipeline and still being able to create databases and stuff that needs secrets
45:21
but still maintaining that you want 100% automation and you want to shield off developers from stuff that he shouldn't know anything about to protect him as well as you. And the build pipeline goes on. Secrets in various environments, they differ.
45:42
Secrets in dev and QA and live, whatever environment you have, they might change. They should be different. And this pipeline, the example here, applies or enables different environments to have different secrets.
46:03
So, that's one use case, automation. That was the first one from the slide, if you slide back. Another one is, which is very common in what we are trying to accomplish at Lego, is that a developer needs access to a secret during his deployment.
46:21
We don't want that in source control, but he needs it. It could be a certificate for AWS deployment. It could be a username password for creating a website. It could be, well, find your own example. It could be anything, but he needs some, we need to provide him with secrets in a secure fashion,
46:43
and we don't want them in source control. He's not supposed to check in those secrets into source control. How do you do it? Use an API. Any questions? No. What we have is that we have a system administrator. It could be the developer.
47:01
It might not. At Lego, we have a different department. I'm with Digital Solutions, and somewhere over there, in a completely different part of the organization, somebody is paying for our AWS accounts, for instance. Just an example. He needs to create secrets, which effectively is being paid
47:22
with a certain cost sensor somewhere, so it ends up on the correct account. It could be that system administrator who creates the secrets for the build and deployment pipeline. Or it could be the developer just tabbing it into a system somewhere else. It's a different rule.
47:41
We need to distinguish the two. So, secrets are being created and made available for the build and deployment pipeline. Then, along the way, we made a feature which unlocks those secrets during the deployment pipeline, and we made what I later came to know.
48:01
Essentially, we just built it, and somebody pointed out that we made a template pattern. I'm not familiar with patterns. Maybe 15 years ago when I was studying, but this is a template pattern. We housekeep. We're doing a lot of housekeeping and firing off a script that the developer owns. They are the proud owners of their own deployments.
48:23
They just need to respect a certain contract that our deployment pipeline enforces, and one of them is that we have a parameter where we store, where they get access to unencrypted secrets. Yes, unencrypted, because they need them unencrypted in order to, for instance,
48:42
create a new website using PowerShell. They have a convention for getting access to that secret. They know if they obey that convention, they can get that specific secret from this list, this item, I want the username,
49:00
or the password, or whatever. So, by obeying the convention, they can get access to secrets. So, we provide them with a PowerShell object, which is for them to use, and this is enabling them to be creative. They can request somebody to create them with,
49:24
to create secrets for them, or they can create them themselves, and they can write their own deployment script. If they want files deployed differently in different environments, go do. They don't have to interact with us. They just write code and make it their own. It doesn't happen, but we still have governance.
49:40
We are able to roll that password if we want to. It's immensely strong. I think I'm quite happy about it, actually. So, right, final example.
50:01
This is a part of our deployment pipeline, and this is an example of what happened when we performed a security audit on the template pattern we just implemented here. We have people in Lego who knows immensely much about security, and thank God for them.
50:22
They are very helpful in terms of nailing down the weakest part of the link in the things that we do, because we're developers. We think in sunshine scenarios, and when things are working, hey, we're happy. And then comes a security audit,
50:40
and user stories start to emerge. What we have here is that we have a step where we extract keys from our secure store. That's the chain lock which is being unlocked on this image. That's this one. And then we have the deployment script being invoked at that step.
51:06
And then he started our security auditor. He asked, what happens in the middle? How does secrets get from the step above to the deployment script in the button? Oh, well, we just store it as a property, and boom.
51:25
We just have unencrypted secrets in our log files in every other step between those two arrows, available for everybody with read access to that deployment pipeline to use and see, and we're screwed.
51:41
So, what we had to do and we never thought about it. We just, hey, it works for us. Our users are happy, they can use it, but we weren't finished. And it's important to reiterate
52:01
and think in non-sunshine scenarios. Have people help you uncover that. You don't open up for, you can do so much, but if we had left unencrypted secrets in our deployment pipelines and in our log files,
52:21
we hadn't accomplished anything. And this is a level of security we need when being a company like Lego, because if we have too many of those, essentially someday we've lost, and I think we'll get compromised. So we need to keep a high standard. What we had to do here was to encrypt
52:40
the keys that we just extracted with a credential that nobody is aware of. That's a functionality embedded in the workflow engine, which is deploying code. It can create a credential for you that you're totally unaware of. So we can use that to encrypt the secrets we just extracted and decrypt it in the deployment step.
53:04
So even us, the administrators of our deployment pipeline, cannot see, we can just see a base 64 encoded string, that's all. That's how we solved that particular problem. And that's the standard that you need to have in order to ensure and show to the world
53:21
that we're doing our very best, our utmost experienced people are working on ensuring your kids' data within our systems. Final note, I think it's quite funny, but you can over engineer so much
53:42
with security, and you need to challenge assumptions all the time. That's basically what I'm trying to say with that image, because you can do security within a year, but try to keep in mind which walls you're trying to build. And this one poses, again, you don't do that, ever.
54:07
I will find you and hunt you and kill you with a spoon if you do, slowly. So, please don't. I'm quite aware that many developers at Lego will probably watch this, so I know who you are.
54:21
I can find you. You own the deploy script, so I can see you checking that. Yes. All right. Security is all about removing harm and ensuring trust. That's what it is. If you ever need to explain what security is, use those terms and elaborate upon it.
54:43
You should build walls the right size, in the right place, and have a discussion about where to build the walls. How high should they be? How wide should they be? Do you need to build in cannons and guns and crocodiles and stuff?
55:02
I haven't touched much upon auditing, but if you can show how people interact in your systems, who is accessing secrets, how they're being accessed, which systems have access to which parts of your system, you're in a very good place.
55:21
I have an example of, Michael is probably smiling now, where I rolled a password for a worker account, which effectively shut down our pipeline. It took my team 30 seconds to look into the logs and see that I was the one who at last made the update, and yes, I had to buy cake in that respect.
55:44
But it's immensely important that you think in order to build your capability and visualize and make it transparent who's accessing secrets, both systems as well as people. Who is reading secrets is equally important as who's updating them.
56:03
Never underestimate peer pressure. I will never forget to double check and triple check that I've got the right account the next time I roll something, for sure. Get your stakeholders on board. Try to create a shared language for a company.
56:21
It's agile manifesto, it's not banalities, but it's been around for years, for decades. It's important. Don't forget. If you start a project, start with a security feature, because walking skeletons win. By starting with security, you will be forced
56:40
to start integrating stuff together. If you have a totally greenfield application, you should kick off and start with the login function, for instance. Or whatever function interacts with various parts of your system where security is involved. If you need to exchange certificates, start with that. As well as if you're in doubt, do a security feature.
57:04
It will uncover where to build your walls, and proof of concepts will test if you're building your walls in the right fashion. Eventually, you'll get a good dialogue kicked off by doing the security features first, if you don't know where to start.
57:25
Penetration tests provide great value. Just bear in mind that they should be the kickstarter of a dialogue. It's not just we have so many errors, 72 errors, go fix, now we're safe.
57:43
That doesn't work that way, then you're doing it wrong. But they will find something all the time. These companies, that's what they do for a living. Trying to penetrate systems and show you a report, and they're very, very good at it. You can buy systems for a few hundred dollars
58:03
and just run them internally, and that's a good place to start. You don't have to get people to do it for you. You can get off pretty far by doing it yourself. Getting somebody from the outside to do it, they will have a totally different perspective of your infrastructure than you have, because you're cluttered in it and work with it every day,
58:23
so you don't see the weak points the way that an eventual attacker will do it. Symbol is often more secure, and if you're in doubt, if you make things too complex, too hard to understand,
58:40
you're also probably opening up attack vectors that you're unaware of. So the symbol solution for distributing secrets, for instance, is better, even though it has numerous drawbacks, it's better than doing nothing, and you can over-complify things in a way that you're actually better off not doing anything at all.
59:06
That's all I have for you. Please remember to vote. I appreciate the feedback, and I know the conference does as well, so come to me if you have questions. Thank you.