All right, well, let's get started. If a couple of people come late, that's OK. Welcome. Thanks for coming to my talk. This is pull requests, not just for code anymore. My name is Tim Kreitzer. You probably figured out by now that I'm not, in fact, German.

I'm American. I'm from Portland, Oregon. You may have heard of our trail. I think it's kind of famous. At least it is in the US, maybe not here. Oregon Trail, computer game, anyone? No? Eh, a little bit. OK. In Portland, I work for New Relic. We do application performance management and software analytics, lots of data about how applications are working,

web applications, mobile applications, that sort of thing. Specifically, I am a manager for the New Relic RPM team, the Ruby agent. So if you're a Ruby developer and you want to talk performance, I'd be happy to do that. I also have some New Relic stickers. You're welcome to come up after the talk and get one. Whether or not you want to have a conversation,

that's OK. You can just ask for a sticker. That's fine. It's very trendy these days to put your Twitter handle on your slides. I want to make sure and do it very subtly so you won't have to worry too much about it. But I treat going to conferences as a very cost-inefficient way to buy Twitter followers. So maybe you can follow me and we can justify the cost a little bit.

So this talk starts off with a story. And I hope this story translates to the culture here. It certainly is a common story in American companies. So you're sitting at your desk Monday morning, have your coffee, opening up your email because it's Monday at 8, 9 AM, but already you have email waiting for you.

And you get a message like this. Hello, everyone. Effective immediately, all expense reports will need to be printed and have physical paper receipts attached when filed. Thank you, accounting manager. And you're thinking to yourself, this is another change to our process. It doesn't make a whole lot of sense. It seems to add a lot of inefficiency. Why are we moving paper around? There's no explanation. There's no discussion.

So you think, OK, fine. And about 30 minutes later, you get another email. To respond to the 50 people who replied to my earlier message, the reason I need the receipts is to verify that the expense totals match. This is important to make sure our financial statements are correct. OK, fine. No problem. 30 minutes later, you get another email.

Some people have asked if they can send scans of receipts and submit the report electronically. We don't have any way of doing that, and it probably wouldn't pass audit, so please print them. And this is where you start introducing the head to the desk. The fact is this is very common. This happens all the time for all sorts of reasons. Companies, organizations, even open source projects,

messages like this come from accounting, from your office manager or assistant, from your CTO, from the project maintainer, whatever. It's common, but I think it's really, really bad for a couple reasons. It's a bad kind of surprise. You don't know some of this is coming. This isn't like, oh, this is great news. No, this is just more inefficiency, more work.

There's no explanation. There's no reason. It seems vindictive. A good example of this is like a company dress code. It's very, very common, at least in the US, maybe here, too, for companies to declare a company-wide dress code instead of just talking privately to the one person who needs to change how they dress in the office. So for everyone else, it's just unnecessary and annoying.

There's no ownership. There's no transparency. There's no insight as to why actions are being taken, what alternatives we're considering, that sort of thing. So I think there's a couple of reasons why this happens. There's been a lot of research on how small groups operate. You're probably familiar with Robin Dunbar, Dunbar's number.

Wrote a paper in 1992 while at University College London, and he described the limit in the average primate group size caused by the size of the neocortex in the brain. So he came up with this number. And what we've come to understand is that humans can't associate closely with more than Dunbar's number of 150 people.

When an organization or a company or even an open source project gets more than 150 members, the depth of your social relationships will drop. This is when you start feeling like, whoa, this is a big group now. I don't even know who any of these people are. In groups over this size limit, the sort of bureaucratic top-down process will grow and happen more.

The other reason this happens, like I said earlier, direct follow-up is hard. Big rules are easy. It's much harder to have an uncomfortable conversation with someone than it is to just send out an email to everyone and not have to deal with problems directly. But really, I think the reason why this happens is because you allow it to happen. So we're going to talk a little bit about how

to make that not happen. This is what I think of when I think of company or organization or open source culture. And culture, I think, is really important, actually. Jez Humble, VP at Chef, formerly Opscode, says it's organizational culture that makes productive systems, not TEDx developers.

Bad culture ruins good people. Say that again. Bad culture ruins good people. And I think culture is really important. I divided it into two separate categories. There's explicit culture and implicit culture. Explicit culture are things that are written down, things that everyone can understand because it's

written down, it's clear, it's well-documented. Explicit is way more important than implicit. And implicit culture is the things that just happen. You don't know what is going to happen, it just shows up. One of the most important things I hope you take away from this talk is that explicit culture is absolutely more powerful than implicit culture.

Explicit culture gives you clear expectations. If things are written down and clear, people know what consequences their actions might have, good or bad. Explicit culture gives you faster onboarding. Implicit culture takes a really long time to learn. Often, you won't discover a detail of the culture until you make a mistake or you do the thing that you weren't supposed to do.

With explicit culture, you can learn ahead of time what you should and shouldn't be doing and avoid those mistakes. I also think explicit culture gives you much greater accountability. If you share what the standards are and document them, you can allow everyone to hold each other accountable to those standards. So this is really, really impossible with an implicit culture. If nothing's written down and there's no standard,

you can't have any sort of accountability. I also think with explicit culture, you have an opportunity to show your work. You can document thought process, history, that explains the way things are. There's a difference between here's what we do and why, versus it's always been this way, no one knows why, it just is.

So let's talk a little bit about how to build explicit culture. Another quote I love, the biggest way you can build this explicit culture at your company or organization project, reinforce the value of nonproductive activities that enhance culture. Stop treating these activities as nonproductive, because they are very, very, very productive.

It may not be writing code, but it's really important. You come up with these artifacts when you're talking about explicit culture. This can be the corporate stuff, mission statements, corporate values, things like that. These may or may not actually be reflected in the day-to-day operations of the company. But it can also be a lot more interesting than that. I think you can do things like Valve's new handbook

for new employees. Have you guys seen this? Has anyone paged through this? Yeah. It's a very interesting and entertaining read, even if you don't work there. And it's a really, really powerful recruiting tool, because this has gotten out to the public, and people read this and say, whoa, I want to work for this company. Another example, Netflix's culture slide deck. This was widely circulated a few years ago.

This deck was kind of aspirational for Netflix. They weren't quite where they said they wanted to be, but it made a clear message of who they wanted to be, internally and externally. Hopefully you agree with me by now, explicit culture is really important. Let's talk a little bit about how to build it. Hopefully, you don't build it like this,

with a committee full of dozens of people meeting every week or every two weeks to do things. I think this is really terrible. Please don't do this. It's very, very cumbersome. It's very slow. It consumes a massive amount of time, and you're probably going to get the wrong people in the room. You probably won't have people with the proper authority to decide

what the culture should be. So everyone wastes their time producing a bunch of stuff that's not actually followed. Now, a lot of companies use Wiki for this sort of thing. You might have an internal Wiki where you can write things. And Wiki is OK. It's not good, it's not bad. I think there's two big problems with Wiki. One is that it requires boldness for editors.

When you make an edit to a Wiki, it's usually live right away, which means you have to be pretty confident that your edit is correct. This sort of culture stuff, it's not always easy to get right the first time. It's better to have more discussion. It's really hard to have a productive and well-contained discussion on Wikis. Now, of course, you know from the title of this talk

where this is going. We figured this out a long time ago in open source. If you keep your processes and culture documentation in markdown files or HTML or something like that in GitHub repo, you can leverage some really, really helpful features using pull requests. Pull requests do some really pretty great things, actually.

You have a nice system for notifications. People can watch the repo for notifications on every pull request if they really care about company culture, or they can just subscribe to individual things if they don't care about everything. You can also tag people who you know will want to give input on something. If you know that Bob really cares about this topic, tag him in so he can see the process happening.

You have integrated support for tracking a conversation, including automatically hiding out notes on out-of-date content. This is incredibly helpful for discussing process documentation because anything that's changed gets hidden right away. If it's a discussion on something that you've already moved past, gone.

You can also use GitHub's built-in labels feature for workflow management. I'll show an example of that later. And assign PRs to people using the Assigns field to make it clear who owns a proposed change, who's actually responsible for reviewing this and changing it. This really gets awesome if you use GitHub Pages, because then you can turn your markdown files into an actual static website that can be browsed easily

by anybody, whether or not they understand how GitHub works. If using GitHub Enterprise, you can even do this behind a firewall and protect your content. This has a really great number of advantages over publishing the content in a CMS or by pushing static files yourselves. You don't have any servers to manage. You don't have to wait for someone to come around and deploy your new code.

And there's no CMS to keep up to date with security patches to keep out spammers and hackers. So this isn't just a theoretical idea for me. We're using this exact model internally at New Relic for several really, really important pieces of our explicit company culture. First off, there's what we call our processes pages.

These are our attempt to comprehensively document what it means to work in engineering at New Relic. The full technical details of every single job isn't in here. Not yet anyway. But we have information on our overall development processes, details like expense reports and hiring and taking time

off, and even a collection of random notes that we think are important details of our company culture, like the fact that you're not allowed to eat at your desk. Before this was around, you would learn that rule by breaking it, and someone would code and tell you, hey, don't eat at your desk. Now we have it written down. New people can read it right away and understand. The processes pages have been such a success,

we also started using GitHub pages and pull requests to manage our overall architecture of the company, of the software architecture. We've adopted a system of numbered architecture notes, very similar to RFC process, which you might be familiar with. This has worked really, really well. We have dozens of architecture notes now. This is just a few of them. This page goes down, down, down, down, down.

We've also started using markdown files and pull requests to track the specs of how all seven of our language agents, we have agents that run in Ruby and PHP and Java and .NET, all of those, plus of our server monitoring agent, we've specified how they should behave, how they format data,

how they interpret configuration values, and so on. This repo doesn't have GitHub pages, because we know only software engineers are looking at it, the people who are actually working on the agents. But we do make extensive use of labels, as you can see here, to track which teams have approved specs so we know when they're ready to merge. Teams are responsible for coming in and putting the label on the PR when they've read it and approved it.

And if the content changes, they can take the label off again until they've had a chance to review it. So we're a company full of data nerds. Here's some data on how it's going. Since July 2014, so about 13 full months, 214 pull requests in our processes repo from 48 different people. The most comments we had on a single PR

was 40, which is not bad. Architecture has only been around for five months, have 137 pull requests from 32 contributors, and we had 129 comments on a single PR. If you're wondering what 129 comments looks like, it looks like this. And this is even hiding the ones that were on out-of-date content.

If you're wondering what we were talking about, it's to discuss the list of approved programming languages for new projects. Not surprisingly, people have a lot of feelings about this topic. So I want to wrap up real quick with a formal presentation, then I'll have a lot of time for questions

because I know people might have some, and I'm happy to take them. I want to share some of the things we've learned by doing this as a company. Maybe you can take this process back to your company, organization, or project, and implement it too. The first thing we learned is that Git can be really hard for non-technical people. If you're not familiar with this, this is the Git random man page generator.

It just spews out a bunch of words that sound sort of like Git documentation. So you can use this command to gain any non-specified downstream changes over the fast exported non-reset unstaged stashes. Obviously. Now, of course, if you're familiar with Git, you can recognize this is nonsense. It makes no sense. But if you're not familiar with Git,

this looks an awful lot like the real documentation. It's very, very complicated. The second lesson we learned is that developers love to bike shed. Is this a common term here? You guys know bike shed? Yeah, a couple, OK. So it's from a British scholar of public administration. He noticed that a committee was formed to approve plans for a nuclear power plant, and they spent the majority

of their time discussing the materials for the staff bike shed, rather than discussing how the plant should actually be designed, because everyone can understand how to build a bike shed, but nuclear power plants are actually kind of hard, and big, and complicated, and they're hard to discuss. The other thing we learned is that it can generate a lot of email spam. So you need to be prepared

with people to do filtering, and rules, and I talked a little bit earlier about all the subscription options, and managing that, and being thoughtful of people's setup. It goes a long way to get buy-in on this process. I also think it's really, really important to encourage action. When you first implement this process, people are going to comment on things.

They're gonna get excited. They're gonna get, oh, this is a pull request I care about. I'm gonna go comment. What's better than a comment is a pull request. Instead of saying, I think it should be blah, blah, blah, have somebody come in and say, what's your actual idea? Write it up in a pull request so we can talk about it there. This takes a lot of careful practice and encouragement to people to say, it's okay.

Open a pull request. We'll talk about it, rather than just making suggestions or even pointing out typos. You don't need to point them out. Just fix it. With merge rights comes merge responsibility. PR sitting open for weeks on end will kill any enthusiasm people have around this work very quickly. It's natural to want your leadership to be the one who are in charge of merging things.

They need to make time to actually do it, and if things just sit for days and days and days, people will lose interest really, really fast. I also think it's important to make clear that it's not a democracy, but open source isn't either, so that's okay. We noticed quickly that this process, much like most open source projects, moves best when it's not a democracy.

Build shared motivation and input and trust from people and build consensus, but at the end of the day, the majority opinion doesn't always win, and that's okay. That's all right. I think the bonsai tree is a really good metaphor for how this should work. You have one or maybe two, three, four caretakers whose job it is to let things grow organically,

but once in a while, there's a bit of careful, delicate trimming to be done, redirecting an unproductive conversation, making concrete decisions on areas where there isn't consensus and that sort of thing. It's also important, I think, to deal with entropy in your processes. Things get old and out of date

really, really, really quickly. You should designate one or more people as entropy fighters and empower them and give them the job of going through and updating or removing things that are out of date on a regular basis to keep things fresh. Otherwise, it just sits and it gets old and it gets out of date. So, we're not the only ones who have figured out this process and are following it.

Even the United States government, which is well known for being a wretched hive of bureaucracy, is onto this. This is the FITARA project, the Federal Information Technology Acquisition Reform Act, which was passed by Congress and is essentially guidance for the US federal government of how IT projects should be run. Can you imagine a more concrete example

of bureaucracy and that sort of thing? But, as it turns out, they take pull requests. A US representative from Virginia opened a pull request, a congressman opened a pull request against this repo and had some warning changes. It wasn't a huge pull request, but it did have some interesting ramifications. I think this is awesome.

I think this is really, really, really cool. I hope it spreads throughout the rest of how the government does its work. So, let's go back to the story at the very beginning of my talk and see how it could have gone a bit differently if we'd had this pull request model in place. Our accounting manager opens a pull request against our processes repo. Already, this is a little bit better.

We have a why attached to this from the first moment to help explain the change. We're concerned that our previous process won't pass audits. So, now we're requiring paper receipts to be submitted along with paper expense reports. There's an immediate opportunity for comment. Someone asks if we can ask IT to add support for our existing electronic system for receipt images.

The accounting manager is able to respond inline to say a scanned copy of receipt is fine. They update the contents of the pull request with a new commit and get it merged. This is how I think it should work all the time. That's what I have for you today. I'm happy to take any questions if anyone has some. I have a lot of time, so feel free.

Yes? If it's not a democracy, what kind of form is it? Well, I think the democracy of the employees of a company might want something to be a certain way. But the leadership might say, you know,

that's not actually the direction that we have. We want to go this way. And it's a little bit much to perhaps say democracy doesn't win. But just because 40 people want it to be this way and 38 want it to be that way doesn't mean it has to be this way. There's still a decision to be made. It's very much similar to open source, although in open source you can fork and do things like that.

Yeah. Let me show you.

I can show you real quick. So for the purposes of course, of course. Well, let's see if we can turn it on. But if not, I have another idea.

The answer really is actually very simple. It's just markdown files, so text files that have the content in it. Markdown is a very easy syntax to learn. Anyone who's non-technical can understand very quickly how to mark things up, make lists, links, that sort of thing. We use HTML for things that need it. But this is my page.

So this is my little sample. You can't see it so well. I'll try making it bigger in just a second. So this is the, I actually wrote this out for the purposes of doing my demo. It can be that simple.

The markdown doesn't have to be very complicated. For pages that need it, we use HTML when we need big tables and that sort of thing. But markdown is actually really, really great. Yeah, ODF, yeah. It's a special version of ODF.

It's only one file, and if you put it in a node and get the code, it can also store something like that.

Yeah, we haven't had a need, actually, to store that many documents. Everything gets translated into either HTML or markdown. That way we get nice diffs and that sort of thing. Other questions? Do I see a hand over here? Yeah.

Yeah.

Yeah. Yeah, that's a great point. I'll repeat for the recording. Dealing with colleagues who purposefully hide their work in progress until it's done so that the peanut gallery doesn't weigh in constantly. We've had that issue a couple times. People get in early and want to work on things.

But we've also found that sometimes the first draft is authored by a small group of three or four people who come together for an afternoon and write something and put it together. That way you get at least some of the initial round of peanut gallery out of the way. I wish there was a better way to do attribution for multiple authors so that they would get credit for their work.

Someone has to be the person who made the commit, unfortunately. But we've used that a couple times. For things that are very complicated, bring a group together just for a short amount of time, come up with the first draft, put it up, and then go from there. And I think that's an OK way to do it. Yeah. Does this process scale down?

I think so. The agent specs example that I showed, our agent specs, the number of developers who really weigh in on that is around 10. So I can't say that I've seen it work firsthand for less than 10. But I think for a group that small, we've been able to use it really well. We used to do Wiki for our specs.

But it got very, very confusing very quickly because it was difficult to keep the old revisions tagged. I mean, yeah, of course, Wiki has a full history. But which is the exact version that was implemented by this agent at this time? Using the RFC numbering system, we were able to say, OK, we're implementing spec 18. And maybe spec 18 gets replaced later with spec 25.

But you know what 18 was, and you have that link. And you say, OK, this version was on 18, and this version was on 25. And I think that's important. I think it can scale, though. I don't think there's any reason why not. It's just like code review, I think. Yeah, go ahead.

Would it work for things like UI design? That I'm not sure about. I think it would be difficult without finding a format that you could use that's easily comprehensible. I mean, I guess you could put PSDs. But any time you're putting binaries in Git, it's not very good. So I think if you did some work

and you thought about how you were doing your UIs, I mean, maybe it's just because I've been computing for a long time. And I remember the good old days of BBSs. But I love ASCII art. Do your UIs in ASCII art for a little bit until you get down the road a little bit. Maybe you can prototype that way and do diffs that way. Yeah.

Sure. So using a tool to generate the UI from a text file or a language or something like that. Cool. I like it. Yeah.

Why do we use GitHub and not our own GitLab server? That's a very good question. I'm actually not sure of the reason. We use GitHub Enterprise at New Relic. So we have an internally hosted version. I suspect it's mostly just for familiarity. GitLab is not very popular among US companies,

whereas everyone knows how to use GitHub. So I think there's just a little bit less transition cost of getting on board. That's a theory, though. I'm not sure. Yeah. Yeah.

Yeah, I think there's an advantage for us. Our competitive advantage is not in our source control. It's in our software that we write. So letting someone else worry about that. And they provide the patches. And all we have to do is provide a server. And it's taken care of. I think there's an advantage there. Yeah.

Yeah. I know GitLab does pull requests. Bitbucket does pull requests. I think most of the Git repositories these days will support this kind of model, maybe with different features, but it's pretty similar. Yeah.

How do we get non-technical people to understand Git? That's a topic of a whole other talk. But I think there's two ways.

One is pairing, working with someone who wants to make a change and giving them some education. If you teach people, they're usually willing to learn. There's a lot of great clients out there. The other way is actually GitHub and GitHub Enterprise has a built-in editor. So you can click Edit on most files and get an editor. And if you're using Markdown, there's no code.

I mean, it's just the text. And that walks you through the process of pushing a commit, and opening a branch, and making a fork, and all that stuff. So if you show someone, OK, we're going to have to learn a little bit here, because you have to learn the GitHub interface. But then when you click Edit, OK, editing a file, that makes sense. And then when you hit Save, there's all these weird terms I don't understand at the bottom. But I just, OK, save, let's go.

And it works. So we've had some success with that. I wouldn't say it's perfect, though. But I do feel very strongly that setting up GitHub pages so that things are served in a way that's attractive and people can just browse to a URL is really, really important. Because even if people aren't willing to edit it, they can at least find it and view it easily,

assuming you have non-technical people reading it. If it's just engineers, then you don't need it. So, yeah. Right.

Yeah. It's at the boundary of what they're willing to do, but it's working OK. So before my current job, I worked running a Drupal development team. So I've trained many, many, many people on how to edit Drupal documents. I don't think it's that much more complicated than Drupal is.

I'm sorry, say it again. Text, yeah. And once you get to the text, everything is good from there. So the question is, how hard was it to convince the bosses

to change the workflow? And the answer is, it actually came from the bosses. So it's very easy. But our SVP of engineering was the gardener of the repo. He has the merge rights. He decides what makes it in, what doesn't. And this is something I'm talking about, but it was his idea, so I should give credit.

Bjorn Freeman-Benson came up with this model and really pushed it hard. But he's also been able to step out of the way, too. Like the agent specs example, he doesn't have anything to do with that. That's all our team, and we just do that ourselves. Get my time here. Hello. Yeah, another five minutes if anyone has questions.

Yes, up front. Yeah. How do we decide who gets the merge rights? That's a good question, and it depends on the project that you're talking about. For our processes, that goes through our VPs. Because if something's going to change to our process,

they're the ones who merge it. Now, that said, they merge most of what comes through that has consensus. But once in a while, something gets kicked out. For the architecture notes, we have an architecture team of principal engineers who share that responsibility. And each one of them will kind of captain a pull request and say, OK, I'm responsible for this area or that area or this area.

For the agent specs, there really isn't anyone. We have a director of engineering, but she doesn't really worry as much about how specs work. It's more peer-based. So what we've come up with there is we have a time-based system. When I open a PR against an agent spec, I say, I'm going to merge this in three days, unless anybody has a question. And I'll extend the time happily.

But if nobody has any feedback, it's getting merged. That's what's happening. So we bias towards merging and moving forward rather than waiting for comment. Question back there? Yeah, so you mean prior to the system?

Yeah, so the processes used to be a Word doc that was the employee handbook. Had a lot of stuff that's not very pertinent to engineering. And it was missing a lot of things that we had. And I think that's pretty common for most companies.

For specs and technical documentation, we had a Confluence wiki, a Atlassian Confluence. And we still use that for a lot of things. But for things like specifications, like I said earlier, wiki is not great. If you make a change, it's done. The change is done. There's no period for comment or questions or anything like that. It's just there. So you have to do a lot more work offline

to socialize your changes and make sure everything's good, whereas with this system, you just tag the people that you need. All right, thank you very much. That's what I have for you. Happy to talk about this more if you want to come up or get a sticker as well. Thank you. Thank you.