Okay, hello everybody. Welcome to FOSTA in 2016. We are in the room with Vincent. In a few minutes, or just in half a minute maybe, we'll give a talk in water. We hope to bring you to Microsoft and to source the projects that are in common.

We're here at dental function. He's working as a professor of technology. He's researching there. Well, let's begin. Thank you very much.

It's closer to you. I'm very excited to be here. That's a picture I took back in 2009 when I was in the office, as you are now.

And I really loved doing FOSTA. So for me it's super exciting to be here and talk to you about what I'm doing, my research, and things that I hope you will find very interesting. So just to give you a bit of background about myself. I am Italian, as you can probably guess from my accent.

I started in Bologna where I'm a Bachelor and Master and I have quite a few friends here. We meet together at FOSTA every year. And then I spent one year in Brussels, USA, so FOSTA for the first time, back in 2007. I worked as a software developer after my Master's for one year in Italy,

and then I started to teach software engineers. Being a software engineer, I realized how difficult it is to write good software. So I said maybe I can help the developers become more productive. During that period I had the chance to visit Microsoft Research twice.

And now what I'm telling you today comes from experience I had there. And apparently I'm an assistant professor at the University of Technology in the Netherlands. And there, besides teaching, I do research on software engineers.

How to help developers become more productive. Okay, and that's it about myself. So today I'm going to talk to you about modern code reviews. And this more common tells you everything about what modern code reviews are. So they are informal.

There is no meeting that you have to attend with many people together at the same time. They are tool-based. And they are synchronous. These are the three main characteristics of modern code reviews. As opposed to code descriptions. So especially the most interesting thing which makes being informal and synchronous possible

are the tools that are used for code reviews. You may know for example Garrett. Many of you probably also know GitHub for requests. They are becoming more and more popular every day. There are some proprietary tools like Atlassia.

And there is also an internal tool developed at Microsoft called Microsoft Codeflow. And it's this one. As an example, the idea is that all the tools, they work basically the same. They offer the same functionalities. So you have some code that you have changed.

And you want other people to review this code. You prepare a package, either to a pull request or creating a package separate from the version of the system as it happens here. And then some of these go to some reviewers. And then they have the chance to look at the changes using a normal action leaf

to see whether these changes that you created are correct. And then they can, for example, put some problems and say this is good, this is bad, you should change this and so on and so forth. This, for example, you can have some logs of what's going on. And then the thing is that you can have multiple iterations.

So you can submit some code for review. Reviewers can say, well, you should change these things and you'll iterate over this whole thing, either the reviewers decide to not merge the code that you are proposing or to merge it with a code base.

Okay, so I would like this small talk to be pretty much interactive. So I would kindly ask you to take three minutes, load this website, and fill the survey I prepared for you. And I promise we are going to see the results during the talk to see what you say about

this talk. If for any strange reason this link doesn't work, that's the Google form you need to write for.

I've seen that almost 80 of you replied, so thank you very much.

That's amazing. Okay, so I told you that I'm doing research on modern code review. So why are taxpayers paying me to do this? Well, I think there are two very important reasons why you want to do research on code

review. Nowadays especially with the popularity of GitHub and code requests, code reviews are becoming more and more popular, especially also in open source software systems. They have been popular with Linux and everything, but nowadays it's so easy to create code requests, each project can easily have code reviews.

And what happens when something becomes popular? Well, people start saying things about how something should be done in the right way, why it is useful, so on and so forth. But unfortunately there is not much research that tells us exactly what code reviews are

used for and why and how you should do them. The second point is that it seems there are many liters with modern code reviews and it's very important to understand that, to understand what the challenges are, what the outcome is, to see what can be improved. So these are the main two reasons why it's important in my opinion to care about

code reviews. So to focus on these two things, the first thing that we thought about doing was trying to see what happens when real developers, professional developers do code reviews.

Because for literature, what you have is literature on multiple discussions, something that was done around the 80s and 90s in which you had very formal meetings in which you would stop coding for like three days, send a code to the other team members, the team members, you check it and then you meet in a meeting and go live on your own.

One line by one line to find all the others and let go and once everyone is happy then this can be merged. This took forever and not many partners are doing this nowadays. But what we know about code instructions is that it engages developers and others in

a formal process that usually detects more defects in the product than does machine testing. So the question that one can ask is, can we trust these results also on modern code reviews that have been used in software code but are done completely differently? This is one of the things you want to test before going on and writing about code reviews.

So how can you empirically test what's going on? Well, one of the places in which you want to be if you do empirical research in software engineering is Microsoft Research. I know it's a strange place to say this thing but Microsoft Research is very open

about the studies they do. So this is one of the research subjects that really publishes in academic values the results of what they are doing. And these people are in tight connection with Microsoft product groups so you can really

talk, discuss, ask things to developers, analyse their data to ask the empirical question you want to ask them. So you went to Microsoft Research in summer 2013 and studied together with a researcher based in Europe. What happens, what are the practices of modern code review at Microsoft so that

we could understand them better as he was really working and what should be improved? How did we do it? Microsoft is very interesting as a place to do this kind of studies because it's all in the same place but you have very, very different groups doing completely

different things. So for example, you don't expect people in SQL Server to use the same ID that people use while you are in the Windows world or other applications. They have completely different ways of creating their products. But they have one thing in common, they all use the code flow review tool.

So all these different products and many others, they use this tool for record reviews. So this means that we can filter out problems due to different tools. They are all using the same tool.

If we see some results that are different from group to group, they are not due to the fact that they use different tools. This is a very interesting opportunity to do research about this. So what did we do? Well, when you want to do this kind of studies where you really don't know where you want to go, you do something called quantitative research and study without hypothesis and say,

okay, I want to see what people are telling me about this phenomenon. So we said, okay, now we want to go and serve developers where there are new contributes and afterwards interview them and ask questions about what they think and the challenges and so on.

So in this process, we start and that's the first part of the study. We start to serve 18 developers. So we have different roles that sign at least 50 reviews in the lab in the previous three months. So the idea was that the code flow tools has an analytics background that could tell us

whenever a new code review request was created. Once that was created, we will try to let developer and ask if they could go and serve that developer in action while doing the code review. So the Microsoft campus is really huge.

So I spend a lot of time on this shuttle at night. Every time a developer would say, yes, please come and see me when my code review will jump on this shuttle and go to the meeting. They have around 100 meetings. Okay. And then we got some findings from these interviews and observations.

And we wanted to see, the most important thing we wanted to see was, first of all, why do they do code reviews? We got some results from the interviews, but of course, we wanted to have more people telling us what they think about that. So we also sent a survey to managers asking them, why do you ask your developers

to do code reviews? And these are all the reasons why they asked them to do so. And these are exactly the same reasons that we found out in the survey that we just did. So to find alternative solutions, to improve the code, to find errors, so on and so forth.

Okay. So in this first place, we got the possible reasons. Then we extended this to developers. We sent a survey to around 2,000 developers, almost half of them replied, asking them exactly what you did to run these motivations for your code reviews by important,

according to their opinion. And these are the reasons. Why do Microsoft developers do code reviews? The first reason to do that for them is to find the difference. We sent it to managers, and 3,000 reasons for developers said that this is the main reason

they do code reviews. Okay. Let's see what you think is done.

If you get some help, it's not important. Okay. So I'm happy that you found your reviews important. And if you do that, it is super important.

Okay, so why? We needed to look at the top three reviews, and we put this data available. So according to your results, it seems that the top reason is to improve the code quality.

Not much finding difference, which is also, I would say, the separate reason at this point. But it's more about improving code quality rather than finding difference. That's interesting. It's a bit different from what Microsoft developers said. And I guess most of you are working on open source.

So let's see what the results are. That's very interesting. And you'll see why. Okay. So according to Microsoft developers, the main reason to do code reviews is find the defects. The main reason for you is to improve code quality.

Now, these are things we asked them. Like, what do you think? Would you trust anybody? No, I never trusted anybody. Regardless of Microsoft open source, I want to see their own data. So now we know why they do that. The question is, does the outcome match this expectation?

How can we test it? Well, we took advantage of the fact that we used the same tool for doing code reviews. And that they leave comments. In these comments, I can really see what reviewers said about this particular piece of code.

So that if they were finding defects, they would say, okay, you should change this piece of code. So what we did was taking a sample of almost 600 of these threats and we manually categorized them according to what you were asking.

We did this classification of review comments and we compared this with what they thought the reason for doing code reviews was. And we did a thing called card sorting. This is another thing we did research. We printed all these 600 threats. We spread them in a super large table and then we started clustering them when they were talking about the same topics.

And this is, for example, a stack of cards talking about, with comments about no necessary code. Okay? This was halfway through the catastrophic process. And these are the results.

So as you see, the main outcome of code review is code review and union death. Very good. While defects, it's only around 14%. Right? This is very different from Derek's potentials.

And what is more, if you look at the comments about defects, you really see that they are very low-level trivial defects. Like, what if this part is used? Hanging out about precedents?

Should this be and instead of or? There are both questions because of course it's nice to be nice when you do code reviews. You don't want to attack people. That's very important for them. On the other hand, there are very, very low-level issues. Not talking about design. Nothing.

So basically, code review of Microsoft expectation do not mean reality. Okay? Now the question is, what is the study outcome of code review in open source software systems? Okay.

For this study, I didn't have to go anywhere. I could do that from home, from my office. And I did that in collaboration with a very good case in the school. Mark Butler, who did this study as part of his master's thesis. So we took a sample of two open source projects we knew very well.

In this case, Chromats and Anka. But we've seen very similar results in many other projects. So what we did here is we manually analyzed more than 1,000 code changes that were introduced due to review. What does it mean? This is, for example, a review happening on Garrett.

This is the code submitted. And you see that, for example here, the name of this variable has changed. Here, this comment has been removed. And here, this commercial.close was also removed. Sorry, it was ugly.

Okay? So we analyzed more than 1,000 of these changes. And we also classified them to see which kind of changes they were. And we had this classification. So maybe it was an implementation change, it was a visual representation, like we did white spaces,

change in limitations, so on and so forth. And some sort of more functional effects. So the three bars on top are non-functional changes. Meaning changes that affect the code in a way that doesn't change the way it's coordinated. This is very similar to what Microsoft developers

would recommend in comments in their code reviews. While these ones are the functional changes, things that really relate to bugs or to changes in functionalities, important ones. So to make this a bit bigger, basically, both in GoCAD and in Grommets,

the biggest amount of changes triggered by code review are about evolvability. Not really so. Better comments, better variable names, better white spaces, and so on. And much less changes were about functions in the source code.

And if you compare this to Microsoft, results are pretty similar. And if you also look at another two studies they have done, different researchers completely separated from us, they also found this thing.

So basically, the answer to the question, the title of this talk is, when do they have in common? Well, it's the outcome. The motivation for Microsoft developers is finding bugs, but it's not that it's motivation.

You know it's for improving the code. But it's really amazing how, given that you have different motivations, the outcome is actually the same, isn't it? So, especially after the days of Microsoft, I thought, why is the expectation not really met?

And this is basically what my research nowadays is about. I thought really hard why this was happening. And the reason is that if you, who knows this book?

I really recommend this book. It's about research done in software engineering that can be useful for practical purposes for real software engineers. It's a very beautiful book. And there is one chapter about object theory. And it says, find bugs faster or effective than testing

or other non-buggy techniques. But, when done efficiently, they can quickly become unproductive. Okay? So, apparently, you can do good at this. You can get very good results. And this is what you read in blog posts and so on.

But you have to do that carefully. And the problem is that to do that carefully, you only have to rely on yourself. Because nowadays, code review is a fully manual task. If you look at this tool,

which is exactly the same as all the other tools, when you receive some code, what happens is that you have yourself to understand what's going on in code review, in the change you are doing, and decide whether it's a good or a bad. Do you have any support? No. It's just you and your time

that makes a code review good or bad. So, I asked you what are the challenges of doing code reviews. I would like to see if there are a few that stand out.

So, let me respond. What are the challenges understanding the way other people think? Finding proper time to do it. You're boring.

Keeping it... Would you agree? Sorry? I agree. Takes a lot of time, hard to wrap your head around other people's code attacks, it's hard to settle the pace,

be consistent, proud of their code, make it more simple, time consuming, get enough time to move, take off, so on and so forth. So then, I asked this thing also to Microsoft developers,

and basically the results were pretty similar, but one thing really stood out as the main challenge of doing code reviews, and it's understanding. Understanding the code you receive is the most difficult thing, and it's the thing that requires the most time.

If you... We asked them without letting them know that we were thinking about understanding as a challenge, how much understanding they needed to achieve these goals. And in fact, if you say finding defects is the activity that requires the highest understanding.

And this is why they didn't have so many defects from the interviews, because it required very high understanding, which means a lot of time. For improvements, on the other hand, you don't really need to understand the code so deeply. You can always suggest

how to make your code more readable, right? And this is something you can easily do with the diff tools you have in code reviews. But understanding, there is no support for that. And so, understanding means that if I wrote something, I generally understand what I wrote. So to have a...

to try and get this finding about understanding, we asked, so, what happens if you send the code to the owner? Does it take longer to review files that you are not familiar with? The files that you don't know? Yes. Okay. Not surprising. Just for making sure we are seeing the right thing. Is there a difference in how much

feedback you receive when the reviewer is very familiar with the order of the files? Yes. Definitely. So understanding, knowing what you have to review is the key point. Is there a difference in how much feedback you receive? And this is the kind of difference.

People that understand the code, they have to review it. Their problem is that they are deeper understanding, more likely to find subtle defects, feedbacks with more conceptual, better ideas approaches instead of superficial problems. So, the main challenge, according to

Microsoft developers, is understanding. But also social challenges are very, very important. And all these things are not supported by the unique tools. So, what I'm trying to do with my research is hopefully improve code review tools, and I hope in the next few years you will see some

of these in practice in some tools. And to do that, you need the right data and the right tools. So, who knows about software analytics? Who knows about data science? Okay?

So, data science is very popular nowadays. The idea is that there is a lot of data that you can collect from different sources and you can analyze this data in existing results. Software analytics is basically the same thing but applied to software engineering data. As a couple of researchers have put it, software analytics

is analytics of software data empowering software development individuals and teams. Let's make this a bit more clear. That's the typical data for a software developer and while she's developing, she's using a lot of tools. And all these tools, the IDE, of course you don't use

Visual Studio, the debugger, the versioning system, the issue tracking system, executions, the main issues, all of these systems, they generate a lot of data. The idea of this of software analytics is to use this data to help software engineering itself.

So, the idea is to take this data, you do some data mining and software analytics and then you manage to help some of these tasks. An interesting thing is that when you build tools to help this real system that you can use.

I'm going to name this more practical than just this type. The idea is to use the same process for code review. Also, I think review data. Okay? So, what does it mean? What can you do with these kind of things? My research is about data support and code review. So,

one thing you can do is turn it back to the university and tell them the changes. I'm going to tell you what this is. And, for example, telling you where to look at when you have to do a project review in the first instance. So, recommender for the university. We've just seen that it's very important

that the people that receive your code understand your code. So, how do I know which people I should send my code? But if you ask these to Microsoft developers, they tell you, I perfectly know who I should send my code. And then you write some of these and then you suggest different people and they say, ah, that's a good one. I didn't think about it.

So, the recommender for reviews. The idea is that you take the changes that you want to be reviewed. You add data from the versioning system, for example, about who changed this code first, who reviewed this code, but also from the issue tracking system, who thinks about it, so on and so forth.

And with this data, and not the mining, you can easily suggest who should review your code. Isn't this something useful to have, for example, GitHub when it's on a pull request that automatically tells you, okay, these are the best people that should review your code.

And then if you're in a team, you also want to include some rules yourself like saying, I always want a junior person to be there, so he or she can learn what code should look like. The second thing is intelligent changes. How many times have you seen a moment

in which there are multiple tasks being addressed, like bug fixing and refactoring. Like, who's told this more than 50 times? Or, the idea of this tool that we already developed is that it basically takes

the changes that humans like other people to review, it uses data from the versioning system about past changes, but it also records a lot of data while you are developing with the idea for this. And automatically, it splits this change into side-contained changes that

should be reviewed in different instances. I don't have time to give you all the details, but you can come with me and I'm very happy to talk about these things with you. But just to make you realize how much power you have if you analyze the data that developers produce,

who has the data? And finally, developers always don't have much time, so you want them to focus on things that may be the most recent, right? So that they check these things first, and then if they have a little more time, they maybe can check the other things. And this you

can do. If you analyze the history of past defects, the history of changes of files, you can really pinpoint parts that are very likely to have bugs. And for example, we did create an app library in GitHub that tests you also using

static analysis tools like find bugs, PMD, and so on. But that's you. Wait, look carefully at these lines, because they are critical. There might be errors. Okay, so these are three things you can do, but we have many other things in mind. I hope this

triggers some interest from you about these things. And one thing we're also working on is creating an analytics platform for data. Basically, a plugin that will record how you use data to request, so that you can understand the data and provide method tools, because

it's not about, of course, these data can be optimized, but the idea is to use data for good reasons, for good reasons. Okay. So, that's the overall idea, and it's based on the difference between what the expectation and outcome of coding is.

So, this concludes my talk. I would like to summarize it a bit. So, the question was, what do we want to use a Microsoft and another server-side problem? Explaining why I think it's important to do research on how to use, we did an

empirical study at Microsoft to see how developers do code reviews there. We found that expectation do not match reality. And the outcome is exactly the same in other software systems, even though we knew that the outcome was a different one than Microsoft code.

And why did this happen? Why don't we do deep code reviews? Why do we find shallow things? Because great code reviews is a very difficult task, and now it's proven. The idea is to bring data to have code reviews. Before concluding my talk,

if you like these topics, I'm looking for two kids, these students, working with me. And if you happen to be a master student, I'm also always looking for great master students to work with to write a great master student. And this concludes my talk. Thank you.

I think you're completely wrong about everything. This is rather a lot for me. Right. In that face, I always say, you're encouraging. Well, I think that the more you have it, there's going to be an experience, which is

closely connected, really intense. To me, the number one goal is to improve the quality of the code, the accessibility. The reference is secondary. So, in my experience, this bit where I do most of your review work,

doing superficial reviews, is really just getting the first hurdle out of the way, which is I don't want this code. I must be able to understand it. And the best review I've had was the ones who were able to say, I'm not an expert on this. Make this clear and more accessible to people. And so,

when we select the right review for the code, you're not going to get that feedback from somebody confident enough to do it. When you unattain the code, you permit code to write large projects, to write large change sets, but what you really want

is for them to come back in for review very early with small increases. Okay. So, thank you so much for this fantastic point of view. So, you're basically saying that

it's better to have someone who doesn't understand the code to review your code so that you know what you have to improve to make it more real. I totally agree with you. And this is why it's very important when you have a commander for reviewers that you let people

with the possibility to put rules by the one junior people or someone who's really not expert. But as long as you don't have data to know who's not expert, how can you know that? This is one part. The second part is, unfortunately, sometimes

there's not a lot of time for people to put rules. So, when we have one field study analysis for example, we can do the formatting, white spacing, variable remaining, all for you. So, why not using that in the context of code review? And let reviewers spend a bit more time on

the more important parts. So, I totally agree with you. This is an important point of view. Thank you. So, you said that the code reviews you said the code reviews

you focus on two main things, like improving code styles, standards, and everything, and removing bugs with code reviews. Then there's one thing that's really hard to test, but if you're already doing research into this, I think it would be really

fun to look at, is how many bugs are actually avoided because of refactoring the code so it's more usable in the future like, if you have a lot of copy-pasted code and the reviewer would say, hey, I mean this looks the same on these two levels, and you can reuse

it and just refactor it so that you can put it in a function and then in the end, when someone changes something in there, the copy-pasted part will be affected as well, and you will avoid bugs in the future because of improving code quality. the distinction between those two

is not as clear cut as one would think and my question is have you looked into this and it would be possible to have some metrics on how many bugs are avoided if you had better code quality to get okay, this is very fascinating

question so it's always very difficult to do studies and find the reason why a bug was found and it's even more difficult to find the reason why a bug was not found what I'm saying is it's very hard to say

if a practice had a reasonable amount of bugs especially because there are many factors that lead to bugs that's very difficult I agree with you that code quality also improves what makes bugs less likely

people don't understand code better so there will be less bugs so these are very interesting studies I will think about how to do that to go back to the previous comment in our organization for junior bugs actually there are two core reviews

one that focuses on correctness and you know bugs and the second one which is called credibility I focus only on style and clarity and it's self selective reviews and it's been very good and it does a lot thank you, so it's very

interesting to hear about these different practices so in the summer when we started at Microsoft they didn't have this division they may have some like a test work or even a product manager to do as a reviewer the code just to get the effort somewhere but they wouldn't comment and say anything about it

this is a good question so I'm going to force more lessons but even more the input of good review is actually to get the design so that's

in some sense much easier in finding bugs as long as you have experience in the theory you get immediately to the design so that's the point of having your subject matter it's not just people who work for finding bugs there's stuff like static analyzers

finding bugs review is important but so I haven't found any traces of these high level issues that you are talking about this code should not be this way

but change completely change the code I haven't seen that probably because they did this kind of design we used a bit easier but that's another thing basically you don't have any support for the project and it would be good for example thinking about

software visualization these things talk to your microphone

so you don't have any more what's up you talked about how developers think that spectroscopy points come from people working on the actual code have you done any actual function of

years of experience in a particular area versus actual coding finds to confirm that or is that all just subjective? you talked about the correlation between more

experience in an area of code and better or more consecutive code is that based on actual numbers in terms of analyzing so is it just subjective? ok, so this was based on the historical data but that's something I would really like to test the code

and this is something that helped me a lot seeing the effect of experience on some of the other comments that's a good point thank you

have you studied some advanced concepts like people coding together and then it's like a code review at the same time? so I'm looking with another research from

Brian Gerstmann he is doing research on peer programming and we are trying to look at the similarities with coding what really changes is the way we should share knowledge but I haven't looked at this part very deeply into that one thing I want to report here

is that at Microsoft where developers are correlated 20% 30% of the time instead of using the tool even if they are separated by many offices they prefer to meet and talk in person these are the problems with bandwidth

you have to share knowledge but also if you have to write a harsh review it's better to stay somewhere in person so these are the things I know about when you are in person one more quick question and answer and the time is over in this situation, recommendations

how many reviewers should be taking in this review? so when does it make sense to involve all the team in the review? or when does it make sense to pick one expert one or two experts

so there have been a few empirical studies looking at what's the sweet spot of the number of reviewers it seems that two to three reviewers is a good amount to have a third review so you don't have the entire team but they haven't studied this

in terms of helping with awareness normally developers wouldn't have time to review all the changes, all of that so two to three seems to be very similar does that answer your question? thank you very very much for the talk

it was very interesting to hear also many questions about that and discussion thank you very much