Building a Community Metrics Strategy

Video in TIB AV-Portal: Building a Community Metrics Strategy

Formal Metadata

Building a Community Metrics Strategy
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
You probably know that community metrics are important, but how do you come up with a plan and figure out what you want to measure? Most open source projects have a very diverse community infrastructure with some combination of code repositories, chat, mailing lists, wikis, forums, and more. Deciding where to focus and what to measure across these many technologies can be a challenge. What you measure can have a huge impact on behavior within the community, and you want to make sure that you are encouraging people to contribute in sane ways by measuring the activities that matter for your project. This presentation will cover how you decide what to measure and provide some examples.
Presentation of a group Metric system Slide rule Building Tape drive Operations support system Strategy game Blog Strategy game Website Speech synthesis Computer-assisted translation Metric system
Slide rule Intel Open source Software developer Multiplication sign Source code Chaos (cosmogony) Mereology Product (business) Strategy game Internet forum Whiteboard Kernel (computing) Core dump Extension (kinesiology) Module (mathematics) Email Software developer Projective plane Feedback Electronic mailing list Mathematical analysis Chaos (cosmogony) Bit Software maintenance Cartesian coordinate system Data management Kernel (computing) Process (computing) Internet service provider Finite-state machine Whiteboard Metric system Sinc function Writing Local ring
Presentation of a group Metric system Open source Multiplication sign Gauge theory Mereology Bookmark (World Wide Web) Twitter Element (mathematics) Measurement Strategy game Energy level Area Focus (optics) Email Projective plane Electronic mailing list Bit Arithmetic mean Personal digital assistant Mixed reality output Self-organization Quicksort Object (grammar) Metric system Arithmetic progression
Metric system Open source Code 1 (number) Chaos (cosmogony) Focus (optics) Number Latent heat Strategy game Different (Kate Ryan album) Set (mathematics) Electronic visual display Right angle Position operator Area Addition Email Focus (optics) Structural load Projective plane Electronic mailing list Measurement Type theory Software Self-organization Metric system Arithmetic progression Resultant
Focus (optics) Open source Online help Multiplication sign Direction (geometry) Projective plane Electronic mailing list Set (mathematics) Bit Function (mathematics) Number Measurement Explosion Software Blog Self-organization Quicksort Metric system Arithmetic progression Traffic reporting
Point (geometry) Slide rule Focus (optics) Statistics Greatest element Metric system Strategy game Open source Link (knot theory) Metric system Pivot element
Ocean current Addition Group action Metric system Common Language Infrastructure Real number Projective plane Commutator Basis <Mathematik> Bit Mass Regular graph Pivot element Smith chart Number Local Group Mathematics Strategy game Self-organization Game theory Metric system Arithmetic progression
Axiom of choice Point (geometry) Statistics Computer file Open source Code View (database) Multiplication sign Chaos (cosmogony) Number Frequency Goodness of fit Whiteboard Different (Kate Ryan album) Office suite Position operator Area Time zone Structural load Projective plane Database Bit Line (geometry) Pivot element Existence Measurement Inclusion map Repository (publishing) Personal digital assistant Normed vector space Order (biology) Self-organization Whiteboard Metric system Window
Group action Building Metric system Open source Code Multiplication sign Set (mathematics) Chaos (cosmogony) Mereology Lattice (order) Software bug Inclusion map Latent heat Position operator Descriptive statistics Task (computing) Area Structural load Projective plane Chaos (cosmogony) Bit Software maintenance Complete metric space Measurement Inclusion map Word Personal digital assistant Self-organization Whiteboard Quicksort Metric system
Email Group action System call Link (knot theory) Open source Variety (linguistics) Projective plane Electronic mailing list Chaos (cosmogony) Chaos (cosmogony) Electronic mailing list Group action System call Inclusion map Figurate number Metric system Capability Maturity Model
Area Group action Focus (optics) Multiplication sign Projective plane Chaos (cosmogony) Cartesian coordinate system Event horizon Measurement Dimensional analysis Radical (chemistry) Inclusion map Process (computing) Personal digital assistant Repository (publishing) Different (Kate Ryan album) Canonical ensemble Videoconferencing Row (database)
I'll be discussing the
strategy thank you for coming on [Applause] thank you everybody can you hear me in the back okay awesome so you can find a copy of this presentation it's already uploaded so you can find it on fast wonder blog slash speaking or it's also uploaded on the FOSDEM site so it's all there so you have it already now as you'll notice throughout this presentation community metrics strategies are indeed based mostly on cats with measuring tapes so that's the primary thing you can take away from this presentation I have actually spent
more than twenty years working in the technology industry and for most of that time I have indeed been focused on open source and metrics was always a big part of my work with an open source projects as director of community fat puppet we used our metrics dashboards to provide updates and recognize community members before puppet I was community lead for Intel's open source Technology Center where I did similar things with me Joann ties and metrics I just finished a PhD and as part of that work I did a whole bunch of analysis for the Linux kernel mostly looking at mailing lists but also its source code data and a few other things I'm on the governing board and I'm a maintainer for the chaos project which Brian mentioned and I will talk about more later in the slides and it's a Linux Foundation project open source project based on community health metrics for open source projects I joined pivotal back in October so relatively recently to focus on open source strategy and right now I'm mostly working on our open source strategy for contributing to kubernetes so of course I've been digging into some kubernetes metrics since we're talking about
community metrics strategies I am gonna start with how I think about community since this drives how I think about metrics and I tend to define community a bit more broadly than how I see some other people define it so we'll start with that when I talk about people I am indeed really including all of the people who work on the project so I'm talking about core product developers release managers Quality Assurance localization and other people who contribute to the product itself application developers who are maybe using the API or other features to write applications on top of your product through applications modules extensions things that build on top of your project users people who actually run your software and with any luck maybe provide some feedback vendors so the companies who are creating their products based on your project and hopefully giving some engineers back to the project as a part of that work and then other contributors people who are working on promotion organizing meetups doing moderation documentation and all of other stuff that happens around open source projects some of these people contribute as part of their day job on behalf of the employer and others contribute on the free time but they're all really important parts of your community now before we dive into the
strategy let's talk about some of the ways that you can use metrics so metrics allow you to measure progress to see if you're achieving your goals so using metrics to measure whether you are successfully achieving your goals is a big focus for the rest of this presentation so I'll talk more about that a bit later but as you're defining your goals it can also help to use any existing metrics that are already available and just sort of lying around to learn more about your community and explore in more detail how people are currently participating and this can be a really valuable input into your goals and strategies another use for metrics is to look at your community over time to spot trends gauge interest you may notice that people are showing a lot of interest and something you didn't really expect and knowing this might help you encourage further contributions in that area on the other hand people might be showing less interest than you had hoped in another area and this gives you an opportunity to dig a little bit deeper and talk to people to understand maybe why they aren't as inter and something that you thought was really really valuable or really cool however one of my favorite things about metrics is that it tells me more about who is contributing to the project the people make the community and for me the people are the most important element of the community so my metrics do tend to focus more on the people than anything else and metrics not only tell you who contributes to your project but they also tell you where they're contributing within the project and this allows you to learn more about your key contributors and I like to use this also as an opportunity to recognize them it's nice to recognize that person who maybe answer so many questions on the mailing list or maybe you contributed something awesome that a bunch of other people enjoy using now when I say start with
the goals I don't mean is to start with your goals for the community I actually think you need to take a few steps back and start at the very top so what's important for your organization or for the project as a whole now this is typically been a company in my case except which for lots of big companies that are doing up and source but it could be an organization like the Linux Foundation it could also just be a project like the kubernetes project instead of an organization and you need to start by looking at what the organization or the project hopes to accomplish and what its goals and strategies are and its objectives and then you can take this down a level to start to figure out what your goals are as a candidate and as I mentioned earlier looking at some existing metrics can also help you explore what people are doing now as input into defining your metrics or defining your goals for the community what I think the most important part of putting together strategies and plans for your open source contributions is really aligning them with these overall goals for your project or organization if your goals for the community don't support the overall goals for your project then you aren't likely to be successful because the community is just doing Rann things that nobody cares about so it's worth the time to figure out what you actually want to do and how it supports the rest of the project or your organization and once you figure out what you want to do as a community and can tie it back to the bigger organization or the project as a whole then you can start looking at using metrics to measure your progress now
some of you might be thinking that starting with the goals really isn't that important so I'm gonna spend a couple of minutes talking about what can go wrong and trying to convince you why the goals are important but I think it falls into a couple of a couple of different areas so first you you just won't know if you're successful if you aren't measuring the right things and if you aren't measuring the things that are important for your project or organization you won't know if you're making progress in the areas that you actually care the most about and second measurement actually impacts behavior and this is something that is really important to think about people will do different things depending on what you measure for example if you public publish metrics that focus on the number of mailing list posts and that's the focus for your metric you're likely to start getting more mailing list posts because we're vain and we want to move up in those in the rankings and if what you're trying to do is maybe get more people to contribute code to your project or get more people to do something specific then additional mailing list posts probably aren't going to help you all that much
now I love data so measuring open-source project success is kind of a hobby and I'm pretty passionate about I'm not sure what that says about me this is my hobby but that's such as it is but as a result people tend to ask me a lot of questions about metrics usually the questions are focused on how to decide what to measure and people often ask for examples of projects with the best metrics but I really don't think that's a good approach what you measure depends on your goals and what you're trying to achieve which may be completely different than other projects so I really prefer to encourage people to start by defining their goals and ultimately you need to look at your strategies and plans to come up with criteria that will help you measure whether or not you're successful for example if you want to improve the performance of a particular piece of open-source software you'll want to have success criteria and measurement based on specific types of performance if you want to gain influence within an open-source project maybe you need to measure increases in contributions or the number of employees moving into positions of leadership or positions of influence and as with any good strategies and plans the outcome and the results should be measurable so that you can actually tell whether or not your efforts were successful and this is where your metrics come in handy once you decide in your success criteria you need to make sure that you can actually get the data that's required to measure these things and measuring it start measuring it now to get a really good baseline for what you have and there are loads of tools available to measure this stuff so there's lots of lot of tools available to gather a contribution data about open source projects and some of the commonly used tools can be found in the Linux foundations chaos project and which I'll talk about later and many big projects already have dashboards using these tools and other ones to display contribution data for the project now
that I've talked a lot about focusing on the right metrics for your project I'm also going to recommend measuring other things that just aren't even important to you now and going a bit overboard on your metrics because I love data so just I just want more data but you may not spend much time looking at this data now but it might help you later because there are there are a few things that can happen so the obvious is that your project goals might change and something else might matter in two years that don't doesn't really matter to you now and if you've already been capturing a bunch of data you can probably track progress towards this new goal with some existing historical data and the other piece and I think this is probably more likely and maybe a bit more important is when something unexpected starts to happen so this is actually one of my favorite things about working at open source project is you just never know the direction the people will take with the stuff you're working on and how people will use that software so with a really robust set of metrics then you can dig in and learn more about something that you didn't expect to see and you might need to adjust your goals based on some unexpected things that are happening in the project so now that you
have a whole bunch of awesome metrics you actually need to do something with them so historically most of the places I've worked I've reported metrics monthly if you work at an organization that doesn't really get community or get open-source monthly is probably pretty good cadence when I worked at puppet they were a bit more savvy and so I think I reported metrics quarterly up to the exact team but being able to show frequent progress and again the stuff that you're the stuff that you're showing people is are the metrics that actually demonstrate whether or not you're being successful so being able to show frequent progress to help justify what you're doing and show that you are indeed making progress towards accomplishing your goals as I mentioned I was at puppet I did a short report I think it was about quarterly for the exact team that was focused on actual participation in the community I don't usually bother tracking things like people who join just to lurk I only try to count people who've actually contributed something whether it's go door Docs or answering questions any of those sorts of things we also did a blog post on the puppet labs blog one a month the focus less on the numbers and more on recognizing contributors and for this report there was a lot of human filtering that happened we sometimes highlighted someone who made some important but small contribution that just wouldn't have shown up in the you know top participants list and we also attended to filter out people who had behavior that we did not want to encourage people who were using up lots of people's time and not a lot of not a lot of actual output and instead we put a heavier focus on the people who were helpful and answered lots of questions from other users now I also wanted to provide some
example metrics the show how metrics fit into your strategic efforts so at pivotal as I mentioned earlier my current focus is on putting together a strategy for our open source kubernetes contributions so I'll focus my example metrics on pivotal and kubernetes lucky
for me the San CF has metrics for all of their projects including kubernetes so this was really a great starting point for me to better understand the existing kubernetes community they use dev stats to build their metrics dashboards which you can find it a link at the bottom of the slide but first let's start a quick
discussion about vanity metrics so when I talk about vanity metrics I'm referring to those metrics that make you look good but that don't actually tell you much about whether or not you're really successfully achieving your goals so I'll admit it like anyone else I look at these metrics and it feels pretty great when you or your company are moving moving up in the ranking but I try really really hard to keep these vanity metrics out of my strategies and plans since they don't really tell you much about whether or not you're being successful for example let's say that your organization is moving up in this ranking it looks great but what if you're moving up because you have a rogue employee or possibly group employees who are gaming the metrics by submitting a whole bunch of trivial one character changes each is a separate commit with a separate full request so this is going to drive the commuter pull request numbers up for your organization but likely at the expense of your reputation as you annoy all of the other contributors now I know I know none of you would do this none of your colleagues that do this but then that is actually a real example of something that's happened more than once it's in the kubernetes project and we have this discussion and the contributor experience say got a fairly regular basis because companies people at employees at companies do do this and I know this is kind of an extreme example but there are plenty of other ways that people can participate that would drive numbers up without actually helping your organization achieve its goals so I just encourage you to be a little bit to be aware of the vanity metrics now in
kubernetes most of the work is done within various special interest groups or SIG's so the kubernetes contribution strategy that I'm working on at pivotal is focused on coming up with specific strategies for the SIG's that are important to us so that the various teams across pivotal can understand where their contributions to kubernetes you can have the biggest benefit and and be aligned with the SIG's that are going to be important for not just current work at pivotal but also future work I put it all and because kubernetes is such a massive project with daily changes there is no possible way that one person can possibly understand every single detail of every single thing that's happening in the 30 plus SIG's and the dozen or so working groups so to be able to build a strategy for our work within kubernetes I need to be able to find out who I can talk to to better understand what we've done so far in addition to what we might want to do in the future and also you know pivotal is not not huge but we're not a small company so finding these people can be a real challenge and one of the things I did in my first couple of weeks at pivotal was to start to dig into the existing metrics that the CNC already has to understand our current project progress and who's already been contributing within and who's been contributing to which SIG's within the kubernetes project and this helped me find some people to reach out to and gave me baseline for where we were already active now here are also a few snippets
from a grimoire labs dashboard that came from butter Chia with a few more details showing Fidel's contributions to kubernetes over just the past three months so before looking at the kipper Nettie's data I had never really used dev stats so it probably has some features that I don't know about but I found it a little a little bit limiting compared to the dashboards using grimoire labs as far as really digging in and figuring out what individual people were doing what I do love about these grimmer labs dashboards is that they're built on top of Cabana which means that I can filter my view of the data on any field available in the database which is pretty cool in this case I've just filtered it on organization name to get the github commits from pivotal but I can also filter on a single person or a positive time zone a bunch of a bunch of other stuff that's in the database and I find these really useful for better understanding how individuals are participating within a community for example if I look at contributions from from Honus who's up at the top there from our London office you can see that he's removed a whole bunch of lines of code which indicates he's been contributing to existing files right so he's been removing things adding things and contributing to existing stuff and he's done this across a number of different repositories Ben and Leah on the other hand who worked together on kubernetes for Windows in our New York office have only added lines during the time period that I used mainly because I could get this as a good example i shortened to the time period drastically but it corresponds to the kubernetes enhancement proposal that they just submitted for Windows node support and you can also clearly see in the in the top you can see the time zone split right between our Bay Area office our New York office in our London office which is where most of the kubernetes work happens which is kind of cool to look at but again this really helps me understand who's contributed to kubernetes from within pivotal and which areas we've contributed to but with a few more details than what I could manage to extract out of the Deaf stats dashboards now when getting
started with metrics there are a couple of different approaches that you can take before you start building anything it's a really good idea to look around and see what kind of existing dashboards already exist that you can use many projects already have these dashboards like the CN CF dashboards for kubernetes that I used as a starting point and you can also take the DIY approach so I've also done a fair bit of this and built my own dashboards gathered my own metrics it requires a bit more technical know-how but it's a good approach for people who have the time and resources to build and maintain your own metrics if you have really simple needs you might even be able to just easily hack something together with the github API is those are pretty easy to use for most people the tools that then the chaos project like grimmer labs tools for example augers another one are a pretty good starting point dev stats is also an open source project itself under the cnc f so you can always use that there are loads of other tools still called measure that Jeremy Garcia from data doc has been involved in and presenting about but loads of loads of different tools you can use by far the easiest choice is to pay someone so pitcher Tia does this whole disclaimer I'm on their advisory board but they you can get companies like that to build and maintain the dashboards for you which requires way less time and butter dia for example uses all open source tools so you're also supporting their contributions back to open source projects which is pretty cool but in general I recommend kind of doing a little bit of all three if you possibly can and doing them in this order because you can use the existing dashboards see how much of what you need already exists requires a lot less time then if you have the time the skills required hack something together using some existing tools and which if nothing else is a really great learning experience to help you figure out exactly what you need and then if you want someone else to maintain your dashboard you can pay someone to do that it's also really
important to think about other metrics and measurements that aren't necessarily in a traditional community metrics dashboard there are loads of things that I've used over the years to measure whether or not we're successful that you might not find on the traditional dashboard for example when I was at puppet one of the things I tracked was whether we were increasing the diversity of speakers at our annual conference which was one of just several measures that we use to determine success towards becoming more more inclusive and more diverse within our community and loads of these diversity and inclusion metrics aren't often found in traditional dashboards but they're a really important part of measuring whether you're successfully achieving diversity and inclusion goals for your project the chaos project again also has an entire working group devoted to diversity inclusion metrics for projects that are really important to your organization you might have goals around having specific people move into leadership positions or especially in key areas within a project and the leadership positions available so they vary widely depending on the project but they can include becoming board members maintainer zkk emitters reviewers but it's important to measure this separately because these positions rarely if ever appear on the community metrics dashboards and many people who move into these leadership positions spend quite a bit of time reviewing contributions which means that they often actually have fewer commits themselves and this means that the dashboards may actually show them as having decreased contributions when they're actually doing more within the project and having increased influence so it's important to look at that a bit separately in some cases you might have specific features that you want to contribute or specific issues that need to be fixed in these cases you might want to measure a specific item like a specific pull request merge a specific bug fix or some other contribution that you need outside of the metrics dashboard so in other words as metrics dashboards are great I love them I love data but you'll likely need to find other ways to measure things outside of your dashboard to build a complete set of metrics to help you determine whether or not you're actually achieving your goals now I mentioned the chaos project before
and if you're interested in working on metrics I would encourage you to join and participate in chaos so it's an open source project whoo I had several people in here are in the chaos project unsurprisingly since this is a metric stock but it's an open source project it's under the Linux Foundation and we welcome anyone regardless of your skills so while we certainly have people writing code to build metrics tools we also have a lot of people coming up with metrics and writing descriptions for them so we also love people who can help us promote chaos presenta conferences and all sorts of other tasks so this is something that's personally I'm just incredibly passionate about there's the
link to get you started we have three working groups so there's probably something of interest to you if you're interested in metrics we have one focused on growth maturity and decline of open source projects we have a diversity inclusion and then we have one focused on metrics that don't fit anywhere else and we have mailing lists we have a general weekly call each of the working groups has various calls that you can participate in and as I mentioned before anyone with any variety of skills can participate in chaos project so that's my shameless plug for an open source project that I am particularly passionate about and with that there are three minutes for
questions and if the next figure wants to come up and get set up that'd be cool too thank you [Applause]
so what questions do you have for me today hi sorry how do you begin to measure say the diversity of a particular project on any axis of diversity so so the question was how do you measure a diversity within an open-source project the I wish I had like this awesome magic bullet for that because the reality is it is it is super hard to measure diversity and inclusion within a project so within the diversity and inclusion working group we actually have seven different focus areas that we're looking at so we're looking at things like our year are the events for your project do you have a you know attendees that are diverse in various ways and we have a whole bunch of dimensions of diversity that we've defined so it's incredibly it's incredibly complex a lot of times it comes down to one of the ways you can measure it is by actually surveying people and asking them how how they feel within the project do they feel included do they feel like it's a diverse project you can measure attendee demographics for events and participation yeah it's a because it's such a wide topic there is no single answer to that I would encourage you to go to the diversity inclusion repository for chaos and have a look yeah just its kind of interested in that because I'm surveying people from an underrepresented group you're you you're liable to get the answer you know Ferno say I'd have thought in quite a lot of cases so yeah it does seem to be hard yeah that's really hard there was a talk this morning in the Python dev room that tried to answer the previous question so I suggested to you check it out later on because there's a video of it and you can personalize those a recommendation for a job or termination just it's already done ok it's on different luckily fosston records everything because they're awesome alright microphone in transition we have another question I'll give it up yeah so when a small question [Applause]


 2578 ms - page object


AV-Portal 3.21.3 (19e43a18c8aa08bcbdf3e35b975c18acb737c630)