Add to Watchlist

Metrics-driven development

1 views

Citation of segment
Embed Code
Purchasing a DVD Cite video

Formal Metadata

Title Metrics-driven development
Title of Series EuroPython 2015
Part Number 66
Number of Parts 173
Author Root, Lynn
License CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
DOI 10.5446/20154
Publisher EuroPython
Release Date 2015
Language English
Production Place Bilbao, Euskadi, Spain

Content Metadata

Subject Area Computer Science
Abstract Lynn Root - Metrics-driven development At Spotify, my team struggled to be awesome. We had a very loose understanding of what product/service our squad was responsible for, and even less so of the expectations our internal and external customers had for those services. Other than “does our Facebook login work?”, we had no understanding of how our services we’re responsible for were doing. How many users actually sign up or log in with Facebook? How many users have connected their Spotify account with their Uber account? Do folks even use Spotify with Uber? With a 2-month challenge period, my squad and I focused inward to establish those unanswered questions and to establish feedback loops and always-on dashboards. This talk will tell the story of how we chose which metrics are important for us to focus on, what technologies we have used and are using, and how we’ve iterated over our feedback loops to fine-tune what metrics we care about.
Keywords EuroPython Conference
EP 2015
EuroPython 2015
Series
Annotations
Transcript
Loading...
if it's not quite good morning yet so is afternoon and and still stuck the inferences that time and that a lot of coffee there in some sleep so little jittery it I have to pee a lot but a mostly wake and so when I'm I'm I'm the back and engineer for Spotify and I in this
simplest don't listen and I am also and vise
chair of the PSS along with the new music and so you know if you are a PSS member and and quick announcement there is that his staff members meeting on Thursday at 6 pm forget the room but it's on the on line spectral and if you're interested in becoming a case of memory she come talk to me and last and I'm also the founder of pilot is different and I'm 1 of the main like lead organizers of the global organizations In
you of the and so that the
stage and who hearings actually clothing maybe it's better to go who has not heard of highly OK those handle be down at the end of this paragraph good and so this is a mentorship group for women in the Python community in opened to women in France open everyone and essentially where a loose group of new and
in various locations and and 1 on every continent except Antarctica which is my new missions to get in there to cancer upon and both what we do is then we have Python workshops and speaker events happen nights coffee that everything around Python and learning Python in development general and we welcome all experience levels so I think there is this 1 here in Spain and I think this Barcelona and they're highly encourage you to check out and I also have a talk
another 1 on Thursday afternoon at 2 30 forget our future method education some room and and that type the about priorities and and the more in-depth talk about doing and how the work is actually going on and the effects that we're seeing in the work that we still have to do and so on who here has not heard of diabetes and times e to the ends you need more coffee like me are it's like it might be and price of the stock and I will 1st give a quick
introduction to and to Spotify and over an overview of how we use data and I'll go into
and sort of how we use metrics and how we came about to implement that implementing them the agile nowhere in essentially what was learned along the way 119 implemented them to the bigger picture the company is about will give you a link at the end of this presentation with the blog posts on the slides so you can just watch me rather than a computer tablet or whatever and right so but
this is what you to take away is metrics and tracking is is super funny but but should you track all of it everything that moves and we as developers and we check everything like website visitors referrals how folks user services of servers or even out and we have a lot of tools at our disposal like a new relevant prof I do 1 would accentuate the duty a bunch of other things that we can track ourselves like steps and sleeping patterns exercise calories consumed breathing here growth rate and like everything that you can tracked rate on and maybe hopes to get some insights just to feel better about ourselves and that if this should you measure everything and very easy to get lost in that that the forestry and use it or lose the meeting and easy easy to lose the understanding of why you're measuring in the 1st place
so starts background information about Spotify and service on the same page and also is how we use data and so spot
find streaming music service and we update a local this is the not the correct agreement but I haven't gotten interior and so we better launched in 2007 and then we came to Spain and some other parts of Europe and and and in 2011 we came to the US and we were in about 50 countries we have over 20 million paid subscribers and 50 or 75 million active monthly active users we have over 30 million unique songs not including compilation albums and such and we had about 20 thousand songs a day we also pay about 70 to 80 % of our income to rights holders telling about 3 billion dollars to date and while I work in a very small office elements instances go with about 5 other developers and our main injuring offices are in Stockholm and in New York and with a lot of data machine learning in Boston it's so as you can imagine and that's Spotify data is quite important and these numbers that you see here only about a month old and they I have to check every time I do presentation like this because they are always growing and growing fast so attractive user data like sign-ups and logins activity within the application itself even tweets like the good and bad and the ugly and we have and we also tracks server generated data including requests various services services response times response codes among the million other things and each on some wet and they want to collect how and when and how they will consume such a data the only analysis that run thousands of you do jobs today to glean insight from user activity answering questions like how many pizza Scarborough's we have at this moment in time or was this partnership financially beneficial to us engineers behind and platform and watch usage rates tillægger Web API is we have we watch login failures feature usage 600 and we also have a data scientists in machine learning and analyzing wasn't behavior of music metadata and trends that power the recommendations behind their features and use of actually started to analyze actual Lake audio signals and the sound of a song to pick up like John and beats for mineral instruments playing a yeah and it's actually quite difficult to kind of pick up a few things from the audio signal itself like and that the mood like classifying defined mood of a song and that but that's the stuff that were sort of doing of sporophytes and on and scratches the surface and what we collect and what we pay attention to and we do use various technologies and related to data including Hadoop as well as the standard PostScript elastic search and all the user data sets in Hadoop on which we run jobs against using our own Python library or crunch scolding archive and we also use sparked find an I've heard of excellent people use an Python with sighted Learning in and can and of also discovered recently that we have our own IPython notebook server set up so that's pretty and on the back-end side and some of our service actually actually gets part inelastic searches worry have to buy a set up and that the majority in and of service activities actually in a homegrown system of course and what we have open source that it's called fast Fourier FSW d In some written in Ruby and and get was all set up with all this technology and I'm really embarrassed to say that's my team did a lot of development in the dark and we're not tracking anything and we do not know how successful our future integrations worm and we had no clue how back and services and that we maintain how they're holding out and I do want to make note that a lot of squads in Spotify do track what did you do pay attention to all this were just sort of the black sheep consisting of partly because we're at 9 hours behind Stockholm in 3 hours from New York so the story on about himself discovery and how to become how he became a better more effective team and we did this by an capitalizing on understanding around it and not everyone can be data scientists mathematicians statisticians and analysts whatever and their ranking graphs and
why it's important when and 70 % of our users can login for whatever reasons and so this is a story of and and finally developed and adapted to use have logging in metrics and so you
might know that spot I'm very public about using an Agilent and the action of a few videos on YouTube that are very very often very entertaining and also nicely done that I highly encourage you to go but I'm 1 key aspect of agile is iterations and we certainly iterate over product you might need might be as annoyed as I am when you open up the sporophyte Kleiner desktop and has a little blue banner asking you to update incomes like prematurity will de grade so about the agile approach and better and we also iterate over ourselves as as a team of individuals squad trying to find what works for for us as a company and as as a squad and everything in between only last year with my squad began participating in an internal programs very corporate speak called effective quite serpents squad circle and In its purpose was to hone in on the squad itself how we can be more effective and actually
found that and to be very very beneficial for myself and for the team so what was essentially a multi challenges and to figure out the team's current condition and essentially not checking anything not knowing what going on and comparing it to the desired that and condition condition in terms of delivering a product of Feature Service whatever that were meant to answer the following selection and may sound very light and project manager business-oriented but I'm a fan of very useful in pointing metrics and I also find myself and incorporating this thought process when talking to other teams non-techy intact about like diversity and stuff like that like measuring our diversity initiatives so it's very and widely applicable but so main
almost finer targeted condition as a squad hello where you want to be in and certainly difficult to establish a goal without context that an understanding of where we are now and so to figure out and a baseline we sat down an answer a few questions as a group and the 1st question and what do we deliver seemingly easy question yet myself and the squad and initially struggled to answer this right away it does laden roll-off at times so we look at our past enlisted integration projects and in that we delivered and services that we currently maintains includes an uber last of them somehow on twitter hashtag music and among many others and the most critical and is certainly a Facebook login and sign up and registration that hindered before about 70 % of our user base logs India Facebook and the rest is e-mail log which my team and does not the next question is for whom do we produce said the product or service and you actually defines our work as Spotify and we believe that leadership is meant to convey vision and that's what is meant to implement that vision and in a manner that they choose and there is isn't micromanagement is a lot trust actually and that are lead team defines the direction that our squad takes time so there certainly 1 of our customers and also there are many and integrations we've done a lot of external partners thankfully the quite a bit shielded from direct communication but and that makes business business development team and indirectly at the partners themselves are customers as well but who depends on us who actually uses our work product or service or whatever so yes 70 per cent of users you can log India Facebook and safe to say it's a pretty interval and system to Spotify platform and we certainly can't fuck up when they're on and Facebook makes breaking changes to log protocol or API which they often do unannounced before and I've had to life patches service because of this book but there are some other teams and then the company that plug-in to our system 1st social aspects like and sharing and to Facebook within the platform on and the next question is about expectations about what do our customers actually expect from us and try to answer this question occurred to us that we never act never actually ask them whether expert but their expectations and so we did we want to know exactly what was important to them with what we deliver and was it on time delivery was it predictive versus productive and do they expect solutions to problems that there may be no existence and what were the expectations on quality and usability in other measures where the expectations it with how we work as a squad did they want updates on progress and problems and so we can ask all of our customers right we have 75 million customers in and expectations of the difference among the various customers for the team internal teams expected our Facebook service to be reliable and scalable business development 1 has to be clear on what we can see feasibly implement which is definitely hard for web developer to adequately say a problem say how long something will take friend from and it's safe to assume that users will want to log or sign up the face but if they so choose in an effort to just work and so moving on the at the not last question simply and do we meet those expectations had we had we know we met those expectations and this is where we sort of stopped in our tracks right we can no we don't know a fire services can handle the extra load or if slash when users can login and more how many users of activated Spotify with an and of those does the experience actually work so and being a people and have an affinity for attacked and animation and we naturally implemented
a technical solution so on feedback loops and I'm
very generic term just text and make a feedback loop is that to understand against the feedback given 1st client and remain feedback loops and that we chose was metrics and we are 1 of those snazzy looking dashboards kinda graphs individuals using the latest technology will polity obsolete tomorrow but in all seriousness we
wanted intermediate visual of various metrics and but what do you want to see what questions we we want to answer someone with the idiom come
and you throw spaghetti on the wall the see what sticks and that's why the brainstorm for a while and trying to come up with questions that we like the answer answer some ideas included sign up the Oslo abandonment and Facebook connected users the percentage of total users in trend over time and % of users that have signed up to Facebook per hour day week we we need to know what the frequency should be in Facebook related areas which a lot of daily active users by partner feature and registration subscription rate referrals by partner and nobody can use in my partner and we even have like we really wanted like a squad focused Twitter when search for like you spot find the basis see what people were complaining about that like neither override party include could like C analogs and we want to know about sending during issues and the crosstown by an internal requesting service teams the group these metrics and and into buckets like usage system health business performance and is like it's eventually and became the Earl are becoming the on dashboards cycle through a big office monitors like everyone
to have but we also create a
few new processes and based on these questions when the process and reviews are progresses squad and everywhere retrospective we will look at a couple of metrics that deals with the squad performance like bugs close in the past sprint period and we'll also judge of this metric and if we want to continue seeing this metric if we can actively improve upon in like and maybe we only close to add to bugs this week but because it took us 2 days to knowledge 1 but background and in what is new and if any measurable items centrally governed x retrospective and another is to have gold targets and for every innovation project at medium for example we will noblest successful when we have with this integration we have x amount of new users within the 1st 2 weeks and it's true that this sort of goal line can only be judged and based on historical user acquisition numbers and so it is the so we decided to do some work but and this will feed into a retrospective especially once the project is complete like how did we do and we also have a few oppose integration questions and for business development folks ask of our external partners on behalf of this 1 like an understanding their responsiveness how are developer tools are and if they if they met the goals of and then we may and have may think an integration was like super successful but on there and not so much which is definitely happened to us before so and going caring
about metrics for a and
since the beginning of the year so certainly it is only the beginning for us and that allows us to iterate and give us a hard look at what we track
and wine you can track everything that moves that will you get inundated and certainly and so so if you count every single leaf of every branch of all the trees in the forest and so how can you tell what's important and what's just noise and so this goes back to understanding your customers expectations essentially boils down to business volume happy maintain and improve upon business value of the service a product has counting every Facebook but connected user help us to better ourselves so I'm thinking about implementing various metrics of our feedback loops and I came across various questions to help me see the forest for the trees so creating a new metric how do metrics
mapped to business calls and do we lose money there's there's so much money if and only if the Facebook's laughin services enough and how do you prioritize different goals and if we want to drive what's most important does that mean that you're going to neglect others are just allocate time by priority as this new integration project more important to pay attention to you than other ones that's fine and if it is but you just have to prioritize How can we create dashboards actually actionable what is the goal and more importantly how can we drive that all men are just men saying 0 look Facebook site services down that's go have lunch when I'm
representing metrics and how do we correctly
measure what we care about have to break out the old statics restitution book whatever but to like understand like how to best represent all metrics that you take and we have so many tools to help us create engages counters meters histograms timers but what representation is best for that question or metric actually consuming them how often do you check on metrics and dashed costs never looked at which is a common problem I found my team and they just be become like background noise and so how do you make dashboards more visible more in your face I should someone be responsible at once a week to like a goal in you make them more visible by slapping from a TV monitor which often 1 is not entirely work especially the train from images can be or perhaps you and have e-mail snapshots sent out to the team but maybe there are automatically filtered away for you're like me and auto archive all unread e-mails and being and that introspective and the thing is that we don't reach 100 % over goals we need to assess the difference white is that it exists is even solvable if you look at dashboards what actions are we actually going to take and even create a dashboard if the goal on alert isn't set up or if no action will be taken and probably not what about unknowns what is unknown and doing and we know that x amount of I users have connected their accounts to Cuba but how many don't use it because the driver has an Android phone or the driver just isn't aware of this sort and how do you approach those unknowns are you comfortable that with the memory is even worse than to explore them to bring back to the
slide and ultimately the goal and and answering these questions is to give us both of them ashore decision making cycles as well as a more informed decisions about am strategy in partnerships it's super easy to get lost in the forests and doesn't help us that we can get all this instant feedback that all these visualisations just look awesome but in essence an awareness is placing current values in historical context Odyssey patterns developing how long on average does it take for the team to implement a new integration your customers ourselves expect a shorter turnaround time
and do you wish to just be able to properly estimate the time and work on and the work that takes for such a project and or maybe which internal team do we have to educate about realignments on our service and the other 1 here and that that loops and this thoughtfully implemented metrics and we can use these goal line similar to create a more efficient teams and will deliver higher quality software instance will get immediate feedback bugs and that we introduced and any system that fails and and the like and but before I answer this question for a rapid and I do need to be a good friend and with all these stuff questions and contexts in your mind and you should get a Henoch stock tomorrow and it's at 11 45 and and things about practical its title practical logging metrics and basically the technical complement to the stock so alright and the answer this question should you check everything interconnected
answer probably only if defined goal you can define an action is and if you haven't met that goal and in the actually pay attention to it you
interconnected but within reason right so I think you I hope you
took or something and I think I have 1 minutes for questions Our
had that we just like go out and go did someone in and you can find me if you have questions now thank you
Link (knot theory)
Computer virus
Computer animation
Real number
Inference
Read-only memory
Musical ensemble
Computer animation
Whiteboard
Personal digital assistant
Self-organization
Staff (military)
Line (geometry)
Front and back ends
Spectrum (functional analysis)
Goodness of fit
Computer animation
Boom (sailing)
Open set
Local Group
Computer animation
Software developer
Multiplication sign
Boom (sailing)
Energy level
Sound effect
Event horizon
Data type
Exception handling
Graphics tablet
Forest
Metropolitan area network
Slide rule
Computer animation
Presentation of a group
Linker (computing)
Blog
Quicksort
Implementation
Metric tensor
Web page
Trail
Server (computing)
Service (economics)
Trail
Information
Software developer
Food energy
Measurement
Metric tensor
Computer animation
Bit rate
Website
Pattern language
Laptop
Musical ensemble
Server (computing)
Beat (acoustics)
Service (economics)
Open source
Presentation of a group
INTEGRAL
Logarithm
Multiplication sign
Mereology
Login
Code
Metadata
Power (physics)
Number
Twitter
Force
Response time (technology)
Machine learning
Bit rate
Elasticity (physics)
Office suite
Compilation album
Computing platform
Physical system
Standard deviation
Software developer
Surface
Moment (mathematics)
Element (mathematics)
Mathematical analysis
Set (mathematics)
Instance (computer science)
Cartesian coordinate system
Metric tensor
Process (computing)
Computer animation
Boom (sailing)
File archiver
Dependent and independent variables
Right angle
Quicksort
Mathematician
Library (computing)
Computer worm
Ocean current
Computer programming
Group action
Multiplication
Product (category theory)
Process (computing)
Key (cryptography)
Gradient
Projective plane
Metric tensor
Hand fan
Computer animation
Term (mathematics)
Boom (sailing)
Videoconferencing
Iteration
Circle
Selectivity (electronic)
Data management
YouTube
Condition number
Trail
Existence
Musical ensemble
Context awareness
Service (economics)
INTEGRAL
Direction (geometry)
Multiplication sign
Patch (Unix)
Image registration
Login
Scalability
Usability
Twitter
Expected value
Facebook
Mathematics
Video game
Subtraction
Computing platform
Physical system
Condition number
Chi-squared distribution
Product (category theory)
Machine vision
Software developer
Projective plane
Feedback
Expert system
Bit
Affine space
Measurement
Local Group
Computer animation
Web-Designer
Boom (sailing)
Quicksort
Arithmetic progression
Communications protocol
Computer animation
Feedback
Boom (sailing)
Feedback
Generic programming
Client (computing)
Graph (mathematics)
Metric tensor
Area
Metropolitan area network
Pay television
Service (economics)
Multiplication sign
Basis (linear algebra)
Total S.A.
Image registration
Function (mathematics)
Automorphism
Local Group
Twitter
Connected space
Metric tensor
Sign (mathematics)
Frequency
Facebook
Sign (mathematics)
Computer animation
Bit rate
Analogy
Cycle (graph theory)
Office suite
Physical system
Turing test
Process (computing)
INTEGRAL
Software developer
Projective plane
Line (geometry)
Number
Metric tensor
Software bug
Frequency
Computer animation
Dependent and independent variables
Quicksort
Service (economics)
Product (category theory)
Feedback
Volume (thermodynamics)
Branch (computer science)
Metric tensor
Connected space
Expected value
Facebook
Computer animation
Network topology
Forest
Boom (sailing)
Noise
Metropolitan area network
Service (economics)
INTEGRAL
Multiplication sign
Projective plane
1 (number)
Port scanner
System call
Metric tensor
Facebook
Computer animation
Website
Resource allocation
Metre
Ocean current
Read-only memory
Slide rule
Group action
Histogram
Context awareness
INTEGRAL
Decision theory
Multiplication sign
Online help
Wave packet
Medical imaging
Fluid statics
Strategy game
Average
Forest
Representation (politics)
Subtraction
Email
Metre
Trail
Feedback
Information technology consulting
Group action
Port scanner
Measurement
Metric tensor
Computer animation
Visualization (computer graphics)
Device driver
Direct numerical simulation
Noise
File archiver
Dependent and independent variables
Codec
Pattern language
Quicksort
Cycle (graph theory)
Context awareness
Group action
Service (economics)
Trail
Multiplication sign
Feedback
Line (geometry)
Instance (computer science)
Metric tensor
Software bug
Loop (music)
Computer animation
Software
Boom (sailing)
Physical system
Pointer (computer programming)
Trail
Computer animation
Boom (sailing)
Right angle
Loading...
Feedback

Timings

  508 ms - page object

Version

AV-Portal 3.8.0 (dec2fe8b0ce2e718d55d6f23ab68f0b2424a1f3f)