Merken

Metrics-driven development

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
if it's not quite good morning yet so is afternoon and and still stuck the inferences that time and that a lot of coffee there in some sleep so little jittery it I have to pee a lot but a mostly wake and so when I'm I'm I'm the back and engineer for Spotify and I in this
simplest don't listen and I am also and vise
chair of the PSS along with the new music and so you know if you are a PSS member and and quick announcement there is that his staff members meeting on Thursday at 6 pm forget the room but it's on the on line spectral and if you're interested in becoming a case of memory she come talk to me and last and I'm also the founder of pilot is different and I'm 1 of the main like lead organizers of the global organizations In
you of the and so that the
stage and who hearings actually clothing maybe it's better to go who has not heard of highly OK those handle be down at the end of this paragraph good and so this is a mentorship group for women in the Python community in opened to women in France open everyone and essentially where a loose group of new and
in various locations and and 1 on every continent except Antarctica which is my new missions to get in there to cancer upon and both what we do is then we have Python workshops and speaker events happen nights coffee that everything around Python and learning Python in development general and we welcome all experience levels so I think there is this 1 here in Spain and I think this Barcelona and they're highly encourage you to check out and I also have a talk
another 1 on Thursday afternoon at 2 30 forget our future method education some room and and that type the about priorities and and the more in-depth talk about doing and how the work is actually going on and the effects that we're seeing in the work that we still have to do and so on who here has not heard of diabetes and times e to the ends you need more coffee like me are it's like it might be and price of the stock and I will 1st give a quick
introduction to and to Spotify and over an overview of how we use data and I'll go into
and sort of how we use metrics and how we came about to implement that implementing them the agile nowhere in essentially what was learned along the way 119 implemented them to the bigger picture the company is about will give you a link at the end of this presentation with the blog posts on the slides so you can just watch me rather than a computer tablet or whatever and right so but
this is what you to take away is metrics and tracking is is super funny but but should you track all of it everything that moves and we as developers and we check everything like website visitors referrals how folks user services of servers or even out and we have a lot of tools at our disposal like a new relevant prof I do 1 would accentuate the duty a bunch of other things that we can track ourselves like steps and sleeping patterns exercise calories consumed breathing here growth rate and like everything that you can tracked rate on and maybe hopes to get some insights just to feel better about ourselves and that if this should you measure everything and very easy to get lost in that that the forestry and use it or lose the meeting and easy easy to lose the understanding of why you're measuring in the 1st place
so starts background information about Spotify and service on the same page and also is how we use data and so spot
find streaming music service and we update a local this is the not the correct agreement but I haven't gotten interior and so we better launched in 2007 and then we came to Spain and some other parts of Europe and and and in 2011 we came to the US and we were in about 50 countries we have over 20 million paid subscribers and 50 or 75 million active monthly active users we have over 30 million unique songs not including compilation albums and such and we had about 20 thousand songs a day we also pay about 70 to 80 % of our income to rights holders telling about 3 billion dollars to date and while I work in a very small office elements instances go with about 5 other developers and our main injuring offices are in Stockholm and in New York and with a lot of data machine learning in Boston it's so as you can imagine and that's Spotify data is quite important and these numbers that you see here only about a month old and they I have to check every time I do presentation like this because they are always growing and growing fast so attractive user data like sign-ups and logins activity within the application itself even tweets like the good and bad and the ugly and we have and we also tracks server generated data including requests various services services response times response codes among the million other things and each on some wet and they want to collect how and when and how they will consume such a data the only analysis that run thousands of you do jobs today to glean insight from user activity answering questions like how many pizza Scarborough's we have at this moment in time or was this partnership financially beneficial to us engineers behind and platform and watch usage rates tillægger Web API is we have we watch login failures feature usage 600 and we also have a data scientists in machine learning and analyzing wasn't behavior of music metadata and trends that power the recommendations behind their features and use of actually started to analyze actual Lake audio signals and the sound of a song to pick up like John and beats for mineral instruments playing a yeah and it's actually quite difficult to kind of pick up a few things from the audio signal itself like and that the mood like classifying defined mood of a song and that but that's the stuff that were sort of doing of sporophytes and on and scratches the surface and what we collect and what we pay attention to and we do use various technologies and related to data including Hadoop as well as the standard PostScript elastic search and all the user data sets in Hadoop on which we run jobs against using our own Python library or crunch scolding archive and we also use sparked find an I've heard of excellent people use an Python with sighted Learning in and can and of also discovered recently that we have our own IPython notebook server set up so that's pretty and on the back-end side and some of our service actually actually gets part inelastic searches worry have to buy a set up and that the majority in and of service activities actually in a homegrown system of course and what we have open source that it's called fast Fourier FSW d In some written in Ruby and and get was all set up with all this technology and I'm really embarrassed to say that's my team did a lot of development in the dark and we're not tracking anything and we do not know how successful our future integrations worm and we had no clue how back and services and that we maintain how they're holding out and I do want to make note that a lot of squads in Spotify do track what did you do pay attention to all this were just sort of the black sheep consisting of partly because we're at 9 hours behind Stockholm in 3 hours from New York so the story on about himself discovery and how to become how he became a better more effective team and we did this by an capitalizing on understanding around it and not everyone can be data scientists mathematicians statisticians and analysts whatever and their ranking graphs and
why it's important when and 70 % of our users can login for whatever reasons and so this is a story of and and finally developed and adapted to use have logging in metrics and so you
might know that spot I'm very public about using an Agilent and the action of a few videos on YouTube that are very very often very entertaining and also nicely done that I highly encourage you to go but I'm 1 key aspect of agile is iterations and we certainly iterate over product you might need might be as annoyed as I am when you open up the sporophyte Kleiner desktop and has a little blue banner asking you to update incomes like prematurity will de grade so about the agile approach and better and we also iterate over ourselves as as a team of individuals squad trying to find what works for for us as a company and as as a squad and everything in between only last year with my squad began participating in an internal programs very corporate speak called effective quite serpents squad circle and In its purpose was to hone in on the squad itself how we can be more effective and actually
found that and to be very very beneficial for myself and for the team so what was essentially a multi challenges and to figure out the team's current condition and essentially not checking anything not knowing what going on and comparing it to the desired that and condition condition in terms of delivering a product of Feature Service whatever that were meant to answer the following selection and may sound very light and project manager business-oriented but I'm a fan of very useful in pointing metrics and I also find myself and incorporating this thought process when talking to other teams non-techy intact about like diversity and stuff like that like measuring our diversity initiatives so it's very and widely applicable but so main
almost finer targeted condition as a squad hello where you want to be in and certainly difficult to establish a goal without context that an understanding of where we are now and so to figure out and a baseline we sat down an answer a few questions as a group and the 1st question and what do we deliver seemingly easy question yet myself and the squad and initially struggled to answer this right away it does laden roll-off at times so we look at our past enlisted integration projects and in that we delivered and services that we currently maintains includes an uber last of them somehow on twitter hashtag music and among many others and the most critical and is certainly a Facebook login and sign up and registration that hindered before about 70 % of our user base logs India Facebook and the rest is e-mail log which my team and does not the next question is for whom do we produce said the product or service and you actually defines our work as Spotify and we believe that leadership is meant to convey vision and that's what is meant to implement that vision and in a manner that they choose and there is isn't micromanagement is a lot trust actually and that are lead team defines the direction that our squad takes time so there certainly 1 of our customers and also there are many and integrations we've done a lot of external partners thankfully the quite a bit shielded from direct communication but and that makes business business development team and indirectly at the partners themselves are customers as well but who depends on us who actually uses our work product or service or whatever so yes 70 per cent of users you can log India Facebook and safe to say it's a pretty interval and system to Spotify platform and we certainly can't fuck up when they're on and Facebook makes breaking changes to log protocol or API which they often do unannounced before and I've had to life patches service because of this book but there are some other teams and then the company that plug-in to our system 1st social aspects like and sharing and to Facebook within the platform on and the next question is about expectations about what do our customers actually expect from us and try to answer this question occurred to us that we never act never actually ask them whether expert but their expectations and so we did we want to know exactly what was important to them with what we deliver and was it on time delivery was it predictive versus productive and do they expect solutions to problems that there may be no existence and what were the expectations on quality and usability in other measures where the expectations it with how we work as a squad did they want updates on progress and problems and so we can ask all of our customers right we have 75 million customers in and expectations of the difference among the various customers for the team internal teams expected our Facebook service to be reliable and scalable business development 1 has to be clear on what we can see feasibly implement which is definitely hard for web developer to adequately say a problem say how long something will take friend from and it's safe to assume that users will want to log or sign up the face but if they so choose in an effort to just work and so moving on the at the not last question simply and do we meet those expectations had we had we know we met those expectations and this is where we sort of stopped in our tracks right we can no we don't know a fire services can handle the extra load or if slash when users can login and more how many users of activated Spotify with an and of those does the experience actually work so and being a people and have an affinity for attacked and animation and we naturally implemented
a technical solution so on feedback loops and I'm
very generic term just text and make a feedback loop is that to understand against the feedback given 1st client and remain feedback loops and that we chose was metrics and we are 1 of those snazzy looking dashboards kinda graphs individuals using the latest technology will polity obsolete tomorrow but in all seriousness we
wanted intermediate visual of various metrics and but what do you want to see what questions we we want to answer someone with the idiom come
and you throw spaghetti on the wall the see what sticks and that's why the brainstorm for a while and trying to come up with questions that we like the answer answer some ideas included sign up the Oslo abandonment and Facebook connected users the percentage of total users in trend over time and % of users that have signed up to Facebook per hour day week we we need to know what the frequency should be in Facebook related areas which a lot of daily active users by partner feature and registration subscription rate referrals by partner and nobody can use in my partner and we even have like we really wanted like a squad focused Twitter when search for like you spot find the basis see what people were complaining about that like neither override party include could like C analogs and we want to know about sending during issues and the crosstown by an internal requesting service teams the group these metrics and and into buckets like usage system health business performance and is like it's eventually and became the Earl are becoming the on dashboards cycle through a big office monitors like everyone
to have but we also create a
few new processes and based on these questions when the process and reviews are progresses squad and everywhere retrospective we will look at a couple of metrics that deals with the squad performance like bugs close in the past sprint period and we'll also judge of this metric and if we want to continue seeing this metric if we can actively improve upon in like and maybe we only close to add to bugs this week but because it took us 2 days to knowledge 1 but background and in what is new and if any measurable items centrally governed x retrospective and another is to have gold targets and for every innovation project at medium for example we will noblest successful when we have with this integration we have x amount of new users within the 1st 2 weeks and it's true that this sort of goal line can only be judged and based on historical user acquisition numbers and so it is the so we decided to do some work but and this will feed into a retrospective especially once the project is complete like how did we do and we also have a few oppose integration questions and for business development folks ask of our external partners on behalf of this 1 like an understanding their responsiveness how are developer tools are and if they if they met the goals of and then we may and have may think an integration was like super successful but on there and not so much which is definitely happened to us before so and going caring
about metrics for a and
since the beginning of the year so certainly it is only the beginning for us and that allows us to iterate and give us a hard look at what we track
and wine you can track everything that moves that will you get inundated and certainly and so so if you count every single leaf of every branch of all the trees in the forest and so how can you tell what's important and what's just noise and so this goes back to understanding your customers expectations essentially boils down to business volume happy maintain and improve upon business value of the service a product has counting every Facebook but connected user help us to better ourselves so I'm thinking about implementing various metrics of our feedback loops and I came across various questions to help me see the forest for the trees so creating a new metric how do metrics
mapped to business calls and do we lose money there's there's so much money if and only if the Facebook's laughin services enough and how do you prioritize different goals and if we want to drive what's most important does that mean that you're going to neglect others are just allocate time by priority as this new integration project more important to pay attention to you than other ones that's fine and if it is but you just have to prioritize How can we create dashboards actually actionable what is the goal and more importantly how can we drive that all men are just men saying 0 look Facebook site services down that's go have lunch when I'm
representing metrics and how do we correctly
measure what we care about have to break out the old statics restitution book whatever but to like understand like how to best represent all metrics that you take and we have so many tools to help us create engages counters meters histograms timers but what representation is best for that question or metric actually consuming them how often do you check on metrics and dashed costs never looked at which is a common problem I found my team and they just be become like background noise and so how do you make dashboards more visible more in your face I should someone be responsible at once a week to like a goal in you make them more visible by slapping from a TV monitor which often 1 is not entirely work especially the train from images can be or perhaps you and have e-mail snapshots sent out to the team but maybe there are automatically filtered away for you're like me and auto archive all unread e-mails and being and that introspective and the thing is that we don't reach 100 % over goals we need to assess the difference white is that it exists is even solvable if you look at dashboards what actions are we actually going to take and even create a dashboard if the goal on alert isn't set up or if no action will be taken and probably not what about unknowns what is unknown and doing and we know that x amount of I users have connected their accounts to Cuba but how many don't use it because the driver has an Android phone or the driver just isn't aware of this sort and how do you approach those unknowns are you comfortable that with the memory is even worse than to explore them to bring back to the
slide and ultimately the goal and and answering these questions is to give us both of them ashore decision making cycles as well as a more informed decisions about am strategy in partnerships it's super easy to get lost in the forests and doesn't help us that we can get all this instant feedback that all these visualisations just look awesome but in essence an awareness is placing current values in historical context Odyssey patterns developing how long on average does it take for the team to implement a new integration your customers ourselves expect a shorter turnaround time
and do you wish to just be able to properly estimate the time and work on and the work that takes for such a project and or maybe which internal team do we have to educate about realignments on our service and the other 1 here and that that loops and this thoughtfully implemented metrics and we can use these goal line similar to create a more efficient teams and will deliver higher quality software instance will get immediate feedback bugs and that we introduced and any system that fails and and the like and but before I answer this question for a rapid and I do need to be a good friend and with all these stuff questions and contexts in your mind and you should get a Henoch stock tomorrow and it's at 11 45 and and things about practical its title practical logging metrics and basically the technical complement to the stock so alright and the answer this question should you check everything interconnected
answer probably only if defined goal you can define an action is and if you haven't met that goal and in the actually pay attention to it you
interconnected but within reason right so I think you I hope you
took or something and I think I have 1 minutes for questions Our
had that we just like go out and go did someone in and you can find me if you have questions now thank you
Inferenz <Künstliche Intelligenz>
Verschlingung
Reelle Zahl
Computervirus
Computeranimation
Whiteboard
Selbst organisierendes System
Front-End <Software>
Festspeicher
Gruppe <Mathematik>
Stab
Punktspektrum
Gerade
Computeranimation
Offene Menge
Güte der Anpassung
Gruppenkeim
Baum <Mathematik>
Computeranimation
Soundverarbeitung
Datentyp
Ausnahmebehandlung
Softwareentwickler
Baum <Mathematik>
Ereignishorizont
Computeranimation
Übergang
Rechenschieber
Metropolitan area network
Web log
Linienelement
Kombinatorische Gruppentheorie
Binder <Informatik>
Wald <Graphentheorie>
Quick-Sort
Computeranimation
Implementierung
Graphiktablett
Web Site
Linienelement
Weg <Topologie>
Bitrate
Computeranimation
Homepage
Energiedichte
Dienst <Informatik>
Weg <Topologie>
Mustersprache
Server
Information
Softwareentwickler
Einflussgröße
Momentenproblem
Zahlenbereich
Kartesische Koordinaten
Element <Mathematik>
Kombinatorische Gruppentheorie
Login
Systemplattform
Computeranimation
Metadaten
Schwebung
Prozess <Informatik>
Flächentheorie
Gruppe <Mathematik>
Notebook-Computer
Endogene Variable
Programmbibliothek
Elastische Deformation
Response-Zeit
Softwareentwickler
Algorithmische Lerntheorie
Analysis
Leistung <Physik>
Linienelement
Logarithmus
Open Source
Kraft
Wurm <Informatik>
Sampler <Musikinstrument>
Physikalisches System
Bitrate
Quick-Sort
Packprogramm
Office-Paket
Integral
Dienst <Informatik>
Menge
Twitter <Softwareplattform>
Rechter Winkel
Mathematikerin
Mereologie
Codierung
Server
Baum <Mathematik>
Instantiierung
Standardabweichung
Prozess <Physik>
Kreisfläche
Linienelement
Gruppenoperation
Iteration
Strömungsrichtung
Biprodukt
Term
Computeranimation
Videokonferenz
Gradient
Multiplikation
Datenmanagement
Fächer <Mathematik>
Trennschärfe <Statistik>
Konditionszahl
Projektive Ebene
Optimierung
YouTube
Schlüsselverwaltung
Baum <Mathematik>
Rückkopplung
Subtraktion
Facebook
Bit
Mathematisierung
Gruppenkeim
Login
Systemplattform
Computeranimation
Richtung
Weg <Topologie>
Erwartungswert
Skalierbarkeit
Arithmetische Folge
Existenzsatz
Gruppe <Mathematik>
Affiner Raum
Softwareentwickler
Maschinelles Sehen
Chi-Quadrat-Verteilung
Einflussgröße
Videospiel
Expertensystem
Protokoll <Datenverarbeitungssystem>
Benutzerfreundlichkeit
Physikalisches System
Biprodukt
Kontextbezogenes System
Quick-Sort
Integral
Patch <Software>
Dienst <Informatik>
Twitter <Softwareplattform>
Registrierung <Bildverarbeitung>
Konditionszahl
Web-Designer
Projektive Ebene
Baum <Mathematik>
Generizität
Rückkopplung
Client
Rückkopplung
Linienelement
Ungerichteter Graph
Baum <Mathematik>
Computeranimation
Einfach zusammenhängender Raum
Facebook
Total <Mathematik>
Linienelement
Pay-TV
Gruppenkeim
Vorzeichen <Mathematik>
Physikalisches System
Bitrate
Frequenz
Computeranimation
Office-Paket
Metropolitan area network
Dienst <Informatik>
Flächeninhalt
Twitter <Softwareplattform>
Vorzeichen <Mathematik>
Registrierung <Bildverarbeitung>
Basisvektor
Dreiecksfreier Graph
Automorphismus
Analogieschluss
Funktion <Mathematik>
Prozess <Physik>
Linienelement
Zahlenbereich
Turing-Test
Frequenz
Quick-Sort
Computeranimation
Integral
Programmfehler
Endogene Variable
Projektive Ebene
Softwareentwickler
Gerade
Netzwerktopologie
Einfach zusammenhängender Raum
Rückkopplung
Dienst <Informatik>
Erwartungswert
Facebook
Wald <Graphentheorie>
Linienelement
Verzweigendes Programm
Geräusch
Spezifisches Volumen
Biprodukt
Baum <Mathematik>
Computeranimation
Portscanner
Betriebsmittelverwaltung
Metropolitan area network
Dienst <Informatik>
Facebook
Web Site
Linienelement
Systemaufruf
Projektive Ebene
Computeranimation
Integral
Eins
Rückkopplung
Subtraktion
Wellenpaket
Weg <Topologie>
Gruppenoperation
Selbstrepräsentation
Geräusch
Computeranimation
Hydrostatik
Mittelwert
Endogene Variable
Mustersprache
Meter
Visualisierung
E-Mail
Hilfesystem
Bildgebendes Verfahren
Einflussgröße
Wald <Graphentheorie>
Linienelement
Strömungsrichtung
Kontextbezogenes System
Quick-Sort
Packprogramm
Gruppenoperation
Integral
Entscheidungstheorie
Portscanner
Rechenschieber
Codec
EDV-Beratung
Druckertreiber
Histogramm
Meter
Festspeicher
Dreiecksfreier Graph
Strategisches Spiel
Direkte numerische Simulation
Rückkopplung
Linienelement
Weg <Topologie>
Gruppenoperation
Physikalisches System
Kontextbezogenes System
Computeranimation
Programmfehler
Loop
Dienst <Informatik>
Software
Gerade
Baum <Mathematik>
Instantiierung
Rechter Winkel
Weg <Topologie>
Zeiger <Informatik>
Baum <Mathematik>
Computeranimation

Metadaten

Formale Metadaten

Titel Metrics-driven development
Serientitel EuroPython 2015
Teil 66
Anzahl der Teile 173
Autor Root, Lynn
Lizenz CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
DOI 10.5446/20154
Herausgeber EuroPython
Erscheinungsjahr 2015
Sprache Englisch
Produktionsort Bilbao, Euskadi, Spain

Technische Metadaten

Dauer 23:59

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Lynn Root - Metrics-driven development At Spotify, my team struggled to be awesome. We had a very loose understanding of what product/service our squad was responsible for, and even less so of the expectations our internal and external customers had for those services. Other than “does our Facebook login work?”, we had no understanding of how our services we’re responsible for were doing. How many users actually sign up or log in with Facebook? How many users have connected their Spotify account with their Uber account? Do folks even use Spotify with Uber? With a 2-month challenge period, my squad and I focused inward to establish those unanswered questions and to establish feedback loops and always-on dashboards. This talk will tell the story of how we chose which metrics are important for us to focus on, what technologies we have used and are using, and how we’ve iterated over our feedback loops to fine-tune what metrics we care about.
Schlagwörter EuroPython Conference
EP 2015
EuroPython 2015

Ähnliche Filme

Loading...