We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

The intricate art of making your (internal) clients happy

00:00

Formal Metadata

Title
The intricate art of making your (internal) clients happy
Subtitle
the story from a Python-centered Infra team
Title of Series
Number of Parts
112
Author
Contributors
License
CC Attribution - NonCommercial - ShareAlike 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
If you have ever worked on an internal company project, you may feel it deep in your bones. Let’s say that you discovered a need for a generic technological component in your organization’s tech stack. You identified stakeholders, gathered requirements, and started agile iterations on providing it. Then comes a day when you can show the MVP to your internal client! Yet… the client has lost his interest: maybe right now he says that he has already come up with his own temporary solution and he has no intention to switch to another one? Building internal products differs from commercial ones - there is no flow of cash and your clients are fully transparent. In this talk, I would like to share with you my experience and tips connected with developing such internal tools and standards. All of this from the perspective of a member of the Machine Learning Infra team that is delivering its solutions to a rapidly growing ML department in a company whose product is used by 300 million unique users per month. But let’s be specific! I will talk about: - Common pitfalls and try to dig up the reasons for why they happen when developing internal solutions - How one can approach delivering tools (spoiler: pilot programs, guilds, and more!) - Learnings from introducing such approaches (what worked, what didn’t) in our case
29
Client (computing)Multiplication signInheritance (object-oriented programming)Lecture/Conference
Process (computing)Software engineeringArithmetic meanSocial engineering (security)VotingPerspective (visual)Computer animation
Perspective (visual)BitWordClient (computing)Computer animation
Client (computing)WhiteboardService (economics)Standard deviationArchitectureWordBit rateComputer configurationMultiplication signMoment (mathematics)Client (computing)Wave packetDifferent (Kate Ryan album)Uniqueness quantificationVirtual machineService (economics)CASE <Informatik>Latent heatDependent and independent variablesContent (media)Pattern languageDomain nameCuboidScripting languageStandard deviationTelecommunicationComputer architectureSoftware developerDemonQuicksortInternet service providerVideo gamePower (physics)Term (mathematics)Projective planePlanningSpeech synthesisDataflowDiscrete element methodDirection (geometry)Machine learningArithmetic progressionSet (mathematics)Product (business)Total S.A.Computer animation
Moment (mathematics)Disk read-and-write headCategory of beingComputer animationLecture/Conference
Standard deviationMathematicsTelecommunicationContext awarenessVirtual machineClient (computing)Multiplication signStandard deviationMathematicsSound effectTransport Layer SecurityEntire functionLatent heatMetropolitan area networkArithmetic meanAreaComputer animation
Software developerLibrary (computing)Data managementDifferent (Kate Ryan album)Extension (kinesiology)Online helpLattice (order)Standard deviationCASE <Informatik>TrailData managementTask (computing)AreaVirtual machineInstance (computer science)Multiplication signLatent heatTelecommunicationForestThomas BayesState of matterLaceMereologyComputer animation
TelecommunicationSoftware engineeringField (computer science)TelecommunicationMathematicsStandard deviationParity (mathematics)Decision theoryDifferent (Kate Ryan album)Multiplication signClient (computing)Process (computing)Focus (optics)Computer animation
Mathematics1 (number)Category of beingFilm editingTelecommunicationComputer animationPanel painting
Standard deviationMathematicsPoint (geometry)Point (geometry)Multiplication signTelecommunicationCASE <Informatik>Library (computing)Standard deviationDecision theoryBit rateMathematicsLatent heatComputer animation
CASE <Informatik>Client (computing)Real-time operating systemFeedbackPoint (geometry)Standard deviationProjective planeVirtual machineEndliche ModelltheorieDemo (music)Level (video gaming)Right angleWave packetData structureBitWordWater vaporMereologyCopyright infringementComputer programmingObservational studyVideo gameComputer animation
Demo (music)FeedbackStandard deviationTemplate (C++)Form (programming)Frame problemMultiplication signStandard deviationFeedbackProjective planeCategory of beingState of matterDiscrete element methodLecture/ConferenceComputer animation
CASE <Informatik>InternetworkingSynchronizationProjective planeLattice (order)Client (computing)Connected spaceMultiplication signMathematicsWave packetDirected graphComputer animation
Term (mathematics)Source codeLibrary (computing)Integrated development environmentPoint (geometry)Projective planeCASE <Informatik>Group actionRevision controlGame controllerReal numberMachine codeVideo game1 (number)MereologyTelecommunicationPerspective (visual)AreaOpen sourceMultiplication signCuboidResultantSerial portWordRight angleLecture/ConferenceComputer animation
TelecommunicationStandard deviationDecision theoryNP-hardTelecommunicationNP-hardData managementQuicksortMultiplication signError messageVideo gameSlide ruleTerm (mathematics)ResultantCollaborationismLattice (order)GodForcing (mathematics)Staff (military)Image resolutionInternetworkingAddress spaceEndliche ModelltheorieSource codeBasis <Mathematik>Graph (mathematics)Computer animation
Game controllerFeedbackOpen sourceMultiplication signPower (physics)Computer animation
Point (geometry)Set (mathematics)Client (computing)ResultantLecture/ConferenceComputer animation
Direction (geometry)SoftwareOpen sourcePoint (geometry)Position operatorLecture/Conference
Bit rateNumberGraphics tabletPosition operatorMathematicsLibrary (computing)Internet forumoutputLimit (category theory)Group actionLecture/Conference
MereologyArithmetic meanImpulse responseRight angleDomain nameMoment (mathematics)BitGroup actionLatent heatFront and back endsAreaRepresentation (politics)Perspective (visual)Virtual machineObservational studyFocus (optics)Lecture/Conference
OvalPoint (geometry)Automatic differentiationMultiplication signMultiplicationProduct (business)Different (Kate Ryan album)Single-precision floating-point formatArea1 (number)Bit rateSpreadsheetResultantLecture/Conference
Uniformer RaumSpreadsheetLecture/ConferenceXML
Transcript: English(auto-generated)
So today I would like to start by asking you a question So do you remember this time when maybe you were four years old? You're really like dinosaurs and you also liked drawing
So maybe one day your parents asked you Can you draw a giraffe for us and you said like yes, I will do so So you ran into your room, you took the crayons, you took a piece of paper And soon the masterpiece was ready
So then what you did, probably you ran back to your parents However, maybe they were no longer there Which was confusing So you went looking for them around the house and you found them in a kitchen And they took the drawing and said, oh yeah, that's really nice
Thank you for that However, you have never seen the drawing being put on the wall in the living room For all of the guests to see Which was confusing to you, right? So this thing that you got back then is also something that you may feel right now in your job
For example, as a software engineer And this is also something that sometimes I experience So my name is Paulina And today I will tell you a little bit about the intricate art of making your internal clients happy
And all of this from the perspective of the infrastructure team So what do we have on our plate today? First, we'll start with explaining ourselves what this internal client is Then I will tell you a few words about my team
Then we'll proceed to establish what the infra team is And then we'll move on to the challenges And something that you are probably most interested in So, solutions And here I will also mention what worked best for my team
And at the end, the summary Okay, so we are probably familiar with this pattern So you have your company In your company you have some team X And this team X provides some sort of product Some value to somebody outside of the company
Then this person is your external client But it may also be the case that your team does not provide value for somebody outside of the company Rather, your team works for another team And then this team X becomes your internal client
You may wonder, what are the differences? Are there any between external and internal clients? So first up, usually external clients ask you to do something They want to get some value, some product But with internal clients, it may not be the case
Internal clients, basically with them You are the one responsible to learn what they need You need to create this value for them And then try to convince them that, yes, this is something that will benefit you
Next up So with external clients, it is pretty straightforward Because they pay you for the value that they get But with internal clients, it is not the case Obviously, the employer pays you But there is no direct cash flow between your team and your client's team
And what comes with that? When the external client asks you to do something and they pay for it Basically, they will be most likely interested in the progress
But with internal clients, it may not be the case They may be more focused on their own priorities To deliver to their client than learning about some tools that they may use Or maybe not
So a few words about my team I work at Brainy, where we deliver our products to over 300 million unique users monthly And these users come from all over the world So Europe, the United States
But also, for example, from India or Indonesia And at Brainy, my team is called Machine Learning InfraTeam And there are eight people Which, as you can see, very diverse roles I am a Machine Learning Engineer in the team
And my team belongs to the Department of AI Services Where we have five teams in total And this makes around 50 people on board So, as I mentioned, I worked in the Machine Learning InfraTeam But what this InfraTeam is?
The story starts with our CTO Who had worked with a couple of different companies And he observed a pattern That whenever you have a given domain It usually turns out that the teams in the domain develop their own toolboxes
Which may be, for example, a set of scripts that make their life easier Or maybe they try different tools and find which of them work best And they collect all of those things into their own toolbox
But what the CTO also observed Is that you can take in other efforts So have a dedicated InfraTeam for the domain And then this team can see what are the common patterns between the teams
And develop a toolbox that will be generic enough to fit many teams But that will also bring all of them the value So what happens next? Basically, this toolbox can be given to each of the teams And the teams can use it
And what is there for us? Like, what do we want to get from this? Basically, those teams can focus only on their own work Their own priorities And they don't need to reinvent the wheel
And what comes with that is that, you know No reinventing the wheel, no time spent on that So the costs go down Basically, in the end, it benefits the company And as you might expect, my team follows this pattern
So we are in constant communication with each of the teams So we get the requirements, we get feedback from them We learn about their projects But what they get from us is value It can be like a standard
It can be specific tool or maybe something else And if you wonder about specific responsibilities of my team As you may expect, first it is research Because we need to learn about different options That we want to try And out of that, we may propose standards
And sometimes we are also developing our own internal tools Most likely in Python Then when the standard is in place We can also help with the adoption Or we can help with architecture design
Or in general, we can advise the teams So now, let's take a moment to consider What may be a challenge in such a team Think about it for a moment
Okay, maybe something already popped up in your head When I was thinking about it, quite a few things popped up Actually, a lot of things
And when I was thinking more about the topic Those things fell into three categories It was timing, communication and adoption So let's see what exactly is here When it comes to timing
I think it is important to consider the context of my team So as I mentioned, right now we have 50 people on board But back in January of 2021, it was more like 10 people And in our company, the first machine learning initiatives Date back to around 2016
And my team was created in April, let's say I think around this time of 2021 And as you might expect, there was a rapid growth In the department happening back then
So throughout the recent months What we faced when it comes to timing Basically, we were often too late with our solutions Because some of the teams already came up With their own ways of doing things
So it created this feeling of being behind With what the teams do Sometimes we were in time But we were in talks with a specific team And they said they will need a given solution In a few weeks
As a specific four weeks So we proceeded to work on that And we were almost ready, like one week to go But suddenly we learned that the priorities of this team So our clients changed And they no longer need this solution
But there is also the third scenario Which is too early Sometimes we tended to anticipate The needs of our clients So we were asking them Okay, so what will you work on in two months?
And they said something, something And we will need such tools Great for us We have time We can develop such a toolbox for them But sometimes it ended up that those tools Seemed artificial Because they were just too early
There was nobody right there to take it And use it immediately It may be a few weeks or even months Before such a tool or standard will be used So that's about timing But let's consider the challenges of adoption
And let's be specific here Let's imagine a scenario that your team Developed an internal library, internal tool So as to make the management of EC2 instances on AWS easier
Especially for the specific case of machine learning Okay, so you may think that This is something that may be useful For everybody in the department
But it turns out that not always People were not very eager to immediately adopt it And here we need to consider a few questions So first, were they aware of it? I think most of them were At least to some extent But still there were people who were not
Maybe they did not attend meetings Where we talked about the solution So yeah Then they could not adopt it Even if they needed it Because they were not aware that it existed
But if yes Then we are on the right track But the next question is Do they trust it? And here the situation may be You know, it may be that They believe that they have more experience in their area
And they think that maybe this solution that we propose Will not work for them Or maybe they have their own tool already in place They don't want to switch But if they trust it Again, very good
But another question is Do they have time? I believe it is crucial Because if they have their own priorities Their own tasks at hand Especially when the time is very tight for them They would rather invest in delivering their own tasks
Than learning about some tool May benefit them in a few weeks But maybe not Who knows But if they have time Great So is it the perfect scenario?
Not always Because sometimes they may not be involved enough To learn about the standards To try them out in their own work Or they may need help with getting started However, they never reach out for such a help And we may not be aware
That there is a person who would like to check given solution But they just need even like 15 minutes intro to that tool And that's about adoption Let's consider communication I think we all know that communication is the key
Whether you are a software engineer Or work in a completely different field When it comes to changes that we face there The obvious one is Basically deficits in communication And it may be from both sides So we may not reach out to everybody
Or we may not talk enough about the possible solutions But on the other hand Our client may not let us know about the parity changes All of the things that they suddenly need The other thing is something that I found quite interesting
And this is misunderstanding the role of intro team And what comes here? People do not know why we need standards What are possible benefits from that?
And sometimes they may feel that we as infra team Are pushing some solutions on them And then our team is being seen rather as an obstacle And it all creates this thing of us versus them Which is not very healthy for the department
And another thing here is that sometimes There are people who like to voice their opinions So for example we as infra team Would like to propose a standard So we ask the community Hey we have such a tool Is it something that will be okay for you?
Or maybe you see some problems Possible problems with that And people may not voice their opinions For example because they may not have time Again they may be focused on something different And if they want to voice their opinion Yet they don't have this time Later it may turn out that they are not okay with the decisions made
Okay it all seems like there are a lot of different things that may go wrong But I believe that instead of focusing on Challenges, problems
It is better to focus on solutions So I guess that right now you expect magical solutions But I will not give you magical ones Rather possible solutions So something that we tried And I can tell you if it worked Or maybe not And again here we will revisit the same categories as before
But I will add one more Which is the category of involvement I believe it somehow emerges from the things that I already mentioned So let's not wait Let's jump right into it When it comes to timing
We all know those scenarios And I believe some of those can be solved with communication Which we also tried obviously But as a team we realized that it is not enough It is not enough
And we need to try something else And what we decided to do is to embrace it Embrace the fact that these situations used to happen Happen right now and will happen But we may take advantage out of those First up
If there is a team that already developed their own solution For a given problem Maybe their own library Maybe another tool They tried something, they liked it What we can do is take it and evaluate Because maybe this solution is something that can be distributed
To the other teams And may benefit also them Obviously we need to check if this is generic enough Maybe we need to tweak something But this is a great starting point
And it also feels more organic Once you are using something that one of the teams already tested And something that worked for them Okay but this is only one thing The other thing is that is also something that we need to remember about
Is that okay we will deliver something that will not be used for the next few weeks Or even few months But maybe somebody will come in those few months And check out the solution on their case And maybe they will like it then cool
But if not it is okay to iterate on the solutions If we get feedback after those few months That hey this does not work for us Can you change it? We can do it We can iterate on the solutions Find some new tools that will work better
Or maybe Maybe in the market there is this new shining tool That is a lot better than something that already is a standard for some time Then let's check it Let's see Because maybe it makes sense to iterate and change the standard in our company
When it comes to adoption Again here is consider specific case So at some point our team was supposed to find a solution That will allow machine learning practitioners to easily train machine learning models
So we did research, we contacted with our clients And it turned out that stage maker pipelines is something that seems okay As a tool to be used
And we also as a team developed example project structure That we proposed to the teams to be used For such machine learning training pipelines projects Okay but what we could do
We could take that and throw at our clients And our clients most likely wouldn't be happy about it Maybe they would not adopt it What we decided to do instead are four things
So first we decided that we need to dive right into it What it means It may mean that we need to do a demo for our client So that they know what this tool is about And they can get started from that point
This may work for some cases But in our case it was not enough We decided that we need to do something more And that's when the pilot program came What it means Basically we took one of our team members and said that
Okay you need to go now to our client and work with them for two weeks Help them with the adoption Be another extra pair of hands that can help them And we did so
We performed the pilot We helped the team to develop More like incorporate the solution quicker They were quite happy with that But what we got out of it is not only that some team adopted it But we also get a lot of feedback in real time
As the pilot program was going on We learned about all of the things that they struggled with In this topic but also in other topics And we got a lot of insights in what their work is like
What are their projects And it is also important to know what your client is up to Okay this is one thing But what else you could do You could also infiltrate them What it means
If you have a colleague of yours in another team And sometimes maybe you chat over coffee You may also talk a little bit about your projects The standards that you are proposing Maybe your colleague will tell you Okay so these are our projects
And you may find some common points And maybe your colleague will decide to try this solution that you proposed And then if they try it And they like it It may happen that they will promote this solution in their team So there is no need to
You go there Rather it happens more naturally Again hear them out So this is about gathering feedback and requirements But what is important here is to act
Act within the time frame on the feedback that you got You can also indoctrinate them And whenever there is somebody new joining the company Let them first play around with the tools Let them learn the standards
And when they start with it They can then proceed to work on the projects already knowing of the standards in place And now comes the new category So involvement So our case was that we quite early on in our team
Identified that we need to engage more with this internal community And so our natural thought was let's connect with them And what we could do Basically when it comes to connecting
We established sync meetings So every month we are meeting with each of the teams Learning about what they are up to, what they need We are telling them about our projects And we are seeing how we can help them in the future
This is okay, but not very engaging Another idea was to have no lecturing meetings Which is amazing as an idea But when it comes to actually having it It takes a lot of time of the person that is presenting
So you may have your clients tell you about their project But you know, they can do so And people around the department can hear about this project But first this person needs to spend few hours preparing for that
So people are not very willing to do so Okay, so connecting worked okay But not very well So instead we decided to empower the people
And we had two ideas here One was the guilds So a guild is a group of people focused on a specific topic So in our department we decided to have MLOps guild and data science guild
I am a part of MLOps ones So I can tell you more about from this perspective This group is not very big, only a few people But those are people who either work in a relevant role So they are MLOps Or they have experience in this area
Or they are just interested in the topic And what is the result? When you have such an environment It is best place to talk about those solutions And I have seen this in real life Whenever there is something posted in the Slack channel of this group
Some idea to be considered Immediately you have responses from at least a couple of people Which is amazing This is what we want Another thing that we decided to try Is to have our own internal open source solutions
So as a team we started from this approach We should be the ones delivering the toolbox to another team But as I mentioned This team may not be willing to use it Because for example they don't trust it
So at some point we decided this is not the way to go Instead we need to start the project Obviously So maybe have some version 1.0 of a given internal tool Internal Python library But what we do later
Is we give it to our community And we tell them Okay so this is something that we started But now it is yours You can add pull requests You can report bugs and fix them if you want to And you can add feature requests whenever it is needed
And it changed a lot when people learned that Hey right now I can write the code to this library And since I will be using it I know exactly what I want And in our case it worked perfectly Because people felt that right now They are in control of what they will be using
And they are very happy to contribute They are very proud with what can add to this tool That will be later used by them But also by their colleagues And we have the topic of communication again And when it comes to the communication
I believe that there are a lot of books on the topic There are a lot of talks on the topic So I will not mention that I will only tell you about one thing that we tried Which we called for ourselves laundry time What was the laundry time?
So basically we decided that Some things are confusing for people in our department So yeah they are confusing But the error is also like not very clear
There are some things that maybe are not said well enough So we wanted to have this hard questions and honest answers session And how we did it? First we asked everybody in the department
Give us your toughest questions Like whenever you felt like you don't know why the things are the way they are Think about this and ask a question It may be why we need standards? Why we have this strange infra team that forces us to do something or adopt a solution?
Why our priorities are like this? A lot of sorts of questions And once we asked people to do so They really did ask a lot of hard questions
I think there were over 30 of them being submitted And they were very diverse And what happened next? Once we had our internal summit in our headquarters It was the first time that the department met in real life Which was amazing by itself
But also we had this Q&A session Where those questions were answered by people higher in the management And it was amazing Because first you could feel the air become cleaner
When you attended the session more and more People were also following up So they got some answer but maybe they were not satisfied with it So they were asking more questions It was amazing in general And I think people felt for the first time that they are being heard
That their confusions are addressed And that people really care about what is happening in the department So for us it was only three weeks ago So I can only tell you about short term results
But I already see a lot of collaboration emerging from that meeting And I believe more great things will happen And this is also something that I believe our department will be having more of in the future
So maybe in a few months we will repeat the same thing Okay, so I think that right now I owe you this slide of what worked best for us And those are fixed by me And the first one is the pilot programs
Why? Because we got a lot of insights, a lot of feedback from the team that we collaborated with Another one, open source solution Because this is something that empowers the people and gives them the control
And the laundry time, ten out of ten, I can recommend So I mentioned quite a few things today Maybe you remember some of them Maybe there's one thing that sticks with you that you would like to try in your own company
Maybe even if you are working with external clients, maybe some of those points can work for you But in the end, I believe that working with clients in general is an intricate art So it is only up to your creativity, what you will try and what results you will get
Thank you very much Okay, so do we have questions? So do you want to use the mic for anyone who's got questions?
Hi, thank you very much for the talk I was wondering, how do you keep the direction of your internal open source software Because the thing is, if everyone can contribute At some point, someone may contribute to something that might not benefit the other teams as well How do you keep a general direction, let's say?
Okay, so I believe we are in this lucky position that we grew But still there are a limited number of people And as teams, we often collaborate And when there are tools that are used by, I don't know, ten people
Usually those people are in contact with each other So they are also discussing what we want to add And my team is also the one who is responsible for moderation of such efforts So basically, we are happy to take any input
But we may also consider, okay, is this something that will benefit all the teams Or maybe only one of those If one of those, then maybe we can have a dedicated solution just for them And not introduce this change in a library that is used by a wider group
Thank you very much for your talk I'm definitely interested in this being part of a newly developing team at the moment Or a group of teams in my company You mentioned the idea of guilds
I'm particularly interested in this Is this something like having cross-team teams Or is it like having specific topics that people will join into As they like, can you explain that a bit more? Okay, so the idea of guilds came from the backend domain So we already tried guilds there
And we decided, okay, let's see how it's going to work for us And right now in search of guilds For example, the MLOps guilds We have people from each of the machine learning teams in our company And we strongly encourage that to have at least one representative from each of the teams
Because every team has their own perspective on the topic And this is very valuable for us So yeah, those guilds are focused on a specific topic or a specific area For us, we have the areas right now So data science and MLOps But if we want to, we can have guilds that is focused on very specific tools
Even like one of our internal tools Okay, thank you very much Thank you Hi, thanks for the talk I think it's quite common to have multiple product teams Or ML teams in a company And a single infra team dedicated to them
How would you manage the overwhelming growing backlog Like of requests coming from those teams? Okay, yeah This is something that we also find quite challenging Because the teams work on quite the various things
And they need tools at many different points in time That's true So actually this is something that we did like a week ago For the upcoming quarter So we gather all of the possible topics And we ask the teams Okay, these are the possible areas that you told us you will need
So each of the teams, please rate how much impact on you will have this given solution And when we have results from all of the teams We can then have a specific metric that takes in consideration priority of each of the teams
But also in how many teams this priority is somewhere Yeah, and based on that we had a spreadsheet And we sorted it And the ones on the top are the ones that are being addressed in this quarter The other ones will be addressed possibly in the future
Thank you Okay, so thanks very much Paulina Thanks to all our speakers this morning And hope you have a nice lunch And the solution as always was a spreadsheet in the end Yeah, thank you