AV-Portal 3.23.3 (4dfb8a34932102951b25870966c61d06d6b97156)

Hiring Developers, with Science!

Video in TIB AV-Portal: Hiring Developers, with Science!

Formal Metadata

Hiring Developers, with Science!
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Nothing makes or breaks our teams like the members they hire. And so we rate and quiz, assign homework and whiteboard algorithms until candidates are blue in the face. But does it work? In this session, we’ll unpack common interviews and how they stack up with research into predicting performance. We’ll learn to design interviews that work, what kinds don’t work at all, and how to tell the difference, with science!
Statistics Computer animation Software developer Bit
Area Facebook Process (computing) Computer animation Personal digital assistant Software developer Rule of inference Theory Information technology consulting
Perfect group 1 (number) Content (media) Set (mathematics) Line (geometry) Complete metric space Mereology Variable (mathematics) Process (computing) Computer animation Different (Kate Ryan album) Right angle Computer-assisted translation Task (computing)
Point (geometry) Process (computing) Computer animation Natural number Food energy Field (computer science)
Electric generator Computer animation 1 (number) Selectivity (electronic) Branch (computer science) Figurate number Mereology
Validity (statistics) Divisor Digitizing Correspondence (mathematics) Software developer Characteristic polynomial Measurement Field (computer science) Type theory Goodness of fit Process (computing) Computer animation Ring (mathematics) Different (Kate Ryan album) Operator (mathematics) Right angle Software testing Quicksort
Area Multiplication Validity (statistics) Multiplication sign 1 (number) Set (mathematics) System call Measurement Element (mathematics) Mathematics Computer animation Personal digital assistant Different (Kate Ryan album) Mixed reality Right angle Resultant
Point (geometry) Area Group action Wage labour Validity (statistics) Consistency Software developer Multiplication sign Drop (liquid) Measurement Usability Number Arithmetic mean Process (computing) Computer animation Personal digital assistant Different (Kate Ryan album) Right angle Figurate number Family Condition number
Type theory Computer animation Divisor Validity (statistics) Different (Kate Ryan album) Dimensional analysis
Frequency Computer animation Right angle Usability
Optical disc drive Computer animation Numbering scheme Mereology Measurement Resultant
Building Service (economics) Divisor Length State of matter Variety (linguistics) Multiplication sign Resonator Insertion loss Parameter (computer programming) Mereology Rule of inference Usability 2 (number) Revision control Mathematics Causality Semiconductor memory Different (Kate Ryan album) Software testing Codierung <Programmierung> Thumbnail Area Scaling (geometry) Touchscreen Validity (statistics) Demo (music) Constructor (object-oriented programming) Sampling (statistics) Volume (thermodynamics) NP-complete Line (geometry) Measurement Type theory Word Process (computing) Spring (hydrology) Computer animation Software Order (biology) Video game Self-organization Website Right angle Musical ensemble Quicksort Bounded variation Reading (process)
Covering space Noise (electronics) Scheduling (computing) Computer animation Cellular automaton 1 (number) Right angle Writing
Laptop Observational study Code Correspondence (mathematics) Student's t-test Inverse element Number Usability Revision control Goodness of fit Semiconductor memory Computer configuration Selectivity (electronic) Software testing Covering space Algorithm Arithmetic mean Word Process (computing) Computer animation Software Personal digital assistant Radio-frequency identification Network topology Whiteboard
Computer animation Validity (statistics) Personal digital assistant State of matter Code Multiplication sign Similarity (geometry) Video game Right angle Product (business)
Validity (statistics) Observational study Multiplication sign Software developer Bit Graph coloring Usability Goodness of fit Process (computing) Computer animation Bit rate Personal digital assistant Form (programming)
Multiplication sign Feedback 1 (number) Measurement Food energy Category of being Word Latent heat Computer animation Causality Right angle Circle Software testing Object (grammar)
Point (geometry) Process (computing) Computer animation Validity (statistics) Multiplication sign View (database) Constructor (object-oriented programming) Feedback Set (mathematics) Right angle Client (computing) Mereology
thank God and depend on the other hand thanks to all of those things were showed up besides my 1st like a top people like 1 over statistic that all of your way to and there's a lot of jet lag coming out Seattle WA fulfilled a minus Joe and let's talk a little bit about very developments so
I a consultant I work a lot of things companies but and work on there are more their hiring processes which by ships and stuff in 1 of the things that I noticed both companies work in some that the initiatives that and we have a problem with theory and this is a big issue right now how many of you
have at your job a job was that you cannot heard about right there's a lot the big idea as not you know I cases this is not just like I use an extra personality and this is like an exigent threat to your business is a big deal and interview his heart a notion of the lecture I for me and you have received 8 terrible in the there ever attended as you go on like this is done as donor cop ever having given a terrible get back that that get so so you actually you know after that and I think this is funny because you know in big companies you think about like the rules of Facebook's and all these companies that have you 10 thousand developers they're not actually doing that their areas are just those terrible the rest and that to me points to the fact that they're interviewing is in fact of it's expensive and anybody that you have a
new team also has a full-time job right so these engineers you expect to take hours and hours of the day we also have a complete set of tasks and also on the cat's eye so anybody who was applied for your job they may also have another the job of the problem of things going on the like right and if you think that's not your concern remember that the candidates that you really want to hire are the ones that probably already have a job and other places where the line so if you give somebody you know that our homework this can move about right I think they were making it ultimately worse on we're not really doing
anything but my informal sense of how we tend to get an interview is how did I get high right so i've been interviewed a bunch different ways balancing chemical and they will do that 1 variable try something else the so it doesn't do us any favors Watson and I'll tell you now and there is no perfect interview through talk through a lot of things to make and use better or worse but there is no correct per say and I will say that there are a lot of bad the bad are the ones that in large part we're doing right now and the results of that the clicker doesn't work is this so this is all the main contentions owner
makes you want to put in early because I want to think about the reason that are interviews of that is typically we are not measuring what we think are where you have a bad energy when you have enough puzzles when you have an abrasive interviewers you're not measuring the Canada you measuring the interview and the entire point you obviously is to see whether that Canada is doing and so this is ultimately why would a pretty darn good and this is ultimately what you end up accepting back so this is why you have 200 interviews and never offer a job is because you're not much I don't think this is on purpose is bad on purpose what's happening is that we don't have the tools we don't have a mental schema for how to evaluate a further interviews or any good we tend to be an attempts were not coming from nature back so usually it's that made it up so the
good news is that even if we have not invented correct about ways to do interviewing there are other fields that have and
specific ones and then around a lot longer than we psychology what I wanna talk about today
is industrial and organizational psychology is 1 of the major branches of psychology started in the 18 hundreds late 18 hundreds of really came to prominence in 1920 by which the 1st wall war what happened was that psychologists the army needed figure out where to place a million workers clearly 1 new recruits and so they need to come up with a way to handle the process and so they started could fly what they call selection so on included the in this is all references and there's a lot you can look at if you do find yourself wanting to look at primary material that selection is the name of the concept that you want will and so it's cumulative from you cover 100 plus years of psychology and fortunately they have written a lot over the course of 5 generations as it turns out and but so what I wanna do instead is to give you a tool in 3 parts with think about the interviews that you're doing and recovered couple the common kind of troops the things we tend to see in interviews and look at the that once call like prophetic so that no 1 is going the so if we have something we want to match we have con struct is what we call this in psychology loony tells us whether our
measures of right of these bullets do not the books and do not very close to 1 in each other but they all centered approximately on people's at little it's OK that the spread out it's OK that in fact most them wrong because they are measuring the correct concept the and there's a couple different factors to validity things I want considerable here and 1 of them of 1 type validity is the question of whether the thing that the question that you asked corresponds to the concept 1 test so if would say I wanted to test whether you know written if I ask you to list all the digits of pi is that a test of arithmetic not right if I ask I for 5 may not be very question but it is in fact refer to right and the 2nd type of validity they wanna talk about is sort of wider 1 given that I can test arithmetic and does that correspond to a skill but I need you have I I can't answer it would take a job that I'm trying to hire you for these carbon the is that actually about skills for the job the social external validity the native uh and you notice that all the consistently talk about uh a constant you want test you have to know what the balls it's and so this is actually a 1st ring we can diary involved because as it turns out and we probably do not agree on what makes a good developed there is a lot a complication to work field and so as it turns out very difficult for us to say what success even is so think about what quite upgraded operating in you right hopefully
doesn't like this have but what problem does look like is people that you know or people in your team who have been very successful and you think about what you characteristics of those people we've seen more successful and you can't generalize say OK that's a good developed but it's not that is not a new concept of good developer with that is kind of a bag of characteristics some of them may actually be well to whether some has developed summer may not all and so 1 of the things that happens when we start to measure people based on what we've seen from success when measuring all these things we didn't intend to and we get back we're more that that's what our entire team the reliability is
consonant to so validity is when we're centered on the bull's reliability is how close
are measurements are and so this case we in carrots Saturn as long as the measurement result set so just like the 1 in a couple different concepts and here 1 of the big ones that's really important in technical interviewing is that if I give you an interview and if somebody else gives you that same area you should get the same score the if you don't get the same score you not measuring the candidate measuring the inter right and influences I interviewing another day hour interview you bump to stop these things should not impact measures the and that's the inter-rater reliability a is if I take an interview 1 so if I take that interview a 2nd time I should get the same score this call test-retest reliability the all that means is that if there is an element of chance there is an element of you know did I happened following half of the other path Anatoly changes would score again I have not measured me I measured the instruments that measure the random chance that took you know used to be there were hybridized right I and the 3rd Temple reliability reminiscent of this but if I were to have multiple questions yeah those questions need to yield the same result so we go back to arithmetic example if I ask you what's 5 times 5 days some you know that the what's so at times a thank you I would steer 65 times 12 notice it clearly you don't know the right the same for question what happens is we have these questions where there's some other confounding variable in this case we've all memorize a very particular set of multiplications and we're not actually in the had just doing my work this happens all the time interval we measure constructed some people memorize the people not and
all those kinds of reliability and is an interesting 1 point towards approximately the same the point with consistency serotonin the labor this in the in the rest of the talk and I wanna say that and if you give the same answer if you give it the same way if people have a scoring rubric so that it doesn't matter who is there what matters is how you do it right if that's going back objective you will tend to be like so reliability consistency will say consistency breeds rely so number 3 usability hello to figures um me could probably come up with a valid reliable interview for developers and what would look like as having you do 1 of every right if I were to do this
in my arithmetic I would just ask you 1 of every single question I of course is also work right it's tempting sometimes in our in our interviews to be able to just smashing of things that we get the accurate measurement of course it doesn't work so last thing people don't wanna go through it and ultimately what we have is really limited opportunity to take measurements so we need to be really careful about how our usability and this difference between company company so 1 of the things it's really tragic about stealing the Google area if you take Google's and use it for yourself they can abuse the candidates of people still want to work for group right it's true it's condition process but it works for them because people on drop out that probably doesn't work for you this is differ between candidates as well so imagine we give homework would say it's so it's going on right so things appointed time uh some case if you don't job would say you just graduated look and see you left a job fantastic prob it is so you has a real job or got herbivores you have family you have other commitments if you've ever had medical issue this is now what you measure yeah mean imagine you need just exclude anybody you never have them but that's what's happening so we need to keep usability and so through a couple
of confounding factors and your target probably does not look like a saw reality nobody's target looks like this but your target
looks like something that is because your requirements the things that you do deconstructing even measured different for you than are for everybody else right and so you cannot just take somebody of the when you're
thinking about these interviews you can simply say OK will those maximize 1 dimension I can tell you approximately how valid different types of interviews are we have to balance that
against the fact that you need to say should I trade off of the 4 reliability is the correct answer is often yes and then in
reality bring back to the usability we get this step 3 concepts pick to each but we really pick like 1 so there is no perfect and you're not going to get something that is off to the top right here
period something that's nasty and dirty and
what that means is that the only way for you to tell the interviewer works this to you must test your
inner and what that looks like if you have an existing team this is actually nice you can give your interview to your existing the assuming you like if you don't like them just invert the results assuming like and you can give the interference but that's also not good this is this is why it's so odd stuff like measure is because if you have let's say that you managed to come up with a team that all right handed it is now create an interview that happens to be very difficult to with with your left hand your entire team will succeed has uh we're accurate about all those things but again we managed to measure something we don't tend so you need to go out and he detested reviewed and people who were not part of your existing schemes were not part of the kind of experience and demographics of your existing go
the I forgot to use these tools and all that the actor in the but but problem so we have to live a life reliability we have validity where usability was started its so um out
she is not pretty area but it's my talk and you can tell the fast so uh the the every process really does start back jobs this this is good and bad and the reason I want grimacing years because if you would get if you have uh if you actually exclude bunch of people JobPosting causes nobody would you know left-handedness to apply and nothing you do the rest of the interview will ever accept right not even in your I the the thorough watching the this is the opportunity to attract more constructs we talked about you can just steal from somebody else the job posting is an opportunity to think about things that make somebody successful for your organization the you need to think about what things you actually mention the and the rule of thumb is that if you're putting in is a real requirement you should probably actually measured during the interview if you have something you know about a measure that you probably don't need you to prioritize it because everybody is different memory is lost and so in reality if you have no was the 6 things you really need for and probably really only need like 4 5 and if something comes in you may want and then again going to need verses 1 I want somebody who's protesting lots of these data factoring want so scale services and still everything gets buildings but I don't need that and so remember that many people been socialized not to apply for jobs that don't qualify so the more you wouldn't need the more you exclude people happen not to agree with applying for jobs and qualify or but that is the reality of of we're asking for the actual text of your posting is really well and had any of you ever climb to apply for a job or discontinue quite for jobs was 1 of like variation of the new on July the right anyway so I I have neglected to apply for places just not right do you think they meant to exclude How do I manage that I but the reality is that happens and so the words that you use that would you actually ask for that of in the resources on a appointed 2 different things to resources that I like to use for this one's called but what they do is you put your posting American tells you these are things like user for pretty words that people tend not to do very well right and that can help you know is called job went so that's 1 where again if you've got you know these kind of that you rock star and across encode they can point out all of those things you you may not considered so you need to be careful about what we ask and we need to think about where we are if the only place that you poster job is currently not whereby the way you like the 50th most interesting start right your team is gonna reflects precisely 1 back the same year that's not good we have a variety of that same thing goes through network if your network is relatively homogenous and come from the same place the same things that is not the greatest sufficient can you need to think about where you are posting job you need to reach outside that covered so to find more people and ultimately this is a good thing but the initial screen it's resonance the start of the different types of music on and these true yeah I think would be a kind warrior colonists it if you wanted to know if you want only a litmus test if you can google something in 30 seconds articulate 10 seconds and R. B. it's probably a trick question so the signal of validity and in the URI well maybe not as questions but there are questions that are true and you can ask that might be that the problem is that there's not very much 6 usually last 1 thing like what's the name of the order of arguments it's very little data and 1 is a rewarding whatever of measuring instead use recency have you dealt with it later so all times with some works is that the the you know director of the CTO uh you know read something on right it and then they go unfunded you're like OK you know this thing you know as well as the memory for footprint this spring I is that valid job now a minute ago you your refine this is not a real valid criteria and could be reliable if you ask the same question I think it could be realized do people ask the centromere question now mostly get from whatever they were thinking about what so in practice this is rarely is a usable all wanted questions very easy all saying I think that you could get more valid version this by asking like 50 of these the I think that with enough trivia you might be looking get something that resembles a valid question then that's not used is not very you others part of very common country misfits but I I think this was interesting and so generated valid is it valid maybe there is an aspect to it says can you write someone it does suffer from the problem more people tend to study for physical right like if you are a big camper you absolutely a learning write this was called but just track some so is a reliable against this probably rely like and administer it I can think about the different types of things you do right the wrong score is usable that's actually use I think ultimately that's why people use it is because it's very easy to administer and if you want to make is better we would problem want changes from 1 that's really well known when that everyone knows this it would if a job length is possible you know past 30 per cent of all things could work and but which is evidence of its loss of homework who issues homework for the hiring process which I mean the greater going around times of vertebrates I think again homework is an interesting homework is super about AUC is a work sample test so in the terminology of I ISI probably the most valid way to the most predictive way to look at some his work is to have them do the work as it works on the homework works but it has significant problems the the the reliability is an issue because what you have is Canada so can spend 20 hours and so you can spend 5 hours if you grading criteria don't take this into account you end up with a really different process cost of people and you did not intend to measure again whether I have commitments at nite but you did as a way we can fix that is made you put the put some parameters hopefully you given an assignment it's related to your work because the validity hopefully you given us I use a state line up at the Ricci because it gets you closer to having an accurate baseline and hope we don't get any homework that is like these remote are all upper please uh work on this NP-complete problem of this thing that our suffer architects can even solve that like shows a working demo but sometimes we have a tendency to do just like that that's the thing I was thinking about the and then I minus but a lot of time on this 1 but so there's a lot of we have used the sort of sites where they promise 2 units for and so you go there and I think it's super super usable because it doesn't take an engineer and they and can be reliable because you tend ask like the 1 Question Writer a handful of questions about the question of validity comes I think that this is also really where these become problematic is that the questions in the question bank because they need to be auto gradable because they need to generate this big volume of them tend to have very little to do with the actual business of buildings and if that flexible probably haven't seen it yet so
here you a year ago Karin Miller give a talk at risk of about Harry From cells I it's actually talk you should go watch it in a lot of things about you needed that you have the big ones on a cover right now is really minimizing their so anything that differs between can ultimately is going to give you extra noise that not much that means that you should have a schedule when should be consistent schedule you should know your interviews are your candidature no that is what are your interview is needed to train the need understand what it is that they're doing the outscoring reverse and on a container that your candidature probably know you measure this could be sneaky about it it's probably a stupid question right so if you do those things up and tell them what to expect and put them more easily get more signal 1st they go
into the writing and has anyone
been implemented a will say red black tree at work in the last year no no and again you don't do algorithms yeah they look like cover they look like code we use but in reality that's not how we build software nobody ever implements red-black trees from memory at work right so is a promise to validate the really not because the correspondence is very low are they reliable now not really they they have so the same recency bias of the studies on so you just graduated CS is probably gonna remember this but somebody's got much user experience is a great an inverse selection you select for inexperienced because those people remember our good work usability poetry at a better version of this would be if you were to take a number algorithm like don't pick something was so the famous is named dextrose anything the you idea that I'm thinking of really don't really that you make up and how the implement that from she paper say here's the algorithm here is a lot of students this is the pits recently by this is an actual test what we do for soccer friendly requirements into working code don't do of the even worse version like we're
coming there is expected and what was going and I will consist of a you at your job you're required to white word co not if you have the option of people tend to be if you're required a whiteboard code please by all means go if not what you're measuring is a skill that people don't use unless they are practicing it in that case just give a fucking laptop yeah it's really not that at the the
and then the life blood of code and they would do this but donors ahead of its time legal if you're not paying attention and you can have a mortal production goes in most states check your work that's but so the cooling mothers validity was this is like a hundred cent Valid right because this is the work like you could not get more balanced that's cool the problem becomes reliability if you work in a real code you generally can repeat the same problem or if you do you have serious problems and should be higher than few so usually what that means that we have a straight up by the rest of the time and time in my backlog finding things similar age in which case it's not usable or I just can't walk something now in which case it's not like a parallel forms and but also it's illegal so probably should so now the exhaust the rhythms
of Gomes there's some problem solving and they
don't know what the question is is and she hired on this 1 it's are buried up to their necks got hands-on like you have some of these where what color it had to do that and we don't killed yourself I know this is done some of validity no so no reliability now so usability that rate is good possible and which is measured not the not that you can't it's a probably looking this up again here who can prove you're looking for a job just still look up like the 6 or 7 and he's if everybody uses learned called you find pretend that you're having a hard time of it come up with the director and
look better case studies you hypothetical how would you do this can mean that you can use your existing work this can be relied on to give the same case study based on historical precedent immunomodulator little bit and if you have a senior developer and you are giving them say some architectural problem that you can solve that issue here Dude develop them effectively any architectural problem and that's not something that's in the skill sets probably know which mean but even better is my favorite cognitive behavioral synonymous in this so this guy takes the same form every single time tell me about a time when x the reason this works is that despite what they have to say the financial sector of past performance absolutely predicts future absolutely and so this is a great interview in so far as you can test something you expect is now that there were it's reliable you can you can test the same thing over and over again and then the usability there's a little bit of a challenge it turns out that you get good answers of people you to train them on this kind of viewing but that's OK you can you can and
then culture if your culture if it looks like the ones in Florida CTO you directed going intuition for about 40 minutes and and deciding yesterday right so you had this before and you're measuring for people like me there's no criteria there's nothing there's to get the unweighted got real works is that it takes into account every 1 of our preconceived notions so it is again almost you know of a lesson you should be just don't do things we and do them anyway so regardless of the and least think about things in your culture then due cause you can be successful we should 1st all the time we show the highest quality all right we teach everybody or were independent capable of this kind of things that actually could measure success of these measures the the better off to still use
so you set personal another gone no more problems are so energy refinement and the working all the time is of course and we get you a circle and then everybody does this value right purposes is a kid like properties people each other region socially this is what happens at or you end up talking the person I love this person like AI and like handles the words you yeah it 6 because my glottal unity so the way that we fix this we write it down 1st of all have your Añoveros right on specific object feedback not she was cool but she missed the test coverage issue in this question should be solved the problem with 10 minutes to spare and then share all that the back once that nobody can cheat make sense cool so we can get
if you want to hire well book so if you want a higher well you need to pick a set of constructs and you need to design and test interviews the valid reliable use cool there's 1 more thing I want talk
about before we actually the reason is that point which I submitted his talk as having issued a couple clients where and they had teams will say were more than a little bit homogenous and so I talked about the interview process and what they what they always said the feedback in all the time is really wanna hire people really have vastly we don't over the bar in and I hate this right but I can tell you precisely why like can give you the reason you know it's wrong but can give you and I understand the reasoning part of this talk the reality is that for a lot of us with the belief not bad people but they believe they have this bar over your on and in that sense of course you don't but the reality is a bar doesn't look like it's a bars like weird and tilted and fucked up because of all these extra things that you're measuring that have nothing to do with chance the solid say we should probably actually reasonable most people the views are not nearly as tough as but to do that 1st we make sure that the bars string court date did you and I and my of