We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Refactoring the Technical Interview

00:00

Formal Metadata

Title
Refactoring the Technical Interview
Title of Series
Number of Parts
66
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Is your technical interview optimized? Could it use a refactor? Do you ask candidates questions that could actually influence your production codebase? Or, do you ask them about problems that are solved and standardized? Or worse, contrived and trivial? Have you ever asked any of these questions: Find min or max in a list Implement a sorting algorithm FizzBuzz (or any contrived algorithm) “Now can you do it recursively?” Leverage your production code and commit history to make your next interview more effective in identifying the right hire. All while having more fun.
Code refactoringVideoconferencingHill differential equationDreizehnImage resolutionQuicksortNumberAlgorithmProcess (computing)Multiplication signDecision theoryReal numberOnline helpCodeWritingElectronic mailing listFormal languageMereologyObservational studyPerformance appraisalAnalogyFitness functionBlogEntire functionGoodness of fitDrill commandsBitMassBinary treeContext awarenessStandard deviationSoftwareComputer hardwareDegree (graph theory)AdditionRight angleSoftware testingLevel (video gaming)JSONXMLUML
Software testingObject (grammar)Cone penetration testInfinityReading (process)WritingCodeRight angleGoodness of fitStandard deviationRule of inferencePhysical lawWritingCode refactoringFundamental theorem of algebraTerm (mathematics)CodeDifferent (Kate Ryan album)Multiplication signPoint (geometry)ImplementationBitChemical equationQuicksortStrategy gameProduct (business)Reading (process)AuthorizationComplex (psychology)Tablet computerMereologyProcess (computing)Procedural programmingAlgorithmElectronic mailing listVector potentialSoftware testingHypothesisUniversal product codeSoftwareReal numberField (computer science)Expected valueTelecommunicationObject (grammar)Physical systemSoftware developerComputer programmingFiber bundle
Compact spaceOrder (biology)Local GroupCubeThree-valued logicCache (computing)1 (number)QuicksortPosition operatorSemiconductor memoryDifferent (Kate Ryan album)Sound effectReading (process)CodeOrder (biology)String (computer science)2 (number)Group actionComputer fileBitRow (database)Translation (relic)Three-valued logicNeuroinformatikAttribute grammarContext awarenessFormal languageDisk read-and-write headSource codeOperator (mathematics)Point (geometry)Control flowUniversal product codeSoftware bugMereologyDigitizingRun time (program lifecycle phase)Code refactoringStapeldateiMultiplication signAdditionGoodness of fitPointer (computer programming)Process (computing)Area
Local GroupScale (map)Configuration spaceGroup actionAuthorizationMessage passingBitStandard deviationSingle-precision floating-point formatSequelPartial derivativeDatabaseExpected valueEquivalence relationDifferent (Kate Ryan album)Variable (mathematics)MereologyIntegrated development environmentTheory of relativitySoftware repositoryCodeScaling (geometry)NP-hardLine (geometry)Multiplication signInstance (computer science)Mixed realityWritingSoftware testingView (database)Control flowConfiguration spaceNumberContext awarenessError messageDot productSoftware bugComputer clusterInformation securityRevision controlLevel (video gaming)ApproximationMobile appCommitment schemePointer (computer programming)XML
Hill differential equationDigitale VideotechnikSoftware bugComplex (psychology)Independence (probability theory)Decision theoryAxiom of choiceMessage passingDifferenz <Mathematik>2 (number)
Kolmogorov complexityTraffic reportingComplex (psychology)MathematicsCodeComputer fileEntire functionMultiplication signComputer animation
CodeOpen setObject (grammar)OvalGoodness of fitSource codePatch (Unix)QuicksortOpen sourceCausality
Musical ensembleXMLComputer animation
Transcript: English(auto-generated)
In the interest of letting some more people shuffle in,
this is my very first RubyConf, sort of went all in. Thank you, thank you. So if you'll indulge me for a brief aside, I have some time at the end, so we'll just eat a minute into that and give some people some time to shuffle in. Yesterday, I went to a talk by Jack Danger.
Did anyone else go to that one? Yeah, so I went and he started talking about the pitfalls of the technical interview and I was like, oh no, my heart sank a little bit. I was like, I'm gonna have to trash everything that I worked months for and just replace it with this.
Ah, he's got that comic book superhero name, right? Jack Danger? So thankfully, he went much higher level, went way beyond what we're gonna do here. His is more strategic, high level. This is gonna be a little more in the weeds, if you will. If you didn't catch that, it looked like about half of you did, I would recommend if you care about scaling teams,
check that out when Confreaks makes it available. And I'm not making this up, this part is true. I met someone right after that whose name is Justice. He's actually here in the crowd. He looks like a superhero and I thought to myself, I came all the way here from San Diego
and ended up at Comic-Con anyway between Jack Danger and Justice. So anyway, sorry, it's enough bad jokes to start us off. Let's get into it. So in addition to meeting Justice and hearing Jack, some people were kind enough to share with me some anecdotes about challenges they've had.
Some solved, some yet to be solved with a technical interview. So I think this is something that everybody faces to one degree or another. And if you're facing these challenges, like if it's the first time you've done it, someone in the audience, Jeremy, is doing it for the very first time next week, I think.
See if you can apply some of these concepts and we've got a good size here for some audience participation, totally voluntary, but I encourage you to ask questions both when they're prompted and at the end when there's time. So why even bother with a refactor? The fact that you're here means
I think you already see some value in this, but sometimes you're not making all the decisions. So I find it's helpful if you can enumerate these costs to not paying attention to your hiring processes and your interview process. So if you have a CFO, a CEO, a numbers person
that's saying like you're not allowed to spend time working on this, you can point to this list. If you're not making the best hiring decision, it's costing everyone time and some amount of money, resources to on-ramp that person, learn the workflows, someone mentioned that being an important part of it, opportunity costs of someone else
that you could be training that is a better fit, reputation, it's not a good situation for anyone if you make a hire that's not a right fit. It sucks for the new hire, it sucks for you, it can hurt the morale of your team, you can add technical debt, these are real, real things. Tricky to quantify but not impossible.
So if you can answer the question, how much is it costing you, it will be easier for you to justify spending time and resources to make it better, right? Is there any questions on this piece? Good, okay. So it's important, but is anything wrong? Again, I think everybody kind of has an idea
that there's some opportunity for improvement here. I'm gonna walk through a couple things, right? So this is part of the Joel test. I actually like this 12-part test. Candidates write new code during their interview. I think it is an important part of the technical evaluation of someone.
The issue is the kinds of questions that we ask. Now, on this list, has anyone been involved in an interview that asks at least one of these questions recently? Yeah, most of the room. Any other examples of something that someone's been faced with recently?
A question, Jack mentioned balancing a binary tree. I think I would probably struggle with that one. I've never used it in my entire career of coding, but in an interview, you might need it. So does anybody understand, does anyone think
that these are good questions to ask, that you can learn something from this? Well, yes, yeah, yeah, you can definitely make some evaluations based on those things.
We're gonna revisit these in a minute. First, we're gonna walk through an analogy. I really like analogies and tangible things. Bear with me for a couple slides. This will relate. So ask any question in a carpentry interview. Here's some ideas that are analogous. How do you build a drill? Imagine asking a carpenter that.
Maybe a circular saw. Okay, what about you have a structure. Every third plank is birch, fifth is maple. Right, what's a carpenter likely to say? What, why?
This is what you're gonna get if I build you a drill, and I don't wanna use that on the job. Do you want me to use that on the job? So, why do these seem silly in the context of a carpenter? So the issues with these are solved. Long time ago, people figured out
how to mass manufacture drills, quality control, all of that stuff, standardized. Anywhere with a hardware store, you can find a drill, easy to source. Or the last one is trivial, right? Who cares if you can figure out the number of pieces of maple unless you're doing a very, very specific thing, right?
So related to software development, same kinds of questions. Solved, quick sort, 1959, right? Before the electric drill, before the electric drill, cordless, cordless electric drill, right? Merge sort, earlier, we're talking real history books here, right?
1945, radix sort, 1887, right? Here, if you wanna have a little fun, if your next interview, they want you to implement a sorting algorithm, and you get to choose what it is, do radix sort. It's kind of a fun one, and just tell them
that you really care about backwards compatibility. In addition, these are standardized, at least the sorting, min-max, those kinds of things. If you can find me a language that doesn't have this, I would say, why are you using that language?
Studied, right, hundreds of publications, blog posts, books, online tutorials, you name it, that prepare you not for a job, but for an interview, right, or trivial, fizzbuzz and bubble sort. Interesting thing about bubble sort, there's an academic paper that is suggesting
that should be entirely removed from CS curriculum, because people remember it on the same footing as other sorts, and they consider it dangerous, because the time complexity is bad. So if you're asking someone to implement bubble sort, something is wrong.
Okay, so why do we ask these questions? I touched on this a little bit, so let's revisit it again, because I'm not totally sure. So some people had their hands up before, you said that you might be able to figure out where someone lies in their technical abilities. Any other ideas, these kind of standard questions, what they're helpful for?
Yeah, so it's a good initial gate, potentially. Yeah, yeah, I appreciate that. So here are my thoughts, my hypothesis on this.
Those kinds of questions are optimal for evaluating algorithm writers and procedure implement, procedure implementers. Once upon a time, I think everyone in this field had to be highly skilled at this. It was just part of the job, some things weren't figured out, some things weren't standardized.
I think that was a long time ago. Not optimal for these kind of developers, right? If you can implement bubble sort, that doesn't tell me how you might design a system, or create objects in an object-oriented language, right? Said another way, the skills on the left may be valuable,
but the skills on the right are more valuable, so you should optimize for those over the algorithm writing. Cool, so if you already have some questions that you use for your interview, I have a two-question litmus test to see if it could use a refactor.
One, would you be happy or very upset and confused if someone did this in the job? Oh, I spent last week, I implemented FizzBuzz instead of whatever feature you wanted me to do. You'd probably be pretty upset about that, right? So don't ask them to do it in the interview. Next one, which book would better prepare someone
for your interview? This one, or this one? Of course, these aren't the exact two, whatever the defacto standard is for interview prep at the time, and we're Rubyists, so Pooter is a pretty good one, but if you're in a different field, I'm sure there's a defacto standard
for learning coding styles and best practices. Cool, so if you answered yes to either of those questions, consider a slightly different approach. So algorithm writing is still important, right? That's not totally absent in our field. Maybe you're not sure what the right balance is.
Maybe you need somebody that has some of both of the skills in those earlier bullet points. So let's take a step back. More generally, what is the point of an interview? What's the basic thing that you're trying to figure out? Any suggestions?
Yeah, that's definitely part of it. I would say, more generally, to see if someone can do the job. Being a part of the team and being an effective member is definitely a piece of that. So if we want to see if someone can do the job, we should line up the job expectations
with the interview expectations. This is my list. I think it's broadly applicable, but you might have your own. It's very short. I expect someone to be able to read code. This is mostly about the technical aspect of the job. Communication, soft skills, things like that are also important.
The scope of what I'm doing here is on the technical interview. So read code, understand it, make it better. For me, that's what I care about in a developer. You might have a different list. But what this means is, when I set up my interview questions,
I have certain expectations there. I want someone to read code. I want them to explain it. I have not figured out mind reading yet, so if you understand it, I also need to explain it. That introduces a communication aspect as well. And then of course, make it better. Does anybody notice any words, verbs,
or terms that are absent here? Write. Fast is not here either. Write was the answer I was looking for. We'll come back to that. So what about writing new code? The Joel test said, people have to write code during their interview, right?
Joel also said this, right? And he didn't just say it. He said it is a fundamental law of programming, right? Sandy said this,
and Robert C. Martin quantified it. Well over 10 to one, okay? Now implicit in making code better, you're gonna at least be editing. You might be authoring new things. But reading is, if you believe those experts,
90% of the technical work and more difficult, writing 10% of the technical work and easier. I contend that if you wanna follow the 80-20 rule or the 90-10 rule, you care about reading and explaining more than you care about authoring something new, especially if that authoring is something trivial, right?
Even if someone does write new code on the job, you're often referencing examples, using generators, different things like that, right? How many times do you start with a blank slate and just type? Maybe never.
Cool, now we have a new problem. What code should they read? Does anybody have any ideas here? You got it. Yours, your code, your code.
I mean this in the plural, possessive, somewhat abstract way. It doesn't mean your actual code, it might be. It could be something that somebody wrote five years ago before you were at your company. The closest to what you think the person will be working on. Now, I'm gonna give you a couple strategies later
about how to curate this. And if you're in a situation where your code is confidential and there would be too many NDAs and all that, you do have another option. We'll talk about that later. But I would bias towards production code, right? That's sort of the main point here. Okay, so let's try it.
I've got a couple of examples here. The first one is the very first bit of code that I ever used experimenting with this. And it is somewhat simple, it's not very much code. It's not a good example of writing software
by any stretch. It was a real production code at a former company. The reason that I still use it is that it is a good example of how many opportunities there are for insight with this kind of approach. And then the other one is more recent, we'll get to that in a minute.
Cool, step one, read code. Can everybody see this? Good, okay. So I'll give you just maybe 30 seconds to digest this before we jump in. If it's not obvious, this is the active record that Rails uses,
so we've got a couple of those. Obviously in the production code base, these are in their own files. This has been distilled down and packaged together for an interview situation so that you don't have to go digging around files. You can do it sort of in a closer to whiteboard scenario. Cool, anybody want a little more time
or we're okay with the reading part? Okay, so first question, what is this code even doing? And let me be more specific here. It's the sort by bit that I care most about here.
What is happening with that? The other stuff is more for context. Does anybody have any guesses here? It is sorting them? Yeah, sorting by packages based on the position attribute
and then by, I'm sorry, the group and then the package. Is there anything else important going on here about the sort order that we need to consider? It might come to light as, oh, yes, mm-hmm, yeah.
So you see the turn area here is basically saying if the package group is falsy, usually nil, then put a zero there and that will sort to the top if you do not have a package group
for a particular package. It's a good point, with string concatenation, you've got high and low order
or significant and insignificant digits where you might have, yeah, a 10 and a 10 versus 101 and a zero. Yeah, there's some bugs in this code for sure. That's an excellent point. So if you have a candidate that's struggling through,
you can ask some directed questions, right? Order and sort by. Is there any difference between these two? Yeah, exactly. So one happens in the database, one happens in Ruby memory. There are trade-offs there.
You might ask, should we swap one out? Is there anything we can do to make them both order? What are the side effects of that, right? Some questions like that. Why the ternary? We covered this a little bit already. You need the ternary for nil protection
and to try to move the packages without a group to the top of the list, right? So that must be one of the features for when this was implemented. Why the 2s and why the 2i? Any thoughts on that?
We're doing 10s and ones. Yes, thank you, you nailed it. We need concatenation, not addition. So we have to change it to a string at the top?
Right, depends on if it's a stable sort, right? It's an excellent question. If the packages are already in order, will they remain in order during the sort by? It's a good question. I actually don't know the answer to that
off the top of my head. But I would wanna hear those kind of trade-offs. And the 2i, I'm not even sure that's necessary. It might be, but I don't think so. Ruby can sort strings pretty easily. Cool. So let's review. While someone is explaining code,
you may learn how they translate code into human understanding. To me, this is kind of the crux of what we do. Even if you're a rockstar developer, if you don't understand the human problems you're solving it doesn't matter how good your code is, right? And vice versa. If you're very good at understanding the human problem
but you can't code, you won't be able to tell the computer what needs to happen, right? So I think you get some sense of that when they have to translate from computer code into human understanding. So order and sort by, you get some sense of the depth of knowledge, right? The tools at our disposal here are enumerable,
active record. What are the implications of using one versus the other? Do you consider trade-offs? Maybe if you do it all in an order by clause, it's faster, but you might break something on accident, right? Why the ternary? Why the two S?
And we kind of hit this on the head. How does the language treat operations, both sorting operations and the addition operator? Cool. So now imagine that you take a minute to step away and you give them an opportunity to make it better. You can review what you discuss
and give them time where they don't feel pressured to perform in front of you. If, or if you want to be there to answer questions, that part is kind of up to you. But pretend you gave somebody 20 minutes to make it better and you came back. We're gonna run through a few of these. Has this improved in anyone's opinion?
Show of hands if you think this is any better. Few people? A few people? Is it more readable? I think so, but maybe not. I don't think it does anything to the runtime. I think it's more or less equivalent. So it's a cautionary refactor
with an attempt at improved readability. Okay, how about this? Any improvement here? Maybe? Is this a naive solution? Are we missing something?
Yeah. Sure, use like a find each, like a batch processing. Yeah, that's a great point.
Now we've broken things. This eliminates packages that don't have a package group. Now that's not necessarily a deal breaker, but it's something that you wanna discuss. And when we review, we'll talk a little bit more about that. Is this an improvement?
Does anybody love SQL? I love SQL. I think this is great. It's fantastic. So bringing it back to Jack's talk, one thing, if you're looking for somebody where your overlap is not so duplicative, and maybe you're not a database person and no one on your team is, this might be the most interesting solution for you
because now you've got a piece of the puzzle your team is missing. Same for the other, some of the other solutions. Now, is anybody in here a SQL person? A few? Are there any issues with this solution? And if you are a SQL person, the answer is easy.
It doesn't take a lot of thought. It's more of a trivia question. Yes? I'm sorry? Yeah, if null is specific,
it is not part of the SQL standard. So is null is the equivalent for MS SQL, NVL for Oracle, coalesce is actually the SQL standard. So this will not work for, you're no longer database agnostic. And if you have a database person, they should point that out.
Is this an improvement? Maybe, this is very active recordy. I believe it is correct. It requires knowledge that you can merge basically other relations to allow more complex joins and ordering.
Have we introduced anything here? Again, this one should be a little bit more obvious. It doesn't take a lot of thought, so if you're thinking too hard, don't think so hard. Anybody? We lost an instance variable name.
We don't have app packages anymore, right? Maybe a view breaks now. We gave more descriptive names, but we forgot about backwards compatibility. Cool, so a recap. Basically, you wanna know if they ask good questions. Any mention of writing tests?
Do tests exist? Should I write tests? What would they cover? We talked about how we introduced some bugs where we eliminated packages that we shouldn't have. Now, I'm not expecting them to write many tests or RSpec on the spot. That's a little too time consuming. As long as the mindset is there, I'm happy with it. If you're more about TDD, maybe you don't even care about their solution.
You just wanna see the tests they write around the code. About the database. What if there's a foreign key constraint? All that's for not, right? Again, if you don't have a database person, they're asking those kind of questions, might be a good complement to your team. Rails and Ruby versions.
I don't know about all the compatibility. I think left outer joins was not available until maybe 4.2. I'm not sure. But you might need to make sure some of those things, depending on the level that you are evaluating. How about commit messages or reasons behind the initial? This is my, this is the one that I care about the most.
All the other stuff is some technical detail. This is weird code. This is a weird bit of code. And maybe somebody was under the gun to get it done. But you don't write code like that on accident. You're hacking at it until it works. Is there a good commit message that says, yeah, I know this is ugly, but this thing happened
and this is the only way to solve it. We'll revisit it later. And maybe there's not a good test, but there is a message that communicates what you need. That, to me, I care more about the other things. Cool. Oh, I forgot to introduce this. So does this approach scale?
What if you're interviewing a junior versus a senior or staff? Does this single question scale to those different roles? I would say, at some level, it does. If you're asking a junior, you might have more prompts. You might have lower expectations about
the trade-offs that they mention and implications of different things. More senior, you might expect unprompted answers. You might expect them to talk about database compatibility, all of those things. So your expectation of answer and how much you guide can scale this up or down. Maybe not infinitely, but to degrees, for sure.
So the next one that we're gonna talk about is more recent, and we have someone that's going into a technical interview next week. This example is hot off the press. There's an open PR in our repo for this right now. I pulled this last night.
So you don't need a lot of time to prep these kinds of questions, right? My coworker, David, this looks very familiar to him. He worked on this pull request. So I included these line numbers on purpose. This looks familiar to anyone who's used Rails recently, at least some of it. So here, there's not a ton of context
for the definition of requires MFA. So I'm not worried about details about how that's implemented. What are some concerns with this code, though? Just right off the bat, if you're thinking about OO principles and how this is gonna break when you run it, what are some concerns? Anybody have any ideas?
Yes. Yeah, config, look how many dots there are. Method missing, nil errors. You're asking for trouble if you have, whatever that is, five dots. There's a few code smells here. We duplicate that call. It's Rails config, blah, blah, blah, blah, blah. We do it again on the very next line, right?
There's some issues like that. Any thoughts on how we make this better? Again, without full context, how can you apply some principles to make this better? I'm sorry? Get rid of some of the dots. Yeah? So what are some tools that we can use to do that,
to get rid of the dots? Yeah, exactly. New object. Right, great idea. Send in the user, implement a new object, have it take care of the configuration weirdness
and madness. Right? This is an approximation of what we came up with. There's some stuff here that I was a little bit worried about for security reasons, environment variable names and whatnot, so I redacted the private methods, but you get the idea. The other thing that this leads to, and so imagine that we didn't have this solution in place,
but I knew we're basically changing this. The user requires MFA as a new feature, an enhancement to this, and that's why we revisited this code partially. So imagine we didn't have an answer to this, and we brought it into an interview. You get a couple benefits there.
One, you get some free labor, right? The other is you don't have expectations about what the answer is, right? I've heard some concerns that I'm afraid I'm guiding or leading or projecting the answer that I want. What if you don't know the answer? Right? It's an interesting spot to put yourself in,
maybe a little nerve-wracking, but I think if you mix in some where you know details of the answer, like the other one I've had for years, I've seen different people give answers, I've had time to think about it, so it's helpful to have that bigger scope, but also mix in something that you don't know the destination, right? Let them drive it.
Cool. So how do you curate these? Easiest way for me, you're doing a bug fix, save a before and after snippet, or if you'd rather use your git diff between two hashes, that's if that's your tool of choice. If you need to go back in time,
of course search your commit history. If you do things like put tags or references to ticket numbers, look through your commit messages for those, look at the before and after, pull a ticket from your current sprint. If none of that works, run a churn versus complexity report. There's a great tool called Ruby Critic for this.
Independent of the interview process, I highly recommend Ruby Critic, and we'll look at that in just a second. So for your commit history, interesting bugs you wanna look for, also a new feature that required some design trade-off decisions,
and you're not sure if you made the right one. It's like getting a second opinion, right? That can be a fun way to approach it. Cool. This is what a churn versus complexity report is. That user file that we just looked at, that's that guy way out there. By far the highest churn of anything in our code base.
And our final solution actually, if you're curious, removes that method entirely, right? So that helps us move that this way. Complexity goes down, and over time churn will go down, because changes to the multi-factor thing no longer touch this file.
Cool. What if my code is confidential, right? So this is a big bar, right? Because then you have to get NDAs involved and maybe lawyers, and that's no fun. Any ideas before I jump into here what you can do if you're not allowed to use yours?
Yes. Sure, anonymize your own code. Sure, sure, tacos are great. Everybody likes working with tacos. Any other ideas? Open source, right?
If you don't use open source, I am shocked, right? So if you're in Ruby land and you're using zero gems, I'm not, I don't know, I think you have other problems. You need to leave and solve those before you worry about the technical interview. So if you can't use your own code, at least you can get them closer
to the code that they will be using on the job, right? If you heavily leverage gems, maybe you monkey patch some of them, maybe you extend them, dig into that source code, open it up, have them look at it. I don't have any examples of this yet, because I haven't had to do it. But it's certainly an option, right?
And it's out there. Half a million on GitHub alone, and this is stale data. It's probably more than that now, right? And let's say that you're working on something as obscure as whatever Objective J is, you still have hundreds of examples of open source, right? Again, the closer to your,
what they're gonna be working on, the better. So if you have, oh, someone asked a great question during Jack's talk. They said, what if you need a skill, but no one is qualified in your team to evaluate that skill? That's sort of a chicken and egg problem, right?
Pull one of these things out, give that to them, and say, I'm a Rubyist. We need someone that knows C sharp, Java, whatever. Explain this to me in a way that I can understand, right?
So there's a saying that you don't truly understand something until you can explain it to someone else. So unless you have a really good con person on your hand, if they can explain it to you, then they probably understand it pretty well. And I think that's the best you're gonna get in the absence of getting the expertise yourself ahead of time.
So, and I actually have an example of that if you guys are curious to look at it. But with that, thank you. And I encourage any questions. We've got about three minutes left. So if anybody has anything that they wanna ask now, or I'll be available afterwards. Thank you very much. Thank you very much. Thank you.