Yeah, my name is Sheena it's nice to meet you this is a picture of me a remarkable likeness, so My job title is but weird. I'm the product and tech enablement lead Adam was Eve, and that's a mouthful, but Basically what a moosey is is we we train people in different digital skills. I'm involved in software development things

often in Python things sometimes JavaScript sometimes other things And Basically what I do it was years. I build remote education systems, and that's composed of a bunch of different Parts, so there's development work. There's running a dev team there's

Creating and vetting syllabuses. I do you X and then there's pedagogy and regard to you Goggi, which is all hard to say, but it basically means the way that we teach That stuff that sits with me Python is my love language And these are my emojis so a

moosey Our purpose is to reduce social inequality through education, and we're based in South Africa That's that's our vibe, but in order to get this right we need to teach people about skills that they'll need for the long game they need to be able to Like we don't want to just set people up with little micro gigs and a way to get through the month

We want to set them up with a way to get through the next five ten fifteen years and to do that We we need to play the long game And so we need to think a lot about what the future will look like and that's really really hard like in tech That's really hard generally, but it's even harder now because of this whole AI thing. It's

There's a lot of there is a lot of hype going on And there's a lot of excitement going on and things are changing very very quickly and so looking to the future looking Like we need to look to the future to figure out what to teach people now So this talk is going to be a lot about cutting away some hype of AI

But before I do that, I want to make it clear that I'm actually a huge fan like a big fan I use it all the time. So I use LL M's while I'm writing code. I Think they're very useful in education generally if used correctly But I also believe that they're about the biggest foot gun around for people who are busy learning to code for the first time and

that's because there's a lot of assumptions that people are making and a lot of bad conclusions that people are coming to so Before I can talk about How people are shooting themselves in the foot while learning software development. Let's talk a little bit about what software development is

so a Software developers job is to write lines of code, right? No, that's not how we measure our productivity And so typing out lines of code quickly is not the vibe what we are trying to do is solve problems and code Is an artifact it's it's the output of our problem-solving

method But we're solving problems But even beyond that like sometimes you don't even know what problems to solve So a software developers job is often just to like figure out where to even go What problem the what the problem even is so? It's not like this. It's not like you have this idea and you're like great and then you translate the idea into code

It's not like I have a problem and I have a solution and I translate the solution into code It's not a specification that you translate into code. It's not a translation service We do start off with some kind of idea some kind of problem and it's a bad idea

It's a bad idea because it's full of assumptions It's got assumptions about the world. So let's say that you want to build the next uber of dog walking apps Prague is great. I've been seeing so many people walking dogs and you like target market fantastic This idea would be full of assumptions about how people care about their dogs and feel about strangers and what?

their economic Situation is so it's full of assumptions about the world It's full of assumptions about what is possible with current technology so This is especially true If you're working with like a non technical stakeholder who like often they just aren't super sure what you can do

If you are a technical stakeholder You're also pretty like not sure what you can do a lot of the time like does this tool do what? It advertises. It's hard to know without even trying and and With LLM's and AI is developing the way that they are It's very very hard to know what what these tools are even capable of until you get your hands dirty

There's also a lot of assumptions about shared understanding So if I'm talking about the uber of dog walking and so are you then we might be talking about Slightly different things we have slightly different context. We are going to be using the same words and understanding them differently. So

We do our best we we come up with these ideas and we try to communicate them with each other and We communicate them imprecisely like We draw pictures We we have flow charts and research artifacts and conversations and wireframes and documentations and text messages And it's it's a lot and it's insufficient

The most precise way to describe what a program should do is by writing the code So It's not like we start off with a perfectly specified idea and then we translate it into perfect code If this was the case, then sure AI could do our jobs easy peasy, but it's not the case

We have follow-up questions. We push back on Requirements we consider trade-offs we make suggestions we get inspired. We have empathy even developers need a bit of empathy we create an architecture we Like try our best to choose the right tools because that's freaking difficult actually and

We spend a lot of time trying to just validate and revalidate our solution. So it's it's a mess. It's like It's a collaborative process. It's an iterative process and it's exploratory. It's not a straight line. Oh Golly there's a video here that I'm linking to it's got a great thumbnail. I promise and it's by Dave Farley and

he He He explains this thing that I just explained. He does a better job of it I highly highly recommend that you watch this fantastic video on your own later. I Wasn't gonna play it now watching YouTube together Not the most fun. So we've spoken a lot about what software development is

Let's talk a little bit about code because code is the output of this process So code has changed a lot over the years it started off looking a lot like this or early on in its evolution It looked like a lot like this. So what we're looking at here is a punch card and

Who hasn't heard of punch cards who has heard of punch cards? Yeah, basically everyone so punch cards They did their job. They were the only tool available at the time, but they kind of sucked It's a physical thing that you need to make holes in in the right place And if you screw it up you have to like throw the whole card away and get a new one and make the holes in The right place eventually you have a program which is the stack of cards that you carry to a room sized

Computer and you feed in in the right order and if you get it wrong you have to start again And if you make a mistake and need to debug this thing, there aren't tests. There isn't a debugger You just have to lay these things out and think really hard and it's it's tough

This type of programming doesn't feel very human it doesn't feel very humane either it's very close to the machine It's very close to the mechanics of how the machine works But at the time that's what we had time passed abstraction happened so abstraction Can be here's a good example of abstraction a car

So if you're driving a car, then you are sitting in the driver's seat. You've got a wheel. You've got some pedals You've got a way to turn the thing on you don't need to think about what's under the hood When you start a car when you turn the the key for the ignition or press the button You don't need to care about the starter motor

You don't need to care about the carburetor or the alternator or the fuel injection system or if it's an electric car You just turn the thing on you've got a little tiny mechanism and a whole lot of complexity happens in the background and that's abstraction so Programming languages became more abstract

we got to a point where we had things like C and C++ and Those are great because they introduced concepts like functions and loops and variables and that made life way way easier and that made programming much more human and much more humane Eventually, we got to things like Python which which are fantastic like Python

Abstract things even further in C and C++. You still need to care about memory management But now you you don't so much anymore you are able to just focus on the algorithm on the problem that you're trying to solve and Not worry about the details and that's fantastic As time continues we move further and further away from the machine and the languages that we use become more human

They fit in our hands like like tools that were designed for our hands and human languages like the one I'm speaking now English is Like they they're even more human Is it possible that our programming languages will become human languages will all these things be interchangeable now?

It's reasonable to assume that things will keep evolving and they'll become more human but How might the languages change? So human languages are inaccurate. They're imprecise and they're confusing We talk across purposes all the time, I'm gonna give you a bunch of examples as how about how things go wrong. So

The presence or absence of a comma can change the meaning of a sentence. So the koala eats shoots and leaves Is it a hungry koala? Is it a hungry and violent koala? It's hard to know

She ran away from the fire in her pajamas, why were her pajamas on fire or was it at night? Now these might seem like silly silly examples, but there's there's more so a lot of words just get commonly Mistaken they sound the same they look the same. So people swap them around there and there to into its and its

Things are misused a lot borrow versus lend. A lot of people have asked me like Can can you borrow me your pen things like that? It's like no, I can't it's physically impossible I'm sorry, but I can lend it to you Literally now means figuratively which is the opposite it's also important to stay hip with the times like it's very

Human languages are not only for describing Like situations in life and algorithms and instructions. It's also about group membership. And so they change all the time We have words like goat that no longer means these things It's something else now and like all of this for me just seems pretty bananas, which is a type of fruit

So These might seem like silly examples But we have a certain profession of people that exist they get paid quite a lot of money and their job is to be Very incredibly precise in their speech and to use language

Perfectly. Well, these are called lawyers They mess up a lot There's a semi-colon in the US Constitution that is quite famous Because it confuses some very clever lawyers and changes the meaning of the US Constitution You can learn about it. If you want to there's a

revisionist history art Podcast about it. So as I was quite a big deal and A lot of the time you might think like English is good enough and it often is like we can explain certain things in English I'm not Like I don't want to bash human languages. I use them myself sometimes even in this presentation

But we have to use the right tool for the job so in English like a Human language is a little bit like a hammer. It's a bit like a blunt instrument. It solves certain problems Where would we be as a species without hammers? Like we wouldn't have gotten that far hammers are great But if you really need a laser beam, you need a laser beam. So

Human languages are a lot like hammers Programming languages are way way more precise. They cannot be mistaken if you write something down in Python It means only one thing So it's more like a laser beam, but C is still a thing. So sometimes Python is not accurate enough

It's not precise enough. And so you need to get closer to the machine assembler is still a thing So I spent some time working in power stations for a while and basically all the code we wrote was in assembler language It was loads of fun, but I wouldn't want to build a web app in it But it's still it's got its use cases even now

So code is a precise description of what machines should do and English and other human languages are not precise enough for many problems if you look at code up close It just tells you what order things should happen in under what circumstances they should happen We're not just comes from where it goes that's parameters its variables and how do things change over time when things repeat

That's loops. If you look at it up close its logic and algorithms and it's just written in a way That's come that's clear for machines and it's also Importantly written in a way that's clear to people good code makes its intentions clear to people

So what's going to get abstracted away? There will be more abstractions I am quite quite sure but we're not going to get rid of if statements loops or flow control I'm going to give you some very human examples of these things if I'm out of milk go to the shop and buy some milk and While there are dirty windows clean the next window

We're not going to get rid of functions. A function is just a name for a section of but for like an algorithm So go to the shops and buy milk that could be a function with an input milk. That's a thing you want to buy And it's got a series of steps that it contains like head out the door take a left take a right

Not variables. We're gonna keep those like how much does the milk cost? Not even class as an object. So for people who are new to programming they might feel like classes and objects are advanced things But really this is an object. It's a class cell phone. You've got one, too Yeah, so it's it's a very normal human thing

Even exceptions and errors if I go to the milk if I go to the shop to buy some milk and there's no milk Error go to a different shop. So exception handling and error handling is very very normal So to say that these things will get abstracted away. That doesn't make sense We're still going to want to implement algorithms and tell machines precisely what we need done

So what's going to get abstracted away? Not our ability to solve problems not our ability to express ourselves not our ability to describe algorithms precisely We're still going to need to do that so So far we've spoken about what software development is as collaborative. It's exploratory. It's an iterative problem-solving method and

The output is code which is a precise and clear description of what machines should do and it's clear to humans so Now the crux of the talk AI is a foot gun for people learning how to code It's a foot gun for a lot of different reasons. Isn't this a cool picture?

So The reason people are shooting themselves in the foot with AI is because things are genuinely confusing When people are starting out in their coding career, they're not really sure what software development is They are given certain things to do and they're given certain advice and they follow that advice and then they're like wow This this does not compute So one thing that new developers are told to do is like go and practice their problem-solving skills and they'll use tools like hacker rank

And lead code and all of that So these are basically platforms where there are word problems and the coder will take those problems and they'll translate it Like come up with a solution translate it into code. And the thing is that

LLMs do that better That's what it seems like if you if you take a problem from hacker rank and feed it to to chat GPT Then it'll spit out an answer. And so it's like why are we even practicing the skill? LLMs can do it better Why not rather just get really really good at using LLMs and then get them to do a hard part

Which is writing the code writing the code isn't the hard part you guys so another way that another confusing thing is is this so There are a bunch of problems that new developers can solve and they're small problems and there's a few of them and they're this big and then there's problems that chat GPT can solve and they're this big and

For a person to become better than chat GPT They need to at least solve problems this big which are far out of reach of chat GPT Most software development problems are like much much bigger than that many many thousands of lines of code

That's are contributed to by like large teams of people over months and years all of that stuff is out of reach but AI is going to get better right like yes, definitely definitely it's going to get better It's they are going to be able to do harder things 100% They are going to be able to do things with larger token limits

They're not going to be as limited in terms of the amount of text They can take in and the amount that they can produce and so it's easy to say like Why should we even try to outpace the LLMs? Are we just going to be redundant like they're going to they're going to keep like getting bigger and better And if we're trying to stay a step ahead, is that just fairly pointless and

Personally I do not at all believe that software developers will become redundant I think things are going to change definitely But I think our capacity to solve problems and tell machines what to do that sticks around so I I definitely

Will say that software devs are not going to become redundant What I think we need to do is look at these problems in a different way So let's say you're learning a language like French or Japanese or whatever When you start learning the language you start off by translating simple things from one language to another You you start off by taking a word or a sentence a hello

How are you and you just translate them and you realize that this is a first step You realize that the goal isn't so that isn't to translate simple sentences The goal is to express yourself and understand other people's expressions. The goal is to unlock ideas and communication And it's very obvious when you're learning a human language

But it's less obvious when you're learning code when people start learning code They start off by translating simple problems into code and this is done on things like hackerrank But it's also done in like programming courses in programming textbooks anything like that Like Write some code that does this and you're like, okay, I will translate the

Human language stuff into the code language stuff. That's not what software development is But it's the first step Remember, we're not a translation service We are what software development is is a collaborative iterative exploratory process And code is the output of that process in order to do the stuff you need to learn to be literate

And so the first step of learning to program is learning that literacy it's not the whole step it's not the whole process So when you're learning to code, this is what it's like, which is why people get confused But can't an AI learn all the skills of software development?

So I mean that whole process of like having pushback and considering different ideas and researching different tools. Can AI do that? the thing is that it's like We're pretty far from AGI still and this is becoming quite a general problem. So There's a lot of hype to cut through

part of the hype so Something else that is is very confusing to people is that they believe that AIs can in fact Solve a certain level of problem without When it's a new problem, so you you give it a little word problem and it can output the code

and people see this in hacker rank and whatnot, but there are There has been some evidence that AIs or LLMs are only really good at problem solving problems that they've seen before so Jody, can you wave? Hey, so I've got these references from her talk at PyCon Italia and it was just freaking cool. So

Basically There's another like code practice platform called Codeforce and they published a bunch of problems before LLMs were really such a thing before ChatGPT was trained and then they published a bunch of new problems afterwards And then they're like hey ChatGPT solved these problems and it could solve the old ones and it sucked at the new ones

Even though they were at the same difficulty level there's strong strong evidence that these things have simply memorized the solutions rather than Being able to come up with the solutions themselves and in a lot of fields Memorization and mastery are kind of

Very interrelated. So if you wanted to be a lawyer, for example There's a lot of stuff you need to memorize if you wanted to be a historian You'd need to know dates and names and places And if you're really good at remembering that stuff and it's like, okay, cool. You're a good historian But with programming memorization is not understanding But people just memorize things in pattern match like that then that's called cargo culting and that is bad so

Another video with a great great thumbnail. This is jody's talk Which you should watch because it's full of references and it cuts through like a lot of the hype. Um yes, so In order to become a software developer. I want to start with literacy

You learn to write by translating simple ideas into code and then You learn to apply that literacy to solve small problems so that you can start to Apply those skills on larger problems. So so tools like hacker rank and lead code and all of that and code force I'm, i'm all for them. I think they're fantastic

Solve the problems yourselves so Let's just we've been talking about code a lot. Let's talk about this shiny shiny man. Um So, how do you think he got so good at lifting heavy things in his underpants it wasn't like this

Software developers we solve problems and we don't learn to solve problems like this problem solving is not a skill it's a set of skills and In order to develop that set of skills, you actually need to practice you need to do the reps you need to Work on your brain. So Then

Apply what you know to solve small problems so that you can learn to solve bigger problems The other thing is precision So these small problems maybe a lot of them can be explained in english Maybe they can be maybe you can be precise enough in human speech in human written language in order to describe the algorithm that you're trying to implement but

These are beginner friendly problems beginner friendly problems are Simple enough that maybe a blunt instrument can do the trick but If you want to actually get to a point where you can talk about very complicated algorithms, then you need to practice precision. So Sometimes you can solve things with a hammer, but it would be a good idea to learn how to use a laser beam

So, how do you do that? So precisely describe some simple problems actually use the programming language Again like learning the programming language is not the hardest part of learning to be a programmer And then once you have those skills, you can do bigger problems

so Remember that the most precise way to describe what a program should do is in code you need that skill. All right so Another angle to think about coding is you learn to code in order to learn to think And this might seem like a very self-important thing to say i'm like i'm a programmer I can think better than you but it's um

It's actually quite a solid thing because if you can learn to code you can get feedback on the way you think so If you can express your thoughts precisely Then you can get feedback from people So if you're working on a team, then your teammates can look at your code and say like have you considered? Um, did you think about doing it this way? Why did you do it that way?

You can get feedback from the world. So if you have that dog walking app Um that I spoke about earlier, maybe you will learn some things about people and their dogs and their spending habits and their walking habits You can get feedback from your future selves. So this is quite a big one if you are Learning if you're working on a small beginner friendly problem

Then it's going to be like tens or hundreds of lines of code and it's not going to take long Like maybe maybe minutes maybe a week something like that. It's small stuff But As you build bigger and bigger things you end up with these code bases that are huge like many thousands of lines of code

many people involved many um Like a lot of time spent you might write a piece of code And leave for like months and then go back to that piece of code and you'll look at that code and you'll think to yourself What bastard did this? I will find them And then you get blamed you're like, oh no

Oh, no is me and By going through that you you see your own Rational shortcomings and you see the things that you weren't paying attention to that You weren't diligent enough about where you weren't conscientious enough to be like, let me just do this right the first time you see the debt that you implemented and From doing that you get to be better you get to learn how to pay attention in different ways

And that's super powerful. So that act of just like often getting good at a thing is learning to You learn about what to pay attention to and then you learn how to react to the things that you notice And that's something that you get from working on big pieces of code in a very precise way, so

Let you harden your own thinking You guys have heard of sam altman this guy So he says you should learn to code learning code was great as a way to learn how to think I think coding will still be important in the future And he still writes code himself sometimes

Um open ai still hires developers as well if you need like a metric of how How useful it is right now um On top of that there's like the act of writing code, but reading code is also quite amazing So imagine one day you wake up in the morning and you can read japanese. That's like damn. That's cool um, you have access to great minds great thinkers you have access to

literature and other ideas and you you have access to a whole culture that you otherwise would not have access to and It opens doors Um in your mind and in your ability to affect the world and you get to turn the subtitles off on your anime Which is also a winner Reading code is a lot like that. Um

so If you read code you get to read the minds of your teammates, um code is a little bit like crystallized thought It's beautiful and you can see a lot about how a person thinks and how a person approaches problems by reading what they produce You can read the minds of powerful thinkers as well. So we are at an open source conference

and If you're working with open source tools, like even if you're not contributing to them Sometimes you're just like how does this work? Let me go read the source code and then you learn about how other people do things so It's important to learn to code. It's it's very very useful The next question is can ai be helpful in learning to code and the answer is redundantly. Yes

but but Remember our old friend There's this thing called an illusion of competence that comes out up like a lot So i've taught many people to code myself I've taught teachers to teach people how to code like they i've got a lot of experience with teaching people code

And this comes up so much an illusion of competence is when you think your ability is up here But it's really down here and i'm sure that many people here have personal experience in it or Have seen it in other people in other places So an example is like

Take yourself back to school Um, have you ever written a test where you thought you aced the test you thought you you like smashed it And then you got the marks back and it turned out that you flunked Um, that's an experience that many people have had where they're just like, oh, I thought I was good, but i'm not um another experience a lot of people have is like

Meeting somebody perhaps you wouldn't do this yourself But meeting somebody who'd read like three blog articles on the topic and now they're the experts That's also an illusion of competence comes up a lot so this guy richard feynman He's fantastic. He's a he's a physicist He got the nobel prize for physics and he's quite a genius and a multifaceted

Very entertaining person. He says the first principle is that you must not fool yourself and you are the easiest person to fool. So Nobody is beyond this. Nobody is above this and people are People at all stages are very able to to fool themselves

so Illusions of competence comes from a number of different things The one is a thing called the dunning kruger effect Which is that feeling that if you're slightly familiar with something and it feels pretty comfortable in your brain Even though you're only familiar with the tip of the iceberg you think that you understand the whole iceberg So that's the dunning kruger effect. The next thing is an overconfidence in your future self

If I learn a skill now and I master the skill now I feel like i've mastered it and i'm performing it and i'm doing totally fine. I will remember i'm confident in this A month from now i'll still remember I nailed that I know how to do it

but the actual memory of how to do it is very likely to go away these things decay and so Mastery is not housed in your short-term memory. If you learn a skill now, you might not retain it You are very unlikely to unless you do some some like intentional things about it The last one is hindsight bias. So this is a big one

If you see a problem and you see a solution to the problem and you look at the solution and You're you understand the solution. You'll think that the solution is obvious. It's it clearly follows from the problem so this happens a lot with people who are learning how to code they'll like Um, they'll see a problem they'll get the solution maybe from an llm and then they'll they'll look at the solution

They'll be like, yeah, i'm really clever I could have come up with that and they'll stroke their ego and they'll move on to the next problem But they couldn't have come up with it. So Mastery is something that you need to pursue in the beginning of a coding journey You don't need to master every framework and every every little thing but mastering the fundamentals is is quite critical

So mastery and familiarity are not the same thing even though sometimes they feel like the same thing Mastery does not come from nodding while someone or something else does a thinking for you It is not housed in your short-term memory. It comes from deliberate practice And it comes from recall. So there's a lot of science to how learning works and how education works

and if you learn a thing and you force yourself to Remember the thing later on then it gets more solid in your in your longer-term memory it gets solidified there The other thing is struggle and that's important So a lot of people avoid struggle if learning a thing feels hard, then they're like, oh, I must be doing it wrong

No, if it's hard, you're probably doing it right. You should be pushing yourself Um, there's actually evidence to say that the more Challenged you are by a subject the more you're absorbing and the more you're actually growing from from digging into it And so embrace the hard path So for people who are learning to code they should always try to do things themselves first

Give themselves the opportunity to be wrong If somebody has the opportunity to be wrong about something if they have the opportunity to stumble and fall And then they have the opportunity to upgrade themselves. They can see where their flaws are and get out of it If they get stuck they should get unstuck

So there's no point in just hitting your head against a wall Like trying to understand a thing or trying to get over a problem that's Actually, like clearly outside of your abilities to get unstuck. That's okay, but Be intentional about it. Don't have that as the first step. Don't be lazy about it. Say like, okay cool This reveals that I have a deficiency in this area

Try to recall those deficiencies try to build on those deficiencies, um get better over time And then the last thing that I like to recommend is the thing called. Um, well, it's a thing related to the protege effect So again, there's a lot of science around how to learn well and how to teach. Well, so the protege effect is The thing that says that if you learn something in order to teach someone else you actually learn it way way better. So

If I learn something simply with the intention of teaching someone else I'm going to pay attention to it in different ways and i'm going to think of different ways to explain it And if I actually go through the act of teaching someone else Then that's a form of recall I need to remember stuff so I can tell you I need to phrase things in different ways

And I need to take questions and sometimes if a person asks me a question The answer will be I don't know and that's revealing for me And sometimes when somebody asks me a question It'll seem seem so left field and off center that will make me look at things in a different way This actually it's very very useful

Because it helps you to build mental frameworks about the things that you know And it makes you retain the stuff better and understand new concepts better. So that is the recommendation So the future proof skill set in summary It's all about discovering and solving problems collaboratively. It's about algorithmic thinking. It's about communication with precision and it's about learning

So becoming code literate is a foundation to all of that the end Cool so we've got A few minutes for questions if people want to go up to the microphones we can we can go through these

um I've got a question. Cool. So if you I was wondering if you use llms in your work and if you do is there a Flow you would recommend to use it like this

Um, so in my personal work, um, so i'll I'll use something like copilot while i'm coding and it's really useful And it does speed me up and it stops me from having to look up the documentation every time I forget like that weird function name, which is great But I do find that if i'm trying to solve something novel I have to turn it off because otherwise it gets just very very distracting

So i'm always turning it on then turning it off then turning it on then turning it off I find um I turn it off at the beginning of a problem and then once i've got like the general flow of things and Once the the pattern of what i'm doing is is more obvious Then I can turn on the pattern matching machine and it can fill in some gaps for me and that's been very very helpful

And that speed speeds me up When it comes to people who are learning to code I say like don't even turn it on like don't That's not the right tool at this point in in the journey Like if somebody's just learning then that can just distract them make suggestions that they can't necessarily use Um so like one thing with using llms while coding is that you need to be able to

Take their suggestions and add a pinch of salt And people who are learning to code don't necessarily have the capability to add that pinch of salt yet They don't have that nuanced understanding and so It for them. It's it's a tricky dangerous tool. Yeah

Thanks very much I really like your focus on Um Isn't something I had clarified as a thought before so that was still processing that Um, I do have a question so Some of the ways of using llms in learning aren't necessarily completely obvious like simon willison, I think talked about um

Instead of getting the code getting the llm to write the code for him He Writes the code and then gives it to the llm and basically says how can I improve this? Which I think is really interesting especially if you're learning in an environment where there aren't other people around to kind of help you

Um, but given that llms also quite often make things up or have bad opinions Yeah, is there a way that somebody who doesn't necessarily know how to judge opinions? You know what I mean? Is is there a way to use an llm to learn in that way? That doesn't sort of potentially

Leave you vulnerable to going astray So that's also ridiculously difficult actually so i've tried to use llms to review the code of our students because code review is just a massive workload that we have and it's act it's pretty expensive to to have humans do it, but it's really tough to get it right because the llms do say very

um, so sometimes There's a few things to tune the one is like how strict they should be And a lot of the time if you don't Push it to be kind of strict. It'll just say everything goes it'll be like yeah, it sort of works. That's great um, or if you push it to be strict then it will

Be really over the top sometimes it'll say like every single one of your unit tests needs to have a full doc string it's like Are you sure? and so it makes up these requirements and a lot of the hallucinations from That it comes up with are You have to remember how the thing was trained as well, so

These llms were trained on a vast body of like open source code and not all code is good Most code really isn't like a lot of the repos on on github Or like a person learning a thing for the first time or a person trying a thing for the first time Abandoned projects a cool idea that died and so there's a lot of bad practices in there

Um, and so you can't necessarily just under like just take it as it as it goes Um when teaching people to code it's like cargo culting is a huge huge problem. Um it's really hard to get over and so if you have a

like an AI Auto marker thing that makes like hard suggestions like do the code like this so that it is correct Then a lot of people will just like believe it and do it that way without necessarily understanding So for us we haven't found a good way to Get the get the learners to interact

About like code projects with an llm some things that have been useful is um so we do have some automations around marking projects and whatnot and one of the things we do is we ask our learners to At some point in the program like write error messages that make sense to people and then we can use an llm to say

Like does this sort of mean this but then we don't really need an llm. We just need a way to vectorize sentences um Another thing that's really really helpful is recall and um asking weird questions So if we know that a learner has gone through like this text over here

Then we can ask them multiple choice questions if we want to we can ask them longer form questions as well And that thing that I was talking about earlier around. Um Dealing with like off-center left field questions from your friends that you're trying to teach like that's actually really really useful So one of the things that differentiates people's ability to learn there

There are a bunch but like one thing that's super powerful is a thing called structure building so a person who's a strong structure builder is a it's the kind of person who is more able to Take a collection of facts and turn it into a framework in their mind where these things are connected and related and affect each Other in different ways not everybody is naturally good at that

But you can take a weak structure builder And turn them into a stronger structure builder by getting them explicitly to look at their stuff from different directions And so we can say something to a learner like given this body of text How does this connect to that? and we can ask questions around that and we can ask questions around this and and so it's

And you can automatically generate these almost like conversation starters, which really help people Yeah Thanks very much. Cool. We have time for maybe one more quick question if you go up to the microphone. Um, yeah

Thank you. Great talk Shouldn't the people who try to enter software engineering those who take the harder? Uh way like learning the fundamentals shouldn't they still be worried that people who take the easier part will be favored by companies and in interviews

um So This this question itself is it's quite a good one because there's multiple layers to it so something that i've seen a lot of is resume padding where a person will get like Basic familiarity with like 20 different tools because they did the basic tutorial and they'll put all that stuff on their tv And they'll say I know this and so they'll look good and they'll get called to the interviews

Where somebody who's like genuine about the skills that they have and they're like, I know this one thing really well They just don't look as good. And so they don't get invited to the interviews and that's That's a big like ecosystem problem that needs to be solved But then when it comes to actually writing the code if the only way you can write code Actually, let's have a scenario. Let's say

um, you're my boss and you give me a project and you're like here is a Here's a specification that I want you to convert into code. I'm like, okay cool And so I take that specification I dump it into chat gpt. I generate some code and I give it back to you Yeah, i've done my work now you Didn't necessarily need to use me as a middleman and I couldn't check my work. I couldn't check for hallucinations. I could just like

Dump the stuff and then dump the stuff again So if i'm working like that, i'm i'm very likely to be replaced very very quickly Like I am already redundant so I think initially people might look slightly better in like an automated online test if they're like, you know, um,

Filling if they're doing like a hacker rank type thing as a test in order to get invited to the interview They can cheat the system and and like demonstrate skills that they don't have which is bad But the thing is that those skills are Foundational and you need those skills in order to build the biggest skills that actually build a meaningful career. So at

The interview level the things are very very messy a lot of organizations are now Shifting from remote interviews to in-person interviews because they're harder to cheat And that's really yeah, that's really rough. Um, yeah, does that answer your question? Okay, cool

Okay, let's put our hands together for sheena again, it was a great talk. Thank you