Good, should we start? I think we are more or less complete, right?

Good, I'm happy to give a presentation today on good AI. All of you obviously have heard and probably have done some work

on machine learning AI methods. If not, that's also not a problem. I'll start from the beginning. And I will not give an overview over machine learning techniques. I think there has been, or there will be already presentations about that.

I will rather focus on some issues which are also important besides the main algorithm, the main method.

Which focuses on what can go wrong in a sense with machine learning, right? So that should not deter you to use machine learning. It's a really good set of tools. But it should make you think a little bit before you use AI methods.

Because some of these things which can go wrong actually can be avoided easily. Some are more difficult to avoid, but some obvious ones you know what you have to look at. Good, so this is a site

which you can go to in the internet which is kind of a block by somebody who has worked a lot in AI and machine learning. This person is Kai-Fu Lee. He was director of Microsoft Asia for some time.

He was responsible for research at Google in China for some time and then he wrote this book which is interesting. It's kind of enthusiastic book of AI to come.

Very optimistic, also very in a way bold saying okay, now everything changes which we'll see what changes. But obviously it is true that some of these methods

are performing better than we thought maybe 20 years ago. And so if you go there, you find some pros or some slides, let's say,

which are in this vein. So he says I overestimated our power to understand ourselves, which is difficult. So we don't yet know how we actually think, but it turns out that some of these AI algorithms actually performed better than we thought.

And they're actually performing better in some areas like pattern recognition better than humans, which is quite amazing actually. Now this is one side of the coin

and this also led to the vision or to the vision. Or to the, maybe if you said negatively, the hype that this will actually change everything. We will progress. We'll finally be able to get an artificial intelligence

which does not just surpass humans in some specific quite narrow areas, but actually which actually can do what humans can do, which is if we think about it quite a lot.

And this is still part of the AI hype in certain areas today, kind of interpolating where we are now and moving ahead in all areas,

which is he also now then starts to acknowledge which is not true, right? So you cannot scale up AI methods in all areas in the sense that we get something which is as powerful as the human mind.

Now why is that the case and what can go wrong actually if we apply machine learning methods, if we try to scale up, if we try to scale wide in different areas, to discuss that a bit, let's look at machine learning,

what we actually have and what we actually do without going any algorithms. Machine learning in a sense is actually quite easy, but also powerful in the sense that we certainly and suddenly do not have to program computers

to do some specific task, but rather let them learn, right? Which is a powerful concept, which is how we learn language, how we learn different tasks.

We just learn, we do not have to be programmed obviously. And this is then true also for a computer who uses machine learning methods. Now how does this work actually? And the idea is very simple. You have a lot of data.

Let me see, you have a lot of data and then you have this algorithm which tries to analyze this data and which tries to recognize and find patterns, right? And regularities. And these patterns, these regularities,

it describes in a mathematical manner, which we call a model, right? This model basically represents these patterns, these regularities, and this is it, right? Based on this model then we can in a way use these patterns to predict things, right?

If we get a new input, then assuming that the pattern holds also for the new input, we can predict the output for this specific input in the same way that the existing input from the data we have put into this algorithm

are transformed into outputs through these patterns. And then we can do different things. Right now, going again in some specific detail, is a very simple example right here. We have 10 examples, 10 samples of something

where we have three attributes and these three attributes should be enough to predict whether the output should be true or false, right? And so we use our learning algorithm, we train a model, right? Which basically represents these patterns

from these three attributes to the output might be some decision tree, might be something more complicated. And then we have this model and then we have to test it, right? And we test it in the same way. We give some examples, some new example,

which we call a test set, give some new examples to the algorithm and see how the model actually works with these new examples. What is important is that these two example sets have to be from the same distribution so somehow they have to be similar, right?

If I, let's say, pick a random subset here in this room and I use this as a training set

and then use the rest of you as a test set, right? You are in some way similar as regards computer science or as regards students for this network. You are different in many cases and this is why it might fail

because the distribution actually is not really the same, you're all coming from different areas. And really this machine learning setup only works if you have training and test set coming from the same area, the same distribution.

Otherwise you learn patterns which are not true for another set of examples. However, if you do it like that,

you can actually get quite powerful systems and if we ask ourselves how good our algorithms, our models can become, then we realize that for tasks where we ourselves as humans don't have to think really,

we map some input to some output, we recognize an object, we understand the sentence, right? We don't have to think about that. If we have this kind of tasks, then we can build a machine learning system which is as good, maybe even a bit better.

If, and this is important, if our model is complex enough to capture, to represent all these similarities and if, and that's also important if we have enough input-output data as a training set and this input-output data should be representative

in some way for the whole set of examples which you can imagine and that's the first thing where something can go wrong, right? Maybe you have, that's too far.

You have seen this picture, right? This is the elephant. And if you, let's assume that these researchers, they can only kind of grasp the part of the elephant, recognize the part of the elephant which they can touch.

They don't see the elephant, let's say it's dark and then this first one says, okay, this elephant, okay, seems like a big snake, right? This, this is what an elephant is. This one actually says, no, no, that's not true. You have this trunks, this is what an elephant is, right?

And so everybody looks at an elephant in a very restricted way, right? The last one says, okay, it's a very little mouse, right? I have it in my hands, so it's obvious. But all of them are wrong because all of them have a correct set of examples,

but not a representative set. And that is what happens, obviously, if you have data which are not representative of everything you want to learn or to represent in that model.

Now I think you have heard in one of the talks about word embeddings and I think not many of you have actually looked at the technique and I will therefore go a little bit into this technique. There is this idea of having a computer

learn by reading, right? So what we are able to do, we read a lot of books and then we know. And that would be nice in a way to have a machine learn by reading, for example, reading all newspapers

which we have in a certain area in a certain time, right? Based on this, we get an understanding of the world as it is and we get an understanding on how things connect to each other.

And how do we do that? We basically learn how things relate, right? And let's keep forward and go ahead with a small example what does it mean how things connect.

Now I have here about 10 concepts, right? Like man, woman, boy, girl, princess, queen, king, monarch, which I want to understand. If I put this into an information retrieval system,

if I put it into a search engine, what the search engine does when it builds the index is just representing each term as it is and then if you search for man, gives you all the document where we have man, right? And this representation, right?

Representing a term by itself is called one-hot encoding because basically you use the terms as dimensions if we have here nine terms and for each term

we say either it's the term or not, right? So one of the dimensions is one which is what this term is, right? It's itself and the rest is zero. Now that's okay if you want to do information retrieval, if you want to search for terms in documents

but it does not help you to understand and it also is in some way quite redundant. You have a lot of zeros, always a single one in each line.

Now you can kind of do clever things to get away with this and not use much space, right? There are clever indexing algorithms but you are not understanding anything, right? Because a term is a term. Now one way to understand how to understand

what these terms relate is to introduce some dimensions which kind of try to characterize what the terms mean and here in this small example I use three dimensions.

I use the dimension male or female, right? Whether it's a male or female person which is expressed by the term use, whether it's young or old, when this person is young or old

and a certain dimension, whether this person belongs to royalty or not, right? Three dimensions and if I use these three dimensions, suddenly I can express the terms by putting ones or zeros into these three dimensions

connecting the meaning of the dimensions to the terms, right? So for example, a man is not female, man is not young, right? That would be a child and a man by itself is not necessarily a royal

so I have all zeros. A princess is female, is young and belongs to royalty, right? I have three ones and so I represent one of these terms by these three dimensions by saying,

okay, this is how they fit into these dimensions. This is a very clever idea and this is in a way the idea of word embeddings. I embed here these nine terms into this three-dimensional space and this makes things possible.

It tells me how terms relate and that's a powerful thing. And I can now find that similar terms

are in the same part of this three-dimensional space, right? I can also use this space to answer questions like analogy questions.

A boy to a girl is like a prince to whom, right? Would find the princess and this is what we have been doing here, right? So these word embeddings and this space where I embed my terms, these word embeddings allow me to say

these are semantically similar terms and this is how they relate to each other and this is very powerful. And in the real world embedding algorithm, of course, I do not only use three dimensions. I use kind of like 300 to 500.

And then constructing three to 500 dimensions manually gets really difficult, right? So I try to do this automatically and this is what word embeddings do. They construct this three to 500-dimensional space and put the words in the right part of this space

by analyzing texts, by analyzing how words are used together, which words are in the vicinity of a word, right? So if you have a word which has a certain vicinity, if you have another word which is used

in the same context, right, with the same vicinity, probably means the same, right? If you think about car and automobile, they are used in the same way so their vicinity looks very similar and will be in the same part of the space and if they're not the same but similar, they're a little bit farther apart

but still the words which go along with them are very similar. And so this is a task which has been tried for quite some time 30, 40 years ago, 50 years

when researchers started to work on AI and it's a difficult task. And here we get it for free. This is actually perfect. We have this algorithm analyzing newspapers, lots of them newspaper articles and we get this semantic space.

Now, however, there are things which can go wrong and one of the things which you can recognize if we again look at this dimension which basically represents male and female,

we realize, okay, fine, we have all these terms here, cousin, son, chap, boyhood, which are kind of male. You have all these others, fiance, grandmother, they are female. However, because this is a dimension which is used for all the words, somehow all the words have now this female

versus male dimension. And in many cases, this is not quite what you would put into an ontology in a definition of the term because, for example, librarian is not necessarily a woman.

A librarian can be a man or woman, depends who takes this profession. A magician can be man or woman. However, if we look at the words in this word embeddings, there are certain words which, and on the right side, we have this extreme he,

words which are more in this male space, right? We have philosophers, captain, architects. I think somewhere we have also genius, right? So it's just nice, right? But it's not true that obviously only men are geniuses. And on the other side,

we have stylists, housekeepers, hairdressers. And while it is true that in our society, many of these professions are more female than male, this is, of course, not a definition. This is something which reflects our society. And this is exactly the problem, right?

Because we gave as input to this algorithm texts which, even though they are neutral in some way, so all these are kind of respected newspapers which were used as input, New York Times, Guardian, whatever,

they still reflect society and therefore this algorithm, this machine learning system, it just takes data and recognizes and extracts patterns. And if in our society we find certain patterns, then we find it in the model, right? And therefore the model is not a semantic space

which defines these concepts, but rather it's a semantic space which reflects how in our society or in the society which is represented by this newspaper, particularly certain relations hold.

And that might be fine, right? If we are aware of that, we have to be aware of that, of course. But then if we use it, we have to watch out, right? Because for example, if we use it for translation, right? And translate secretary where it's not clear

whether it's a female or male secretary. If we translate that into German, German has male and female forms of secretary, then automatically it gets translated to secretarian, which is female secretary, right? Because in my semantic space it was more to the female

side, which is not necessarily what I want, right? And not even if I don't look at translation, if I use other feedback like in a search engine, search engines use feedback, what people want to see, right?

Now, if you do picture search, this is what I did, I think maybe half a year ago, if you search for secretary, you get pictures, right? You get a certain kind of pictures, right? Which do not reflect the definition of secretary,

but rather proto or more in this case, a stereotype, right? Yes, that's true, I mean, they're realized, right?

So if you know this, you can of course contact it, right? But you have to know it and you have to contact it. Also a French colleague told me in France, they never use the term World Wide Web, right? They use internet and therefore if you translate it,

or if you translated it some time ago, maybe you're sorry, now you translate World Wide Web into internet. If you translate internet back into English, you get internet, right? So this is what is wrong. And there are of course other things. So that's actually an article about an algorithm

which tries to evaluate whether a person should be let out of jail, right? Based on the probability whether this person will be involved in another crime in the future or not.

And we have few examples from male and female persons and it's known that people in the US, this is an example from this, also have prejudice.

So that the problem which we wanted to correct as computer scientists in this case was that maybe white people are treated better than black people, right?

So the solution, right? Make an algorithm which is impartial, which is based on all the data you get and you get an objective and better suggestion, right? Which however, unfortunately it's not true. This is the famous Compass case study.

There is this company which built this system. Unfortunately, we do not know how it is programmed but we can look at the outcome of the system. And there is this quite famous algorithm,

this analysis, this article in ProPublica by this association which talks about the bias in the outcome of this algorithm. If we don't know how the algorithm works, we can find out whether it's biased or not. So for example, these are two people.

The left one is a white male. The woman was rated low risk for a future crime. The woman was rated high risk for a future crime. But really, if you look at what they did

and what they did afterwards, it turns out that this was just a really very wrong suggestion that the woman in this case who was rated high risk did not reoffend the man did. Now you could say this is maybe true for this case because obviously prediction always just probabilistic.

But it turns out that if you look at many of these examples, and this is what people looked at, you find out that again, this impartial algorithm which looked at a lot of cases to train the model

was biased and treated white people better than black people. Now there's another funny example. You can use face recognition software by Amazon which is called Rekognition as a cloud service.

It's easy to use. Just upload some examples and then train a model and then you get a predictive model which for example you could in this case just use to find out whether two pictures

represent the same person. And so what people did was they matched all politicians in the US in the Congress and I think also in the Senate with 25,000 criminals.

Let's see if some criminal is sitting in the Congress. Probably there are some criminals sitting in the Congress but here we got wrong results because we got a lot of matches actually. We got 28 matches,

so 28 of the congressmen were identical to people in this criminal database which was wrong. So they were not really the same people. They looked a bit similar.

And interestingly, out of these 28 matches, 40% were of color, 20%, no, so in this false matches we had 40% of color

while in Congress we have only 20% people of color. So that means if you are black or if you are colored, then the face recognition software in this case works less well. Now Amazon afterwards said okay,

you use the wrong threshold, don't use 80%, use 95% and the result will be better. But that's beside the point. 80% was the default, so if you don't know anything, use just this great service. You get lots of wrong results

so this is why you should understand what you're doing before using any of these algorithms. And second, of course, this proportion between colored people in the false matches and colored people in the Congress that points to the fact that certain subsets of people

are not correctly identified by this algorithm. Another thing which is also related to what kind of data used to train an algorithm now,

so this is a comparison of two systems, Fisher system and DeepNet, both quite good systems. They recognize what is on the picture. They recognize, for example, that there is a horse on this picture which is fine and they work very well actually.

They have kind of the same position, more or less. And then a colleague of mine in Berlin looked at, or visualized what the algorithms were looking at, what kind of features they were using in this case.

And so the DeepNet algorithm looked here, right here. So this is kind of, he's looking at the horse, right? This algorithm looks at the horse, which makes a lot of sense. Now this Fisher algorithm looks here, right? I mean, there's no horse on the left lower part, right?

But if you look closely, you realize there is something written here which says www.federachief.de, right? So this is a caption which says it's from a archive about horses, right?

So somehow the algorithm realized, okay, if it looks into this corner and if it sees this caption, which of course it does not understand, right? The outcome should be a horse, right? Which is true, it seems, for the training set, right? Yet, of course, this is an example where the algorithm learns something

which is just true in the training set but not true in the real world. So if you come with pictures of horses, they are classified wrongly once they are not from this specific archive and do not have this caption, right?

But for the algorithm and for the test, right, it seems that the test images were kind of, again, the same distribution, right? So that is fine. Also, the horses came again from this horse archive but it did not represent the whole world in a way.

So these were the original numbers and they were skewed by having too many

of these horse archive pictures there, yes. So I mean, if you put only non-horse, if you put horse pictures which are not from this archive there, then it would go down to zero, right? Because, yeah.

Now, this is a Google car, right, the left. And the right also now, it look more like cars. In other times, it looked like it's an abbey, nice cars which are really nice and they don't do anything bad, right, which is on purpose, right? This is why it was constructed,

like it looks like a teddy bear. And they are driving around. They have been driving around in California, right, and driving miles and miles and tens of thousands and millions of miles. And this is not because people and researchers

wanted to test the algorithm in this car, the self-driving algorithm, but because they wanted this car to learn and look at the environment, look at what happens, right, because this is not a pre-programmed car, it's a learning car, right, and the more it drives, the better it gets, right?

And therefore, they let it drive for a long time in California, 10 million miles, seven million miles, and this works very well if you let the car drive in California. If you move it to Naples, right, it will crash, right? Because even though it did 10,000 miles or 10 million miles in California, right,

if you go to Naples, the traffic there is a bit different, right? And so big data is not necessarily representative data, right, so this is what this picture says. It also depends what you see from the world, right,

so this is, you can buy a Tesla, it works quite well, you test this autopilot, you should leave your hands on the steering wheel, right, because sometimes it does not quite work. Why does it not quite work? So it's not yet advanced, but there was, one of the first accidents was actually this one

where the car went kind of under a truck, right? So this truck was passing the road. It should not have done that, but it's another matter that happens sometimes, right, and the car which was self-driving, which was sensing the environment by radar and by camera,

went under the truck, of course it crashed, right, because it's too narrow. Why did that happen? If you look at the trucks in the U.S.,

they are empty here, right, so they don't have anything. If in UK, for example, you have this kind of sheets, right, so if you are a radar sensor, right, and you drive around in the UK, if you look around, you cannot really look

below the truck, right? You recognize there's a truck, but if you are in the U.S., then you look straight through because the sensor is very low, and then you have to fall back on the camera. Now the algorithm should have said, okay, if you are now using only the,

no, okay, then you have to use the camera, but in this case, the camera was kind of handicapped because it was very sunny outside. The camera could not really see, right, so if you look into the sun, you don't see anything. Also, this camera did not see anything,

and the truck was a light truck, right, so it all looked the same, so it did not see the truck, right, and the camera and the radar sensor did not work, and so in this case, it crashed. Now, okay, also this is fixed, of course,

but there's actually a nice, let's say, a report which was, which was just recently done in spring, in March, right, that was by a security company in China, Tencent Security. Let's look at what they are doing.

Let's see whether this works now here. That works differently. Let's, it does not matter if we don't understand things.

Let's look at it. So they are testing the Tesla autopilot in different situations.

They test, for example, the wipers. They are automatic. If it rains, right, they start, which is useful.

Okay, there it's not raining. You project a picture on the camera, onto the camera of this system, and it thinks, okay, it's raining, let's start, right? We'll look at it in more detail. So here, it does not work, right? You have this strange picture.

This is not so bad, but one important thing is, of course, following lanes, right? You should follow lanes, you should not cross them. So can you disturb this lane-following behavior, right?

So here's clear. You have this lane, you should go straight ahead, put a few dots on the street, right? Let it run, and somehow, these small dots disturb the system, right?

And it goes off, right? That's already worse than the swipers, right? Okay, and then there comes something where you can actually, this is our master hackers.

Okay, here they show how they can steer it using these remotes, which is another thing. Now let's look at what happens here, actually.

And let's look at this first in another set of examples. So we know already that on some benchmarks, algorithms, image recognition algorithms are really, really good. They're better than humans, so they recognize digits,

even if they're a bit strange, right? So this is a zero, this is a three. Looks quite strange. These are kind of fuzzy traffic signs, right? Still, everything is recognized perfectly. And now we change it a bit, we change it a bit. On the left is still a zero and a eight,

gets some dots sprinkled around. And suddenly, the system says, okay, no, this is not a zero and a nine. It's a five, four, instead of a three, it's an eight. Here's even more dramatic. So these signs look very similar, right?

I mean, a human looking at it, I would say, okay, nothing has changed. But the system says, okay, now you can go ahead. This is free to go. Or here you have to watch out, it's slippery. And here you have to stop.

So totally wrong if you have this system in your car. That's actually not really good, right? Another even more funny picture. This is not funny, this is dangerous. This is funny, right? So on the left, left side is a panda. It's not really good to see because you just see part of the panda, so the system is not really sure.

It says 57% confidence, this should be a panda. Okay, it's actually working well, right? Then I change it a little bit with this strange pattern which is invisible to see for humans, right? So the right side is still the panda, but now the system says, yes, I know it. It's a gibbon, right? And I'm totally sure, right?

So that's really strange, right? And this is not just one system. You can do this with quite a few different systems. And this is what happened with the Tesla output pilot, right? I hope nothing broke, right?

Okay, so what broke here, so that you have this kind of state-of-the-art image recognition network, and it just sees the picture and has to find out whether it's raining or not. And what people do is kind of, this is the right, the correct picture on the left.

You change it a little bit in a way which humans actually do not notice, right? So you do kind of very small changes on the right side. It still looks the same, but here in this situation, it suddenly turns out that on the left, it says rain score is zero, zero, one,

so there's no rain. Change it a little bit, still looks the same. It says, okay, rain score is 0.8, right, which is almost one. So it's raining, turn on the swipers. The lane-following behavior is a bit more difficult to trick

because the Tesla is really good in recognizing lanes. You can change the digital picture so that it kind of misses the lane. So there's still a lane here, but it kind of vanishes. If you try to do it in reality, right,

so on the left, you put a lot of dots on this left lane, it's even more visible, right? But it has the effect for the system that it says, okay, there's no lane anymore here. But this is actually very difficult to use because in a way, for humans,

it's obvious that there is a lane, right? I recognize it immediately that the system is doing something wrong, right? But what you can do easily is kind of to simulate lanes where there are no lanes, right? So the Tesla is very keen in finding lanes, right?

And this is what you can use. You put these small dots on the left side, which you don't see, or which you almost do not see, and Tesla sees the lane. And this is what was exploited, right? And so this is what is happening here,

that these are examples which are generated while you have in mind that you want to trick the system, so they're adversarial, and they perturb the input in many, many parameters

to a very small extent. But all these kind of perturbations, right, because there are many perturbations, very small ones, so we don't see them, but all these perturbations add up in the algorithm, and the algorithm crosses a line. It crosses a line from one class to the other class.

And they are not very specific to a particular network. They work across different networks. And so in a sense, it turns out that these deep models are not that deep,

are not so deep than we thought, right? So they do have some, let's say, locally linear borders, right, which are not true, right? So they learn something, and on all our examples they work well, but if we change our examples in this way, they fail.

Okay, so let's conclude what we learned, what I tried to bring across through this presentation was that we should not just use algorithm, obviously.

We should understand them. We should also think about some aspects of these algorithms which are not connected to the simple performance as usual, but which are connected to explanation. So if I ask the algorithm,

why do you suggest that the algorithm should tell me the reasons, right? Why did I not get a credit? Then the algorithm or the human who uses the algorithm should be able to say, okay, you did not get a credit because you did not pay back your last loan or whatever.

So there should be an explanation, not an opaque system like very often is the case if we use just a neural network without anything additional, right? There should be no discriminations for algorithms, right? And we saw this example in the compass case, right?

Because it's an algorithm, it does not mean it's more objective than a human, and we do not want this, and actually we are not allowed to do this. There are laws which say this is not allowed. And so as computer scientists,

we can think about the problem and should think about the problem, how to explain such an algorithm. It's a bit difficult on the model layer if the model is very complicated, might be difficult to explain,

but for specific instances, it's more easier, right? So I can explain why you did not get a credit, right? I cannot explain the model, how it works for everybody, but I can explain it for one example. I want this machine learning algorithms to be bias-free

or to say my data are biased and therefore you get wrong results or you get biased results. This is an important question, and we want them to be robust because we know there are lots of hackers out there, and of course we do not want to have an algorithm

which kind of fails too easily if I attack it. And then, for example, if we are a lawyer, then the question, for example, becomes what kind of regulations should we put, if any, on input and training data, right?

How should I select training data? How should I regulate the use of, should there be a human in the loop somewhere or not? Should it be allowed or forbidden in certain areas? Who is responsible if something goes wrong, right? If this car crashes, right, who is to be blamed?

Right, the car, the manufacturer, whoever. I mean, right now it's the human who drives the car, right, because they say you should not rely totally on the algorithm, right? So the driver stays responsible. If you have really self-driving cars, it becomes different.

And all of this can be maybe summarized in this nice picture, right, which sometimes describes how people use machine learning algorithms, right? So you have data collection, which is difficult, data pre-processing, which is difficult. Then you do model training, and then you have this miracle, and you get the result,

right, there are no miracles in AI, and you should not believe him, then you should be able to understand what happens, and that's the better choice. Thank you very much. Any questions, comments, yes?

Yeah, yeah. Well, so for the face recognition example, you can, of course, get better, right? You might not be perfect in all cases, but if you get people from all over the world, right,

which I guess they did afterwards, then you get better. It's interesting to note that also we as humans are not quite good in finding diversity in faces which are from somewhere else, right? Because also our brain is trained

on a certain kind of examples, and maybe the best answer here is to say, don't rely totally on the system, and be aware that some things can go wrong, right?

So you try to be as representative as possible in your training examples, right? That's already very, very important, and then you probably get better even than a human, right? But still, if you get a really strange example, then things go wrong.

Yeah, regarding these attacks is still a bit unclear how, what is the best way to counter them, right? Because these attacks are something different. They're exploiting how you train these networks, right? So basically, these networks are trained by using this function which says,

okay, in which direction does the function change more, right? So you go, you do some kind of hill climbing, right? You go in a specific direction and to change the output.

So if you find the input leads to the wrong output, you look at this training function and say, okay, I have to change it a bit, and because you don't want to change too much, you go into the direction which changes most the output

while changing not very much in the network, right? So you, and the problem with these activation functions is that you add all these small changes, and people have tried to train with kind of

changed examples, right? So you can take this attack examples and put them as input and say, what should you get? And that works to a certain extent, but it does not really work, right? The problem somehow is the model, not the training process,

and there is not yet a complete solution to this problem. Right? Yeah, that's true. So that's still a interesting research area, right?

Yeah, so it would already help if people say, okay, this system was trained on this and these data sets, right? If you go out of these data sets, be aware that it might not work well or it might not work at all. So that would already be helpful. So finding out that something can go wrong is already,

and children are also more diverse, right? So, I mean, if you learn a new language, you realize that if you want to understand children, it's more difficult than understanding the adults. So that adds to the problem, right? Yes.

The ability is going to be acceptable or satisfactory. So there will be, so I think what we've seen in programs of CNA, you can see, for example,

with that, we're going to reach a stage where we'll have models that,

or is it just a continuous scale? So the first thing is that as you also correctly pointed out,

it's easier to explain certain instances and not the whole model, right? So it's easier to explain why this is the picture which shows lung cancer or not, right? Tumor or not. Or why this algorithm recognizes a horse and how. So you don't explain the whole model,

you explain why in this particular example, right? You get this and this suggestion or classification. And that works already quite well. Still an open, there are still some open issues, but it works well. Whether you get perfect.

explainability I don't think so because we cannot even explain why we make decisions right so also a doctor cannot always say exactly why he came to this diagnosis that he can explain it to some extent afterwards and it's

actually funny if you look at psychological literature that the pain is actually very good in explaining things right post-hoc right so it makes a decision unconsciously and then it explains why it has done this decision which is a rationalization of something which was unconscious so your

rationalization is actually not how you got this decision it's just a post-hoc explanation right and this is what we might be able to achieve here there are these interesting experiments right where you cut I mean some patients have you have these two parts of your pain and they are

connected if you cut it because because of some illness or so then your left brain does not know what the right brain knows that and then you look at a picture with one part of your brain and they do the explanation with the other part and and it's totally unconnected right so the left

brain explains something which which which it thinks the right brain does but is it's just just comes up with an explanation yes yeah so right now

actually everybody says okay I'm doing AI right because this is what is the great thing and and might be totally simple right in ten years this hype

will be gone right and then we will have products which will work well but they will not be magic in the current in the current situation I mean sometimes you are forced to argue against this kind of view right this

are miracles right this is the system which can do everything because also then people tend to use it wrongly they say this is magic I just put in a few X so so one one X one thing for example is if you do some if you

have some recognition task and if you are I don't know mechanic engineer whatever then the ideas okay so for example finding out where the machine

takes right so so that's called so if you have to speak machines which which are used in production there are serviced every year or every half year every two years to make sure that they don't break you don't know exactly when they break of course so you service them more often than

necessary which wastes money and time so there's this area of preventive maintenance which tries to find out whether a machine is going to break in a near future right and then you service it so this is the task right so find out whether a machine breaks before it breaks right reliably enough that you

can service it and otherwise you don't service it it's actually important task if it works it's perfect but it's really hard to implement because these machines usually do not break right they are really good if especially if they're German machines right two machines right they break once in a few

years right so where do you get the training data that the machine is going to play you don't have enough training data right so people kind of collect all or send sensor data from this machine for five years right and almost all all of the time it does not play right and then you have one or

two example where it's like where it's like and then and then people sing okay because I have to say I it learns when it takes it's not true you have no examples right and and the question you can ask then to such a person is would you as a human looking at the data let's let's assume we can

do that I mean we are slower in looking at the data but in principle right would you find a pattern which would allow you to predict whether the machine is going to break right and if you think about it and you have just one example you can immediately say no it's impossible right I have one example I cannot generalize it to right but so that's a good question right to

find out whether you have enough data about what you want to learn so that that's that's something you can do to kind of move away the magic a little bit and say okay if you as a human are not able to come up so if you with a

solution right why should the algorithm do it it's it's simpler in a way it's faster but it's simpler than you yes no no no it really depends on

it depends on it depends on the application right so if you're a doctor

in medicine explainability is very important right because because as a patient you want to know why a diagnosis is made as a doctor using such a system you want to understand why the system gives the suggestion right because otherwise you're liable so explanations are really really

important if you do recognition of digits it's not so important you want explanation if something goes wrong but usually you don't care you just use the system as is so it really application dependent

so so that so one way to get explainability is kind of to fit a simpler model to your example right so you might have a complicated model which is necessary to handle all the examples you saw at it and therefore by

definition it's very difficult to explain as a model right but then you could kind of like your brain does you you you come up with this explanation right which which uses a very simple model which is which is true not for

everything it but it's true for this particular example and say this is this very simple decision tree which says if this and this is true then this is the outcome and of course that's a post-hocs explanation right of a much more complicated model but it's sufficient to explain certain instances

so that's one one one possibility to do this right so use perfect complex model to get high performance and then use a simple model to explain certain cases

definitely so so I mean if if you start being aware that your model is restricted by your training set which if you think about is obvious right but but too many people don't think about it and then you kind of try to extend

your training set your your your examples how you test it and also if you for example maybe manage to to kind of implement some kind of meta reasoning right which sits atop the model and kind of recognizes if you go out of bounds right if your system now is in a situation where it did where which is

not covered by the by the training examples right that's already a good solution yes definitely