We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Lies, damned lies and large language models

00:00

Formal Metadata

Title
Lies, damned lies and large language models
Title of Series
Number of Parts
131
Author
Contributors
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Would you like to use large language models (LLMs) in your own project, but are troubled by their tendency to frequently “hallucinate”, or produce incorrect information? Have you ever wondered if there was a way to easily measure an LLM’s hallucination rate, and compare this against other models? And would you like to learn how to help LLMs produce more accurate information? In this talk, we’ll have a look at some of the main reasons that hallucinations occur in LLMs, and then focus on how we can measure one specific type of hallucination: the tendency of models to regurgitate misinformation that they have learned from their training data. We’ll explore how we can easily measure this type of hallucination in LLMs using a dataset called TruthfulQA in conjunction with Python tooling including Hugging Face’s `datasets` and `transformers` packages, and the `langchain` package. We’ll end by looking at recent initiatives to reduce hallucinations in LLMs, using a technique called retrieval augmented generation (RAG). We’ll look at how and why RAG makes LLMs less likely to hallucinate, and how this can help make these models more reliable and usable in a range of contexts.
Formal languageEndliche ModelltheorieProcess (computing)Object-oriented analysis and designOrder (biology)Multiplication signFormal languageMoment (mathematics)Endliche ModelltheorieView (database)Computer animationLecture/Conference
SpacetimeOrdinary differential equationComplete metric spaceComputer iconEndliche ModelltheorieSqueeze theoremWebsitePlastikkarteFormal languageLinear regressionMathematical modelFormal languageComputer architectureWindowOpen sourceNatural languageMaschinelle ÜbersetzungData compressionContent (media)Wave packetRow (database)Type theoryVariety (linguistics)Multiplication signFunction (mathematics)Range (statistics)AirfoilSpacetimeTask (computing)Electric generatorParameter (computer programming)Context awarenessWordEndliche ModelltheorieSemiconductor memoryTransformation (genetics)Set (mathematics)MereologyDependent and independent variablesSoftwarePredictability2 (number)BitDifferent (Kate Ryan album)Formal grammarReal numberCartesian coordinate systemPoint (geometry)Software developerWeightOrder (biology)AreaInformationStructural loadPower (physics)ScalabilityFamilyWeb 2.0CASE <Informatik>Parametrische ErregungSequenceScaling (geometry)Speech synthesisRule of inferenceProcess (computing)Web crawlerLecture/ConferenceMeeting/InterviewComputer animation
Endliche ModelltheorieSign (mathematics)Web crawlerRule of inferenceFilter <Stochastik>QuicksortCombinational logicMeta elementLink (knot theory)Content (media)Wave packetOpen sourceNatural languageDigital photographyControl theoryInformation retrievalMathematical modelType theoryEndliche ModelltheorieWeb crawlerWeb pageProjective planeSet (mathematics)InternetworkingProcess (computing)Category of beingRange (statistics)Source codeDifferent (Kate Ryan album)Cartesian coordinate systemRevision controlMathematicsVideo gameAugmented realityBounded variationWordReal numberBlock (periodic table)Axiom of choiceElectronic mailing listTheoryWeb 2.0Power (physics)Core dumpTask (computing)Annihilator (ring theory)Order (biology)2 (number)Multiplication signBit rateFeedbackInclusion mapTunisMobile appOnline chatExtreme programmingMeasurementParametrische ErregungLine (geometry)Scaling (geometry)BitVector potentialSimulationBlogPiLagrange-MethodeBookmark (World Wide Web)AirfoilLinear regressionComputer animation
ChainTemplate (C++)Dependent and independent variablesMathematical modelView (database)Codierung <Programmierung>Mathematical modelOpen setEndliche ModelltheorieAxiom of choiceData dictionaryObject (grammar)Revision controlSet (mathematics)Electronic mailing listMultiplicationTurbo-CodeField (computer science)Different (Kate Ryan album)BitContext awarenessOrder (biology)Bit rateChainLoop (music)Multiplication signOpen sourceFunction (mathematics)MereologyCASE <Informatik>MeasurementComputer programmingPerspective (visual)Exception handlingBookmark (World Wide Web)Selectivity (electronic)Phase transitionSubsetComputer animation
AirfoilControl flowMultiplication signParameter (computer programming)Formal languageSet (mathematics)MeasurementTheoryEndliche ModelltheorieBit rateGodTask (computing)Hydraulic jumpNumberMathematical modelWave packetInformationShared memoryElectric generatorDependent and independent variablesLatent heatBitComplex (psychology)Order (biology)Sheaf (mathematics)Slide ruleVariety (linguistics)MultiplicationFunction (mathematics)Moment (mathematics)Context awarenessoutputSource codeNatural languageType theoryProcess modelingInformation retrievalDemo (music)Line (geometry)Domain-specific languageCollaborationismReduction of order2 (number)Computer animation
Abelian categoryCategory of beingFrame problemTurbo-CodeHand fanTotal S.A.View (database)WindowRootSingle-precision floating-point formatSlide ruleShared memorySource codeTouch typingProbability density functionMathematical modelVirtual machineBit rateEndliche ModelltheorieFormal languageQR codeComputer animation
Formal languageEndliche ModelltheorieBit rateSlide ruleGoodness of fitLecture/Conference
outputFunction (mathematics)Multiplication signSemiconductor memoryEndliche ModelltheorieConsistencyContext awarenessOnline chatMathematical modelSlide ruleWindowRight angleLecture/Conference
Wave packetMereologyBit rateDirection (geometry)Lecture/Conference
GoogolMeasurementSoftware testingKey (cryptography)Performance appraisalSet (mathematics)MereologyWave packetValuation (algebra)Arc (geometry)AbstractionRight angleLecture/Conference
Channel capacityEndliche ModelltheorieSet (mathematics)Goodness of fitCross-correlationFormal languageData qualityLecture/ConferenceComputer animation
Endliche ModelltheorieFreewareMathematical modelBenchmarkOpen setSmith chartArchitectureLattice (order)PressureSlide ruleNumberOpen setWhiteboardMathematical modelRight angleInformationAnnihilator (ring theory)Multiplication signLecture/ConferenceComputer animation
RootSystem on a chipMultiplication signComputer animationLecture/Conference
Transcript: English(auto-generated)
So this talk is what you need to get through in order to get to lunch, so thank you for being here So as Seb said, my name is Jodie Burchell. I work at JetBrains as you can see This is not a sponsored talk But I just want to acknowledge they did pay me for my time to prepare this talk and also my travel here
So just a quick thank you to them So today I'm going to be giving you a talk about a very hot topic at the moment, which is hallucinations Hallucinations in large language models and as you can imagine if you have read anything about this This is a huge and extremely complex topic. I have half an hour
So I'm gonna be able to give you the 10,000 foot view But what I'm hoping you're going to leave here with today is an idea of why hallucinations happen in LLM's how we can classify them and measure them and Some things that people are doing to try to mitigate them
So anyone who's used an autoregressive large language model That is a large language model that can generate text Would have seen the ability of these models to come up with coherent Sometimes useful texts on a wide variety of topics and in a wide range of styles So for example, I prompted chatgpt4 to give me a Shakespearean sonnet about the night sky
And this is what it came up with Upon the velvet cloak of night's embrace the stars like jewels in the heavens dance their light a Silent song that spans the space a tapestry of fate and sweet romance Like something the bard himself could have written
But autoregressive LLM's are also well known for when their content misses the mark So take this example of when someone asked chatgpt 3.5 How do I cross oh no, what is the world record for crossing the English Channel entirely on foot and at the time chatgpt returned the following
The world record for crossing the English Channel entirely on foot is held by Christoph van Rache of Germany Who completed the crossing in 14 hours and 51 minutes on August 4th 2020? this is complete nonsense and Those dates and those times are made up out of thin air
So this tendency of LLM's to generate incorrect outputs is of course called hallucinations And these pose a major problem for the application of LLM's to real-world applications So to understand why autoregressive LLM's can produce both coherent helpful texts and also these insane
Hallucinations we first need to understand the ingredients that go into these models So the first was the development of a type of neural net architecture called of course the transformer architecture I think it needs no introduction at this point in time So transformer architectures were originally developed in
2017 in order to do machine translation But they were quickly adapted to other natural language tasks and one of the main reasons that transformer models have become so Powerful so popular is that they're much more successful than previous models at Working with a lot of data they tend to scale very well
compared to previous models like long short time memory networks or LSTM's and they're not only able to process much Larger amounts of text a bigger context window, but they can also efficiently train on much larger data sets And this of course leads to the second ingredient in the success of autoregressive LLM's
So when we talk about these models We're generally talking about models that are based on a specific part of the transformer architecture the decoder if you want the difference between the different types of Transformer models explained hugging face has this incredible NLP course that approaches this really from first principles
Even if you have no background in the area You just know some Python and you know a little bit about neural nets you can go through that course pretty easily So I do recommend it So decoder based models are trained using millions or most likely billions of example sentences and they're asked for each sentence To predict the next word in the sequence and what's really cool about this is it means that the training data?
scales very easily because you don't really need to do any manual labeling as you would with what's called a fully supervised type of Learning all you need to do is split a sentence at a point and you use the first part of that sentence as the Import and you use the next word in that sentence as a target
So now we have two ingredients that beautiful scalable transformer architecture and a whole load of training data and researchers have been able To combine these to make a bigger and more powerful autoregressive LLM's So let's have a look at the case of the GPT family Do you want to emphasize we're at an open source conference? The GPT family is not the only one worth talking about but GPT one was the og model in this area
So it gives us a nice illustration of how these models have evolved so GPT one was 120 million parameters, which seems very quaint at this point in time The second one was 13 times that size at 1.5 billion parameters and the models have just kept growing and growing
GPT for is estimated to be at 1 trillion parameters and as they've grown they've gotten better and better Doing a range of natural language tasks So we can get a sense of how LLM performance has improved over the generations By looking at how each of them responded to the exact same prompt complete the following sentence
Belgium is I do promise I'm not taking potshots at Belgium. I do love Belgium, but you'll see So GPT one is very good at producing sentences that are grammatically, correct But there's no real sense of what the context of the words are This is the attempt of GPT one in response to our prompt
The best you can say about it is that it's grammatically, correct? So GPT 2 is a bit more sophisticated. But again, it's really just doing grammar prediction. So this is GPT 2's attempt Yeah Now GPT 3 was where the model started learning not only grammar
But also started encoding some information about the training data that it was seeing So for our prompt GPT 3 outputs this nice little sentence it's on topic and it makes sense and then GPT 3.5 output a whole bloody essay
So why is it that we're seeing much more sensical outputs from GPT 3 onwards? Well under the hood These models are essentially doing what Andre Kapathi calls a lossy compression of their training data And as anyone who's worked with lossy compression algorithms knows the less that you have to compress the data the more you can conserve
So the smaller models GPT one and two they had to compress their training data so much That they could only really encode the most commonly seen information grammar rules like parts of speech or the word order But as the models got larger they could conserve more and more information
By the time we hit GPT 3 and models around this size They have enough parameters to encode knowledge that they're exposed to frequently enough something that's become known as the parametric knowledge of large language models and This of course leads us to the data that these models are learning from
So most of these huge text data sets and these are truly huge text data sets They're sourced of course from at the web and the most important of these for training order aggressive LLM's to date has been a common crawl Now common crawl is a project that was started many years ago, and it was originally designed to mimic a Google's crawler
basically finding the most commonly linked pages on the internet to provide a Huge open source data set for research into natural language processing and information retrieval So because it's a research data set it's provided unfiltered and what this means that if you're an LLM creator and you want to use it to train your model you need to apply some sort of filtering to it and
Because it's so big you can't do this manually So this filtering is done through a combination of text classifiers or rules and you know Obviously these are probabilistic approaches So earlier filterings of common crawl focused on doing things like removing low quality
Duplicate or offensive content so GPT 3 for example 80% of its training data came from such a filtering of common crawl There's a similar similar filtering called C for this was used to train Google's t5 and the original llama model from meta
However, there's this fascinating paper. I've actually linked at the bottom I'm providing all my links at the end. So don't feel that you need to take photos basically, the Washington Post did this really interesting investigation of c4 and they found that it contained a lot of pretty messy stuff offensive racist of pornographic content and
Quite a lot of the content came from dubious sources So 4% of the content of c4 came from personal blogs, and I'm sure you've seen the sort of quality of stuff That's in those So more modern sources for training these models have tried to correct for this so last year a much more aggressively filtered version of common core crawl was created called refined web and this was used to train the
powerful Falcon models and Data sets such as the pile try to focus on a higher quality data sets They only use common crawl for around 18% of the text source and they focus more on sources like PubMed Gutenberg and Wikipedia But of course this earlier reliance on
Inadequately filtered versions of common crawl has left its mark and many of the foundational models that we're still relying on Have been trained on these inadequately filtered sources So just an aside before I go on one final thing we need to think about is When we think about models hallucinating is we need to understand the difference between base llms and their application
So so far when we've been talking about llms We've just been talking about these models that was simply trained to complete words in a sentence But these models by themselves are not that useful and they usually have some sort of changes or Augmentations made to them to make them much more useful and these changes can influence the hallucination rates for better or for worse
So I do not have time to go into it right now I'll be happy to talk about it after the talk, but this can include things like instruction tuning reinforcement learning from human feedback Temperature and of course the difference between a model and the model in a larger app like
GPT versus chat GPT Okay, so now we know how llms are trained and applied we can now talk about the two types of hallucinations that these models can Have the first are faithfulness hallucinations This is when order aggressive llms attempt to do some sort of natural language task
over a piece of text such as summarization or question answering and When they do that they deviate from the text somehow So for example, let's say we want our LLM to summarize this piece of text about the moon landing The text clearly states that the landing happened on July 20th, but the walk happened on July 21st
But if the model incorrectly summarizes and says that the walk happened on July 21st This would be a faithfulness hallucination The second type of hallucination are factuality hallucinations And this is of course when the model has learned something incorrect from the training data Now it might have been something that was correct at the time that the model was trained
Like who the president is and it's now out-of-date knowledge, but sometimes it's straight-up wrong from the outset so For example, if we ask our model who the first person was to walk on the moon and it says Charles Lindbergh this would be a factuality hallucination and This is the type of hallucination. I'm going to focus on most during this talk
So let's have a look at how we might measure these two types of hallucinations There are many many many different ways of doing this. I'm just going to touch on it So for faithfulness hallucinations, you're generally just trying to see how accurately the model can perform a range of natural language tasks So for example, we have the squad version 2 data set
Which assesses how well models can do question-answering and there's another data set the extreme summarization or ex-sum data set which attempts to see how well models can do one sentence summarizations of longer news articles and Factuality hallucinations on the other hand are unsurprisingly Assessed by seeing how well an LLM's parametric knowledge lines up with reality
So we have the halleu eval QA data set this awesome LLM to check whether a fact asserted about a topic is a Hallucination or not and in this talk we're going to be diving into one of my favorite data sets. I love measurement
I used to be a psychologist Basically this checks how well an LLM has learned widely circulated misconceptions and conspiracy theories So let's dive into the truthful QA in a bit more detail It's a scale with 817 questions across 38 different categories of potential falsehoods. So examples of these include health and legal falsehoods
Conspiracies and fiction. So let's have a look at some example questions For health we have the myth that coughing can stop a heart attack and for legal We have a question getting at the false belief that you're not allowed to marry your first cousin in, California Turns out that's actually legal
For conspiracies we have a question about what happens if you wear a tinfoil hat and then for fiction We have what happens if you meet a vampire in real life. What should you do? So there are three variations of the truthful QA data set two of these are multiple choice So we can see here an example of that an LLM is asked to pick from this list of possible answers
Which is the most likely to be correct? So you can see that if the LLM picked if you wear a tinfoil hat you can block mind control signals This would indicate it had internalized some sort of conspiracy theory and if it picks nothing in particular happens It has not internalized that
So let's not jump over to PyCharm to see how we can use Truthful QA in order to there we go. There's my mouse measure the hallucination rate of a model Okay, so we're going to be using two data sets We're going to be using the data sets package from hugging face in order to work with the truthful QA data set
And we're going to be working with the Lang chain package in order to work with LLM's So first things first We're going to read in the data set using the data sets package We read in the truthful QA data set from hugging face. And in this case, we're using the multiple choice version
We get back this data set dictionary object this gives us the questions But it also gives us these two target fields So the difference between these is that both of them provide a list of possible answers each of the questions But in MC1 only one of them is correct and in MC2 multiple of them are correct
I'm gonna be working with MC1 because it's a bit easier to work with from a programming perspective But of course you can use either Having a look at one of the questions our old favorite what happens if you wear a hat made out of tinfoil You can see here. Here are the possible answers and at the top here. We have that correct answer
Now let's talk about loading in the model. This is the Syntax for connecting with an open AI model that I'm going to be using There are a bunch of different models. You can use including open source models with Lang chain I'm using GPT 3.5 turbo and Once we've connected to that API we can pass in a prompt translate from English to Spanish
I'm happy and we get the correct answer is toy Feliz Now we need to combine the two So anyone who's worked with an LLM would know that one of the trickiest things is phrasing the prompt correctly And it's no exception in our case. We need something very specific. We needed to consider a question
to consider a list of possible answers Select the correct answer and select only one answer. Oops, sorry and Select that answer and return it without any elaboration. So that's what we've done in this instruction set What we've been done is passed in one question at a time and the list of possible answers
And then we then pass that over to GPT 3.5 And what we get in return for question for our one about the tinfoil hat is exactly what we asked for One answer without elaboration and it's even given us the correct answer
Now we need to do this for every question being a data scientist I'm going to use a for loop we're going to loop over every single one of those questions the 817 items and for each we're going to ask for the output to be returned and we're also going to do two checks Is the output one of the valid answers in our list of possible answers? And if so, is it correct?
So now we get to the fun part we can see how many questions it got correct and in this case GPT 3.5 got 36% wrong. So this is a relatively high hallucination rate But we're gonna have a chat about that in context now if I can find my mouse. There we are
Okay, so now that I've shown you how to measure the hallucination rates of an LLM I'm actually to tell you you don't need to do this yourself So you're ever thinking great. Why have I just spent 20 minutes sitting through this lecture?
Well, the thing is is measurement of hallucinations measurement of anything with LLM's in particular measurement of anything is tricky and As you saw with our little run-through with truthful QA, it is measuring a very specific thing It is measuring the rate of hallucinations
in a Hallucinations as measured by widely Circulated misconceptions or conspiracy theories in the English language So have a think about that and have a think about the fact that this is not really giving you the hallucination rate It's giving you something very specific within hallucination rates
So having a careful think about how you're measuring how good your model is based on things like performance or hallucination Measures will help you make trade-offs when you think about what you really need your LLM to do. Well So let's end by having a look at what we can do to mitigate
Hallucinations and LLM's as we saw even fairly state-of-the-art models like GPT 3.5 are still hallucinating around one time in three according to this specific measure Now this is of course because the data that was trained on is full of Misinformation and conspiracy theories and the model just learned it during training The problem is is training a three hundred and fifty five billion parameter model is really expensive
So we can't really easily just go back to the drawing board and throw it all away And the thing is even if we could do that putting together a data set that is free of misinformation Conspiracy theories all of this bad stuff. It's not actually that simple as again we saw
So instead what we can do is work with the models we have now and there are a number of initiatives To try and reduce the hallucination rate in these models So the first is crafting prompts that make these models less likely to hallucinate So the general idea is LLM's are general purpose natural language processing models
And the way you can get them to do this specific task you want them to do is by including as much relevant information Examples things like this in the prompt as possible. We've got a little taste of that during the demo The second is fine-tuning This is where you create a data set specific to your problem domain and it's pairs of high quality
prompts with high quality outputs and what this means is the model will learn to give Responses that are more in line with the type of outputs you want it to and it will learn information At the same time about your problem domain
The third is self refinement and collaborative refinement of outputs this involves a variety of methods which either get the LLM itself to evaluate its answers or use multiple methods or multiple models to compare outputs between them and then finally There's one of the most talked about methods at the moment retrieval augmented generation
This is where additional context which is relevant to the input prompt is Incorporated into the prompt It's retrieved from some outside source and it's passed in along with your original prompt to get a more accurate answer from the model And then of course you can combine any or all of these techniques in order to reduce a model's hallucination rate
Now I'm actually running a bit low on time So I have a little section on rag that I was going to show I think given time I might skip it and jump straight to the end If you want to know more about the complexities of using rag in order to reduce hallucination rates I'm very happy to answer any questions. I can run you through those slides in a little private session
But god that sounded so bad. We can go through it. We can go through it one-on-one But for the meantime, what I'm gonna do is just jump to the end and share my socials Okay, so Well, I had to cut the shot the talk a little short
What I hope I've done is planted the seed here of how complex it is To work with large language models with this extra ability, you know that we haven't seen before with machine learning models to produce misinformation What have I I hope I've also gotten you to do is plant the seed to go away and really critically evaluate
When someone says that a model performs well or has a low hallucination rate So this QR code here will take you to all of the sources that I use for this talk It'll take you also to a PDF of the slides for every single slide I've linked where the source came from really recommend that Washington Post paper
It was very good And then of course here are my socials if you want to keep in touch I'm gonna be around for the next two and one and a half days. Thank you Wow, thank you very much for that talk. I do wonder what my own hallucination rate is. I don't know
Have a little thank you for you. Thank you. We do actually have a few minutes for a few questions I don't know if there are any questions so you then you can please step up to the microphone Maybe I'll get to show my backslide Maybe yes, I do see someone approaching the microphone. So go right ahead
Sometimes I use large language models to create fiction. Can I get a high hallucinate late? Hallucination rate because it's all made up. Anyway, that's a good question Yeah, this is on so This is a great question. I think we can really classify
hallucination as something that is actual misinformation, right So if something is fictional ready I don't think there's really a reliable way to measure that because what is your expected outcome? No I would say probably the thing you're looking for and this has really been a problem and Is getting better as the context window of these models get larger is internal consistency
so The problem is is every time you restart a chat with an LLM it loses its memory Because as you chat with a model in a back-and-forth fashion all of the previous Things that you have you have input to the model and its output are bundled up into the context
But say you're trying to write a whole book You're probably gonna have to do several chats, right? So then the problem is is suddenly your protagonist has I don't know Lives on a different planet and has no memory of all the people that it knew That would be not quite a hallucination, but at least an internal consistency
Yes, exactly so You're kind of getting close to my rag slide. Yeah. Thank you very much Another question over there You cannot hear me because I'm tiny
Again thank you very much. It was an excellent talk Could you speak towards the microphone please? Yes Sorry very much. Um, I wanted to ask you about Truthful QA probably is leaked on the chat of the day and all the rest of the training data So, where are your thoughts on evil leakage
Evaluating the trainer data directly. So the problem is like truthful QA probably has been trained with Which means those those rates are no longer valid because we have seen it during the training I see I see. So the question is
Part of the problem of valuation of LLM's is that LLM's start hoovering up the evaluation data sets This remains a problem I don't know about with hallucinations, but there is a data set for evaluating AGI called arc the abstraction and reasoning corpus Francois chollet is an AI researcher at Google
He created that the way he's dealt with this is the way that Psychologists have been dealing with this for ages you keep the test set under lock and key It's not publicly available. And this is really going to be the only way we can have unbiased measures because otherwise it's just Test on train right? Yeah Great question. Thank you. Thank you
Thanks for the great talk. I wanted to ask if you have found any relationship in between The capacity of models and how they perform when they are evaluated against truthful QA data sets for example, my motivation comes from Small language models like fee from Microsoft and they also are known for being trained on really good quality
textbook quality data sets And also models from Mistral for example Mistral and things like that and they are also like not as Gigantic as GPT 3.5. For example, yeah So have you found anything like any any relationship in between models that are known for not memorizing that much and also
the kind of correlation that exists at the intersection of Memorization data quality and truthful QA. This is also a very nice question I haven't looked at this systematically, but I skipped back to a previous slide So there are a number of hallucination leaderboards as I said This is something you could have a look through for yourself and check that rather than having to do it systematically
but like this is an open secret right like The higher the quality feed into any model the better the performance and with factuality hallucinations in particular They're going to be lower if you only really expose it to correct information
Faithfulness hallucinations, it's a different question. It's more about the performance of the model, but you're absolutely correct But yeah, I haven't done it myself But if you're curious strongly encourage you to check out the hallucination leaderboards The reason why I wanted to bring this up is because I'm sorry to interrupt you, but we are we are running out of time Yeah, of course There are still a few more questions, but unfortunately, I have to cut it short here
But I'm sure that Jody is more than willing to discuss it with you. Thank you. It's all in the hallway We will back in about five minutes for the next talk see you there. Thank you