Safety of AI Systems with Executable Causal Models and Statistical Data Science
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 5 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69635 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2024 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
2
00:00
Web pageModel theoryCausalitySystem programmingStatisticsVideoconferencingConvex hullCivil engineeringInheritance (object-oriented programming)Complex (psychology)SoftwareLocal GroupPhysical systemPublic domainSoftware frameworkAreaModel theoryComputer scienceComplex (psychology)StatisticsSoftwareGroup actionField (computer science)Link (knot theory)CausalityRoboticsGoodness of fitCategory of beingForm (programming)CASE <Informatik>Logistic distributionFreewareInheritance (object-oriented programming)Civil engineeringUniform resource nameNeuroinformatikLogicArithmetic meanOpen setStatistical hypothesis testingComputer animation
05:38
SoftwareComplex (psychology)System programmingLocal GroupMultiplicationContext awarenessVirtual machineRobotInformation securityData integrityInformation privacyKolmogorov complexityDigital signalIdentity managementModel theoryConvex hullAerodynamicsEstimationMachine learningMathematical analysisDependent and independent variablesConfidence intervalVirtual machineDynamical systemEstimatorHost Identity ProtocolMathematical analysisSoftware maintenanceTerm (mathematics)Category of beingPhysical systemSoftwareObservational studyComponent-based software engineeringCASE <Informatik>Complex (psychology)Run time (program lifecycle phase)Arithmetic progressionMultiplication signExtension (kinesiology)Line (geometry)Configuration spaceAutomatic differentiationPoint cloudOpen setIdentity managementAutonomic computingFormal languageDigitizingModel theoryComplex systemContext awarenessAutonomous system (mathematics)SimulationData integrityFunction (mathematics)MereologySound effectResolvent formalismUniverse (mathematics)Adaptive behaviorDependent and independent variablesSet (mathematics)Information securityMachine learningCodeEnterprise architectureStatistical hypothesis testingOffenes KommunikationssystemCombinational logicInformation privacyQuicksortComputer hardwareComputer animationEngineering drawing
11:08
Model theoryHazard (2005 film)HierarchySystem programmingMathematical analysisMathematical optimizationObservational studyProcess (computing)Formal verificationGreatest elementArchitecturePhysical systemSoftware maintenanceResource allocationComponent-based software engineeringSystems engineeringReduction of orderLogicSound effectDirected setEvent horizonEquivalence relationLogic synthesisDirected graphNP-hardSubstitute goodFrequencyWeightoutputLocal ringComponent-based software engineeringModel theoryComplex (psychology)Physical systemSound effectMathematical analysisLink (knot theory)AlgorithmFinite-state machineDirection (geometry)Computer configurationPower (physics)Configuration spaceOptimization problemDifferent (Kate Ryan album)NumberFault-tolerant systemSoftware maintenanceNP-hardCombinational logicFehlerbaumBitCharacteristic polynomialCausalityFrequencyAsynchronous Transfer ModeVariable (mathematics)Mathematical optimizationNatural languageProcess (computing)MetaheuristikWeightSpacetimeLogicCuboidEnterprise architectureFunction (mathematics)Public key certificateRevision controlCycle (graph theory)Video gameTable (information)Parameter (computer programming)Graph (mathematics)Set (mathematics)Insertion lossPropagatorObservational studyHazard (2005 film)Pattern languageVotingView (database)CASE <Informatik>Systems engineeringHost Identity ProtocolComputer animation
17:44
Mathematical analysisPareto distributionWeightAlgorithmMathematical optimizationModel theoryHeuristicMeta elementResource allocationPhysical systemHost Identity ProtocolPersonal digital assistantComponent-based software engineeringData integrityGraphical user interfaceControl flowRaw image formatNetwork topologyLogic synthesisPhysical systemCASE <Informatik>Pattern languageMathematical analysisDifferent (Kate Ryan album)Host Identity ProtocolType theoryAreaSoftware developerProcess (computing)Resource allocationMathematical optimizationParameter (computer programming)INTEGRALComponent-based software engineeringModel theoryAlgorithmEnterprise architectureRevision controlOptimization problemFehlerbaumBlack boxCodeSurfaceMetaheuristikFunctional (mathematics)Pareto distributionAsynchronous Transfer ModeConstraint (mathematics)EvoluteSet (mathematics)Near-ringElectric generatorDistribution (mathematics)Group actionLine (geometry)Particle systemLevel (video gaming)Natural languagePublic key certificateHypermediaNumberFilm editingFood energyMaxima and minimaContext awarenessComputer animation
24:20
Graphical user interfaceComponent-based software engineeringHost Identity ProtocolNetwork topologyRaw image formatTube (container)LaceMathematical analysisSystem programmingMeta elementSimulationAerodynamicsConvex hullSensitivity analysisFehlerbaumState of matterWebsitePerformance appraisalDynamical systemLink (knot theory)Finite-state machineMathematical analysisRevision controlUniverse (mathematics)Host Identity ProtocolComputer animation
25:15
Client (computing)PredictionWireless Markup LanguageFunction (mathematics)Object (grammar)Machine learningError messageSign (mathematics)Process (computing)Event horizonPhase transitionMilitary operationCASE <Informatik>BitCentralizer and normalizerComplex (psychology)Heat transferClient (computing)AlgorithmTask (computing)Probability distributionFunctional (mathematics)Time seriesVirtual machineEvent horizonRun time (program lifecycle phase)Process (computing)Wave packetStatistical hypothesis testingDistribution (mathematics)Physical systemModel theorySign (mathematics)Object (grammar)Component-based software engineeringSet (mathematics)Validity (statistics)Inductive reasoningVolume (thermodynamics)Computer animation
29:00
Link (knot theory)Empirical distribution functionoutputConfidence intervalStatisticsPhase transitionMachine learningPredictionDistanceComputer-generated imageryProcess (computing)Virtual machineStandard deviationQuantificationEmpirical distribution functionMedical imagingAlgorithmRun time (program lifecycle phase)MeasurementWave packetOperator (mathematics)Universe (mathematics)Statistical hypothesis testingVirtual machineMachine learningConfidence intervalGraph coloringoutputQuantificationInformation technology consultingAreaStudent's t-testStandard deviationDifferent (Kate Ryan album)DistanceDistribution (mathematics)Cartesian coordinate systemWordKey (cryptography)Computer animation
32:45
Social classData typeAttribute grammaroutputDecision theoryEmpirical distribution functionStatisticsModel theorySequenceSoftware maintenanceDependent and independent variablesComponent-based software engineeringLoop (music)Open sourcePairwise comparisonFunction (mathematics)DistanceMoment (mathematics)StatisticsPoint (geometry)Set (mathematics)ConsistencyDistribution (mathematics)Wave packetSocial classLink (knot theory)Virtual machineSpacetimeDecision theoryMeasurementInterpreter (computing)Computer-assisted translationModel theorySoftware maintenanceRevision controlData dictionaryCASE <Informatik>Observational studyFormal languageGraph (mathematics)Identical particlesDependent and independent variablesSimilarity (geometry)AreaArithmetic progressionLimit (category theory)Local ringInstance (computer science)Electric generatorSeries (mathematics)Group actionComputer animation
37:25
Open setDimensional analysisOperator (mathematics)InfinityInformation securityHierarchyControl flowModel theorySystem programmingComponent-based software engineeringBayesian networkMarkov chainState of matterDigital signalIdentity managementQuery languageGroup actionPublic key certificateMobile appSound effectEvent horizonParameter (computer programming)Physical systemMultiplicationRobotEvolutionarily stable strategyAreaPersonal digital assistantKolmogorov complexityMathematical analysisWrapper (data mining)Machine learningFormal grammarSource codeComputational intelligenceSoftware frameworkHost Identity ProtocolObservational studySoftwareConfidence intervalStatisticsFormal languageLatent heatTime domainRevision controlRepository (publishing)ComputerPattern recognitionSign (mathematics)Software maintenancePhysicsLocal ringElectric currentLocal GroupModel theoryPublic key certificateRun time (program lifecycle phase)Physical systemBoss CorporationMereologyFactory (trading post)Multiplication signOpen setConfiguration spaceOffenes KommunikationssystemEvent horizonOperator (mathematics)AlgorithmHierarchyState of matterResultantProjective planeNetwork topologyCondition numberWeightSensitivity analysisLatent heat1 (number)CausalityBranch (computer science)Arithmetic meanINTEGRALBuffer solutionDifferential (mechanical device)Enterprise architectureOrder (biology)Communications protocolNumberMeasurementWordElectric generatorCartesian coordinate systemLevel (video gaming)Problemorientierte ProgrammierspracheFehlerbaumFinite-state machineTheory of relativityVector potentialCalculationBitGodSound effectAvatar (2009 film)CuboidAutomationSlide ruleReflection (mathematics)Shape (magazine)Component-based software engineeringVirtual machineRevision controlTelecommunicationSoftware developerSymbol tableAutonomic computingInformation securityEscape characterSoftware maintenanceDigitizingSoftware frameworkIdentity managementGame controllerCollaborationismComputer animation
46:22
Computer animation
Transcript: English(auto-generated)
00:09
I'm Giannis Papadopoulos, I'm a professor of computer science at the University of Hull and I lead a group on dependable intelligence systems and today I will talk about safety of AI systems
00:20
with executable causal models and statistical data science. Mostly apart from scoping the problem I will talk about the work of my group in this area and the work as you will see it has two aspects so on one hand we develop models by analyzing systems and in some incarnations
00:45
of the research these models are executable at runtime and on the other hand we use statistical techniques to deal with some problems of AI which cannot be solved by models. So let me take you back to 1980 when I start discussing this when Saint John writes the book of the apocalypse or
01:06
book of revelation in a cave in the beautiful island of Patmos and I mention this because many CAI is apocalyptic. Now apocalypse means revelation
01:22
strictly speaking but it has developed these connotations of dystopian meaning precisely because of this book of Saint John and why because in the book the opening of the four seals of a divine scroll releases the four horsemen of the apocalypse bringing conquest civil war
01:41
famine and death upon earth. So enjoy the picture it's a very nice place to visit in the summer and after this small touristic kind of intervention I'm going to move into the talk. Today AI is seen by many as an existential risk and this is why I made this apocalyptic
02:05
reference. So Nick Bostrom I don't know if you know him he used to be until recently a professor in Oxford has developed a very nice metaphor about this he has picked through the history of
02:21
human creativity as one of extracting balls these are the technologies that we developed from a giant urn a giant pot we take these balls out and we do this blindly okay so we don't know what will come out and we cannot put the ball back in so we have taken
02:41
out in our history the fire the wheel democracy logic in antiquity these are also technologies the gear the mechanical clock the free market economy the public limited company the car the aircraft the mobile phone the computer most of these technologies of course we know they are
03:02
very useful however one of them could be malignant one of these balls and could devastate humanity and the problem as I said is that we cannot put the ball back in once it is out so for example we took out nuclear power and we were lucky because
03:23
we didn't know at the time whether in the future one could bake a nuclear reaction into a microwave oven because if that was possible we wouldn't exist so the question today is could AI be this malignant ball that will destroy humanity can we have some form of super
03:42
intelligence that will take over the planet you know rogue AI or robots and marginalized humanity some people say in 50 years some people say no it's it's still far away however whatever the case today there is a grand technology challenge towards dependable AI
04:01
and intelligence systems and the implications for industry and society are enormous we know that because we have soon autonomous cars coming on the street we will have swarms of drones doing logistics and delivering goods in your houses and we will have robots in hospitals taking care of patients or robots in in homes taking care of all people
04:25
and we need to make sure that these things are safe so which brings me to the work of my group the dependable intelligence systems group at the University of Hull we do world-class research on complex software and systems including intelligence systems targeting dependability as
04:41
a property including dependability of AI for many years now and we have pioneered some novel techniques for which we are quite well known and but not only methods we also develop state-of-the-art tools that have both academic and commercial impact
05:01
so some of the tools that we develop are commercial the work that we do I think it's at least in in some of the fields of safety or model-based safety assessment is very well known and in the UK our research impact is officially ranked as internationally excellent every seven years we do exercises called research excellence framework
05:23
in the UK nationwide and the our submissions are in the public domain so if you clicked I'm not going to do it now on one of those links you can find it by googling I guess it's you will see our submission and you will see the impact of our work in this area
05:45
so let me briefly talk about the context and motivation of how it says before I start to sketch what we do and I will give you an overview of all the techniques that we develop here so dependability we look at dependability and this is an umbrella term that includes many
06:03
properties which are of interest like safety reliability availability maintainability data integrity security privacy we are concerned about all these properties when we develop systems of course and we know that there are increasing concerns about the dependability of new systems which brings me to what I call the four horsemen of the apocalypse the technological apocalypse
06:26
so complexity intelligence autonomy and the openness of systems of systems like autonomous cars for example now you can see that the four challenges form the convenient acronym ciao
06:40
which means high in italian and let me briefly say something about this so complexity systems become very complex you have complex architectures partly on the ads partly on the cloud you have lots of software so you have modern cars can have 100 million lines of code how do you deal with this with the safety of these systems intelligence it's even
07:07
a bigger challenge why because intelligence systems are not designed are not are not programmed the behavior of these systems is learned by data and this is a significant departure from
07:20
what we were doing in the past when we were analyzing things for safety we had a design we had some code to analyze we don't now with intelligent systems and then you have autonomous systems which have no users and when autonomy fails because it can fail there's no one there to take over and you have this open systems of systems like autonomous cars or multi-robot systems
07:46
which are highly unpredictable you cannot analyze every possible configuration of the systems a priori in design time so to some extent safety must be resolved also at runtime
08:00
so this challenge perhaps you know most people would agree today that it is very significant and it affects many emerging systems including multi-robot systems cooperative swarms machine learning and transport health manufacturing etc it is an important problem so what do we do we develop a few techniques here to try and investigate and partly address some of these
08:25
challenges it's all work in progress of course i'm not going to claim today that we solve all you know these problems i don't think anyone has the solution but i think there's some interesting work here so for complexity we develop um hip hops and dimodia which are model
08:40
based methods and tools for automating dependability analysis and design of systems and i will say something about this today for intelligence we develop a set of statistical techniques um safe ml smile safe llm safe ml provides a dynamic estimation of confidence on the accuracy of machine learning smile is looking at explainability of machine learning
09:06
and safe ml is looking at detection of hallucination and filtering of unsafe responses of llm large language models and finally for autonomy and openness we develop something called
09:21
eddi's executable digital dependability identities these are executable model-based safety monitors for runtime safety assurance and adaptation of systems of systems so for things like autonomous cars and swarms of drones and multi-robot systems to try and resolve safety at runtime so i will say some things about these topics here to give you the scope of the work
09:46
so let me start with the challenge of intelligence of complexity and what you see here is a steel by wire system that from volvo this is i once analyzed that in a case study that i done together with volvo so and using some tools that we
10:05
develop in the university so one of the problems that an analyst will be called to address here is what if a component fault develops here in one part of the system what will be the effect on the outputs on the steering actuator will i lose steering will i have incorrect steering
10:21
and faults can be all sorts here you can have a hardware fault a software fault you can have a security breach you can have incorrect reasoning of a machine learning component if you have one so um it's okay if you have a few faults however what if you have a thousand faults here
10:41
and you want to examine the effect of combinations of two you have 499 500 combinations of two okay so around 500 000 combinations of two faults that you need to analyze it's impossible to do that exhaustively manually or even with simulation especially if you have a more complex
11:03
system and you need some automation here some effective automation so for many years we have been developing a model-based method and tools that simplify this kind of dependability analysis causal analysis cause and effect and the optimization of dependable systems by partly
11:24
automating the process and the method is known as hierarchical performed hazard origin propagation studies or hip hooks and i will say a few things about the method so over the years it has developed quite a bit and it supports now
11:41
um a mode of design that spans the life cycle of systems engineering so we support a safety driven mode of design in which safety requirements that we capture through a process of risk analysis early in the design can be allocated to subsystems and components during the refinement of an architecture we support bottom-up safety and reliability analysis
12:04
of detailed designs to verify that these requirements have been met and when the requirements have not been met we can initiate automatic optimization using metaheuristics of system architectures and maintenance with respect to safety reliability cost etc and we recently we
12:23
have developed some methods to produce certification artifacts like safety cases and to make these models that we create here executable for operational safety monitoring so let me briefly say a few things about the method in hippos the analysis of a system
12:44
are preferred on a model of the system which can be hierarchical here to manage complexity so you have system subsystems components in recent versions we have introduced here state machines as well to describe behavior and the first thing you do is you specify some local failure logic for the components in this model so here you specify how output failures are caused by
13:05
internal failures and and input failures and once you have put this local failure logic you can execute algorithms to automatically construct a set of fault trees for the system which show how component failures logically combine and propagate to cause system failures so here you have the
13:25
loss of steering the incorrect steering and here you have component failures and you can establish the the causal link here using this model so you can do logical and probabilistic analysis of this graph here and you can establish the reliability and safety of your system
13:44
and you can also take this graph of causes and effects and you can simplify it you can remove the intermediate logic and you can reduce it into a failure modes and effects analysis table which shows direct relationships between components and system failures so this table for each
14:03
component a b c and for each failure mode of each component it will give you if there are any effects on the system and one or more and whether these effects are by the failure mode itself or in conjunction with other failure modes so it will give you a simplified view of how the
14:25
can fail now we developed a tool to do all this work and one question that we asked is what if a design is found by the tool not safe enough how can we improve it now design you can scrap
14:42
a design but there are often simpler solutions that you can take so you can substitute a component a sensor let's say with a more expensive a more reliable sensor you can substitute a system architecture with one where that is better designed maybe it has more fault tolerance you can replicate components so when one fails another one takes over you can introduce voters
15:06
and you know patterns of fault tolerance you can increase the frequency of maintenance but here the problem is that you can have too many options and the question is design options and
15:26
um and here we are faced with a set of hard design optimization problems that can only be addressed effectively with automation to give you an example imagine that you have a system with a hundred components and for each component you have two options let's say a sensor can be from
15:43
Bose or from Siemens and it has different failure and cost characteristics now for a hundred components with two option seats you have two to the power of a hundred possible combinations of different systems configurations that you can make that's a very large number two to the
16:03
power of a hundred it's ten to the power of thirty something okay so which solution is optimal when you have this kind of option so we have developed some algorithms in our tool to do this kind of certs and design optimization to find these optimal solutions so we start again
16:24
from a model and this model now is variable it's not fixed that's the difference now so these components can have options so a sensor can be from Bose or from Siemens and a large box like it could be this could be an architecture for the braking or the steering subsystem
16:43
implemented with variant a or variant b by another designer you can put cost weight failure data here maintenance options perhaps and then you will give it to hip-hop which will apply an evolutionary design optimization process so effectively what you do here is you create
17:02
a design space to be explored and hip-hop will explore it using genetic algorithms or other metaheuristics so what it will do here keep hopes is resolve the variability of this model so create fixed variants of the candidate design it will create a population initially
17:25
arbitrarily resolving the viability and then it will evaluate its individual its potential car in the population for the parameters of the optimization which could be cost weight and
17:41
dependability and crucially we can evaluate dependability here because we can automatically reconstruct fault trees for each variant so once we do the analysis we then rank the individuals of the population we create Pareto fronts or Pareto surfaces or Pareto
18:03
best designs we breed between them and we create a new population of candidate designs which we subject to the same optimization process and after many generations we can arrive at a set of optimal or near optimal models which give improved or optimal trade-offs between
18:24
the parameters of the optimization we know that this work works we have applied it to benchmark examples and we have applied it to we have incorporated actually to in a commercial tool so it is possible to do this kind of constraint optimization and evolution using
18:43
these techniques so other things that hip-hop can do is allocation of requirements so you can do cost-optimal automatic allocation of system requirements which are captured by a process of risk analysis examining the system as a black box so this could be a car and these
19:05
functions could be braking or steering and you can do a risk analysis and find the failure modes and what kind of requirements are imposing if they are critical on those functions and you start from these safety requirements for the system and as you develop architecture
19:22
of subsystems you can allocate them to the subsystems so these architectures then delivers the safety requirements and similarly you can allocate the subsystem requirements to component safety integrity requirements so these integrity requirements of the subsystems are allocated to the
19:41
components and all this is done in a way that when you put this system in place the safety requirements at the top are being met now this is an optimization problem we solve it by using model-based safety analysis and using AI metaheuristics like genetic algorithms
20:02
and on one hand you need to satisfy safety here but also you need to do it cost-optimally why cost-optimally cost-optimality is important here you don't want to be allocating the highest safety requirements to components and subsystems that include thousands of lines of
20:23
code because it's very expensive to develop one line of code at safety integrity level for the highest it could cost you thousands of euros to do that but line of code so we have developed here technology that can solve this do this allocation respecting
20:42
and taking into account the assumptions that you have made about how each component either propagates or mitigates failures as well because this defines how the requirements are allocated whether they can be split what component will inherit the highest requirement
21:01
so we have in the context of this work developed a metaheuristic which was actually quite fun and received a lot of publicity and a new metaheuristic which tries to model the social intelligence of penguins it's effectively a variant of particle swarm optimization this one
21:21
which captures some of the behavior of a penguin colony and now penguins are quite good at fishing and the reason for that is that they have social intelligence they cooperate so they work in groups and they work they collaborate within the group but they also collaborate across groups by exchanging vocalizations so if you let the group of
21:45
penguins out in the sea you will see that the distribution of penguins will gradually match the distribution of fish in the sea which makes the colony very effective in fishing so they can get the maximum cuts by spending the least amount of energy which enables the colony
22:04
to grow so we have captured this behavior and in an algorithm and we have used it to solve this requirement allocation problem which captured the imagination of the media here and we have a number of articles that have been published here in in this area and the most recent
22:24
development here in the area of this of this model-based safety analysis and hip hooks is a tool called Andromeda it's an experimental tool that we have developed that can create safety case for certification by exploiting the models and analysis that we create
22:41
automatically with hip hooks so we rationalize the problem of certification that's what we try to do here so the safety cases all of them have the same pattern of reasoning and effectively argue that the system can meet its safety requirements as a whole the system let's say a car or an aircraft which are derived through an exhaustive risk analysis because
23:04
the subsystems meet the correctly allocated integrity requirements and we can prove that the allocation is correct from the algorithms that we have applied and that all the model
23:21
assumptions that underpin this allocation are met because when you do hip hopes you put assumptions here you may assume for example but that for this component to fail both these must fail independently and you have to provide evidence that this is the case and this is true that the subsystems meet their integrity requirements because the components meet their integrity
23:46
requirements and all the model assumptions are met at this level as well and there is evidence in the for the above in the algorithms the fault trees the fmeas and everything that we create with hip hooks and all this is formed into a safety case argument
24:04
which looks similar for different types of systems so it's a kind of pattern so now what is important here is that i think for many people is that the this tool is also commercial so we have developed a commercial version that we commercialize and there are various
24:22
incarnations of this tool so we have hip hopes by the university of hal these are all links you can click and you can find these tools in the in the you know in the websites the corresponding websites and you can download an evaluation version from here you can play with it if you like there is a version that we commercialize with a company in germany esi
24:48
the company is based in dresden and it commercializes hip-hop in conjunction with their own tool there is a version with a company in finland and we develop a new tool called dimodia that enables dynamic dependability analysis using state machines as well as
25:06
architectural models to create state sensitive fault trees it's some new work that we do it's not commercial this thing yet so the important thing about this aspect of the work that we do is that we have achieved technology transfer that has global reach so our clients include
25:24
on them to yota let's say if we if we start from the east and huawei in in china to honeywell in the u.s honeywell aerospace and from volvo in the north then through continental and fiat in central europe to emperor in brazil so we do we do have many clients around around
25:45
the globe now i talked a little bit about how we deal with the challenge of complexity now if i return to tiao here and take the challenge of intelligence which i think it's an even more important challenge today and as i said the problem with intelligence is intelligent components
26:03
is that we don't have design artifacts to analyze you see in hip-hop i was analyzing models of systems and here we don't have design artifacts so how do we deal with this problem machine learning classifiers for example are often trained to detect objects like traffic
26:24
signs or pedestrians in the street or events process failures for example from a time series of data okay and they learn how to do this task by being exposed to large volumes of data of usually labeled data and these are used to train the classifier to do a specific thing
26:49
but the problem here is that these systems learn by induction learn by experience so what they learn is a function of the data that they have seen they have and they have not
27:04
seen the whole data that defines the task so when they operate outside what they have seen they can fail to reason correctly and that is a problem and that is inevitable because induction
27:24
epistemologically is in perfect way of learning and it is by definition prone to failure so there's a fundamental problem here what do we do so you can train a machine learning algorithm and you can validate when you train it its accuracy okay you can use for example a labeled
27:47
data set 80 for training 20 for validation you can calculate an accuracy but this accuracy will only be a function of the training data set and may drift for data outside this set so
28:01
you can see that here imagine that you train the algorithm using data that has this distribution and then you test you validate using data that has this distribution which is similar to that and you get a 93 accuracy on the task and then you release the algorithm to work on unseen data
28:30
and if this unseen data the runtime data has this kind of statistical distribution let's say then which is very similar to the training data you get good accuracy you can even get higher
28:41
accuracy but what if the data the unseen data the runtime data has this kind of statistical distribution which is different from that then you expect this accuracy to fall and it will fall and that is a problem now what we have been doing recently the last few years in the
29:04
university of is to develop something called safe ml safety of machine learning this is a method that uses statistical techniques specifically empirical cumulative distribution function based distance measures to measure precisely the drift in between training
29:27
data and runtime data input data linked to its reasoning outcome of a machine learning algorithm for example to its classification it does so we measure the drift and from that we can establish a measure of confidence
29:47
in the accuracy of the machine learning reasoning so the higher the drift the lower the confidence that we return in the accuracy of the machine learning algorithm
30:00
so that is the key idea behind safe ml and let me give you an example of application here imagine that you have trained a machine learning algorithm to detect lung cancer to do lung cancer diagnosis in other words and you have done that by exposing the algorithm
30:27
to common symptoms of the disease so you have trained it with lots of images that have common symptoms of the disease and then during runtime during operation the algorithm
30:40
is being given images that have rare disease symptoms rare now in those circumstances the algorithm operates outside its distribution and therefore it could be misled it could be it could be wrong so for example it can fail to diagnose this cancer and that of
31:08
know that they have cancer but if you put safe ml as a filter on top of the algorithm safe ml should be able to detect the drift between these and these images the statistical
31:23
difference between these and these images and it will know that it has not been trained the algorithm on these kind of images and therefore it will lower confidence on the reasoning it will effectively tell you do not trust the outcome of the machine learning we cannot trust
31:46
that this is not cancer or what this is cancer and give the image to a consultant to comment on it so this is what it will do now we have published a couple of few papers on this
32:03
and this is cited in the new german industry standard for machine learning uncertainty quantification that was published i think earlier this year so that's quite important for us we also have done we've got some some awards and some distinctions about this work i
32:22
think various people who work in this um so kunico pakston for example is one of our phd students who have received an allen touring institute award for her work on safe cancer detection she's looking at a particular aspect of fairness that has to do with the skin color but she's using
32:42
statistical techniques to do that and safe ml and another thing that we develop in this area using similar techniques is called smile statistical model agnostic interpretability with local explanations and smile looks at explainability so smile will try to explain why a machine
33:06
learning classifier reads the decision it did so for example why it has classified something into class one and not class x why it has classified something as a pedestrian as opposed to a cat or empty space etc and it will do that by using statistical distance measures
33:28
and measuring effectively the statistical distance between features on the examined artifact
33:42
okay and compare this sorry statistical distance between the features of the examined artifacts compared to the corresponding features of the data in the day in the in the training data set okay so we will measure the distance between the feature of the examined artifact and the
34:07
distribution of features in the training data set and by doing that for each feature we can establish a measure of how much similarity and how much dissimilarity exists and therefore
34:21
why something is classed as one because there's more similarity and not as x because there's more dissimilarity this is the idea finally we develop something called safe large language models which apply statistical techniques to do things like hallucination detection
34:43
and distinguish between safe and unsafe responses of a large language model and what we have done here we have done a case study on wind turbine maintenance together with electricity the france so we developed we took an open source llm yama 2 and we
35:06
trained it to do specifically engineering maintenance using a knowledge graph that shows relationships between alarms so series of alarms and maintenance actions so in this way
35:22
we have created a specialized version of this and we created a conversational agent that can work with maintenance personnel to try and generate engineering maintenance advice in the output and now what we have done here to make this safer is we take this output and the first
35:44
thing we try to do is hallucination detection how do we do that we generate multiple instances of the output for the same prompt given and then we do pairwise comparisons
36:00
here of the distance between the outputs of course the outputs have been encoded so that you can measure in vectors so that you can measure the distance and by doing that we are able to detect inconsistencies between the outputs on the same prompt and from those inconsistencies we
36:23
can suspect hallucination and the other thing that we do is we take the output and we compare it to a dictionary of unsafe outputs and if the output has similarity to this dictionary of unsafe outputs then we can also suspect the output as being unsafe and we can provide the
36:43
output to the maintenance personnel and ask them if it is indeed unsafe so we have human in the loop and if it is we extend the dictionary now this so this is our most recent work in this area there is a paper on that I will give you some links to the papers now what is important I think to say at this point is these things are work in progress
37:06
you know I'm not going to claim here that we have solved the problem of intelligence no we haven't okay we are doing research and we are doing some interesting things and these things have their limitations but we try to improve a situation which at the moment
37:22
is is very problematic in this area of safety of intelligence so let me briefly say a few things about the challenge of autonomy and openness so autonomy there are no operators when autonomy fails to take over that's a problem openness you have infinite configuration in a system
37:42
of a multi-robot system in a factory or autonomous cars connected to traffic lights etc and infrastructure you cannot analyze all those configurations of this system at design time it's impossible you can have emergent behaviors and uncertainties you can have higher security threats in systems of systems and these systems are heterarchies of equals there's no hierarchy of
38:05
control so safety is a big problem because there's no boss in a system of autonomous cars that can control safety so the implication is at least part of dependability assessment and certification and assurance need to be moved at runtime so we need to move from the old model
38:26
of doing safety at design time now we have been developing a concept of executable digital dependability identities which are modular composable and executable specifications of dependability of for components and systems in open systems of systems so these are models
38:43
which are executable and these models are derived from safety assessments the ones that we work with so they can be biasing nets fault trees markov models hierarchical state machine state sensitive for trees conditional safety certificates so we have been experimenting with various models and we developed this in a european project called sesame it's just concluded actually
39:04
with the results and we have developed something called the open dependability exchange in which we can encode these kind of models and then make them executable because we have algorithms to make them executable so let me give you an idea of how this thing works the eddi so here you can see the architecture of an eddi so there is an event monitor
39:26
which monitors the system and try to detect significant events faults for example and we developed a little language here that can do long-term temporal monitoring to detect events
39:41
it can instantiate for example in runtime circular buffers to do the long-term monitoring it can do differentiation integration etc various operators to detect significant events some of them are unambiguous but some events may require further diagnosis to find what's the underlying cause and we can pass this to a diagnostic engine that would use selectively traverse the branches
40:05
of a fault tree or use other means to do the diagnosis of causes and then the events that we detected and diagnosed causes once they are unambiguous they are passed to high level reasoning applications that execute these high level models markov biasing nets hierarchical state
40:23
machines and this execution will be able to produce effects possible effects to project risk in the future to generate alarms diagnostics effects of failure guidance of corrective measures perhaps now the important thing about this system here is that it is not isolated to one
40:48
system it works on a system of systems it communicates in other words using protocols with other eddi's of other systems in the in the system of systems so for example we can ask guarantees from other systems to meet demands in some of these models to make sure that we
41:06
are safe so you can see that here you can see a system of systems a multi-robot system so you have a number of systems and you can see the eddi's of these systems they are sitting
41:21
on each system they are executed the models and then they communicate between them in order to resolve safety so they can exchange guarantees and demands and data in order to guarantee safety and we have done experiments with security in this european project as well so we try to inject failures and see how it responds the thing because the the failures
41:44
are modeled in there now the most recent development on this and i'm close to the to the end i took a little bit took a lot of time and we we are currently the most recent development on this is we develop a version of the eddi as a biasing net the probabilistic model to calculate
42:05
potential risk etc and to diagnose and prognose and in a collaborative fashion but which exploits belief values returned by safe ml if you remember saying with safe ml we can give a kind of probability that a machine learning algorithm is wrong has failed
42:25
so in that way we integrate the failure of intelligent components into a probabilistic framework that includes other failures in a system of systems so that's very recent work we haven't
42:41
even published the paper but this is what we do now and the applications are from factory automation to um drones that do maintenance on electrical infrastructure and drones that do disinfection on hospitals in this european project that i mentioned so there's other related work of course we are not the only ones to develop stuff to address these problems so i put
43:03
some names here i'm not going to discuss them just just for your interest and there's much more of course that i haven't mentioned here now i would like to close with something with a couple of reflections in all this so here i'm sure you have seen sagrada familia somewhere
43:23
if not in person a beautiful cathedral designed by a genius architect antonio by ud using intelligent design and this thing surprisingly looking in shape similar to the sagrada familia is a termite mount so this was built by millions of termite ants
43:43
without any design and i think that one possible future of technology is that we will move from intelligent design to things that learn to things that they evolve to things that they are loosely collaborating etc like the ants and we don't want to move
44:03
from something like that to something like that okay so the move is is both exciting but it is also problematic and which brings me back to my initial to my initial slides when i said is ai a risk an existential threat so let me close by looking at ai as pandora's box
44:25
perhaps you know the ancient greek myth of pandora pandora opens a box gifted to her by the gods and when she does so all the evils that we have in the world escape including war etc illness but she's quick enough to close the box in time to keep hope inside so we open the
44:50
box of ai and i think what we need to do we need i think we cannot close it we cannot close sorry we cannot put ai back into the box it's impossible we have to go through the portal
45:05
of ai somehow and we have to go without destroying the planet and humanity etc so we need much more work and investment in the area of safe ai to try and achieve that so i put here a few papers so the latest there are lots of papers on hip-hop and model-based
45:26
safety assessment which might be useful even when you just you know a system will not only have ai it will have ai as part of something bigger so you will always need also a kind of neuro symbolic model to address safety so the most recent paper which was just published is this on hip-hop
45:49
i put a paper on safe ml a paper on smile and a paper on safe llm again all these are quite recent papers and i put a paper on eddi's so you're welcome to have a look at these papers
46:04
if you haven't already and to come in contact with us and and discuss all this stuff and and we are we would very much welcome questions and comments and postdoc communications and possibilities of collaborations etc all these things are very welcome