6th HLF – Laureate Lectures: Autonomous Systems – A Rigorous Architectural Characterization
1 views
Formal Metadata
Title 
6th HLF – Laureate Lectures: Autonomous Systems – A Rigorous Architectural Characterization

Title of Series  
Author 

License 
No Open Access License:
German copyright law applies. This film may be used for your own use but it may not be distributed via the internet or passed on to external parties. 
DOI  
Publisher 
Heidelberg Laureate Forum Foundation

Release Date 
2018

Language 
English

Content Metadata
Subject Area  
Abstract 
Joseph Sifakis: "Autonomous Systems – A Rigorous Architectural Characterization" The concept of autonomy is key to the IoT vision promising increasing integration of smart services and systems minimizing human intervention. This vision challenges our capability to build complex open trustworthy autonomous systems. We lack a rigorous common semantic framework for autonomous systems. There is currently a lot of confusion regarding the main characteristics of autonomous systems. In the literature, we find a profusion of poorly understood “self”prefixed terms related to autonomy such as Selfhealing, Selfoptimization, Selfprotection, Selfawareness, Selforganization etc. It is remarkable that the debate about autonomous vehicles focuses almost exclusively on AI and learning techniques while it ignores many other equally important autonomous system design issues. Autonomous systems involve agents and objects coordinated in some common environment so that their collective behavior meets a set of global goals. We propose a general computational model combining a system architecture model and an agent model. The architecture model allows expression of dynamic reconfigurable multimode coordination between components. The agent model consists of five interacting modules implementing each one a characteristic feature: perception, reflection, goal management, planning and selfadaptation. It determines a concept of autonomic complexity accounting for the specific difficulty to build autonomous systems. We emphasize that the main characteristic of autonomous systems is their ability to handle knowledge and adaptively respond to environment changes. A main conclusion is that autonomy should be associated with functionality and not with specific techniques. We conclude that autonomy is a kind of broad intelligence. Building trustworthy and optimal autonomous systems goes far beyond the AI challenge. The opinions expressed in this video do not necessarily reflect the views of the Heidelberg Laureate Forum Foundation or any other person or associated institution involved in the making and distribution of the video.

Related Material
00:00
Turing test
Internet forum
Autonomous System (Internet)
Musical ensemble
Model checking
Formal verification
System programming
01:02
Machine vision
Autonomous System (Internet)
Thermodynamic system
Service (economics)
INTEGRAL
Machine vision
Disintegration
Predictability
Internet forum
Mathematics
Smart card
Computer network
Object (grammar)
System programming
Directed set
Identity management
01:42
Dynamical system
Process (computing)
Thermodynamic system
Service (economics)
INTEGRAL
Machine vision
Interactive television
Model theory
Axiom
Parameter (computer programming)
Group action
Mereology
Sequence
Revision control
Internetworking
Order (biology)
Internet der Dinge
Units of measurement
Resultant
03:01
Standard deviation
Machine vision
Building
Model theory
Basis (linear algebra)
Computer font
Thermodynamic system
Systementwurf
Response time (technology)
Video game
Mathematics
Machine learning
Dependent and independent variables
Information security
Control system
Service (economics)
Product (category theory)
Thermodynamic system
Software developer
Fitness function
Electronic mailing list
Perturbation theory
Term (mathematics)
Functional (mathematics)
Maxima and minima
Systems integrator
Internet service provider
System programming
Information security
Electric current
Pressure
Autonomous System (Internet)
Product (category theory)
Service (economics)
Disintegration
Mobile Web
Virtual machine
Rule of inference
Telecommunication
Internetworking
Term (mathematics)
Software
Microprocessor
Computer hardware
Integrated development environment
Plugin (computing)
Systems engineering
Standard deviation
Model theory
Division (mathematics)
Evolute
Limit (category theory)
Componentbased software engineering
Software
Function (mathematics)
Computer hardware
Contrast (vision)
Autonomic computing
Computer network
Kolmogorov complexity
Communications protocol
Force
Systems engineering
08:55
Autonomous System (Internet)
Group action
Algorithm
State of matter
Graph (mathematics)
Robot
Infinity
Thermodynamic system
Tablet computer
Object (grammar)
Network topology
Green's function
Reduction of order
Control theory
Physical system
Robot
Game controller
State transition system
Thermodynamic system
Electric generator
Modul <Software>
Model theory
Planning
Automaton
Set (mathematics)
Group action
Computer
Logic synthesis
General relativity
Network topology
System programming
Finitestate machine
Object (grammar)
Game theory
Computational visualistics
11:25
Computer chess
Axiom of choice
State of matter
Model theory
Decision theory
Parameter (computer programming)
Mereology
Thermodynamic system
Medical imaging
Video game
Repository (publishing)
Automation
Control theory
Data storage device
Information
Process (computing)
Bounded variation
Descriptive statistics
System dynamics
Identity management
Enterprise architecture
Thermodynamic system
Process (computing)
Constraint (mathematics)
Reflection (mathematics)
Optimization problem
Bit
Instance (computer science)
Functional (mathematics)
Virtual machine
Category of being
Numeral (linguistics)
Fluid statics
Computer configuration
Data storage device
Repository (publishing)
Control theory
Configuration space
output
Personal area network
Condition number
Data type
Boiling point
Autonomous System (Internet)
Readonly memory
Line (geometry)
Run time (program lifecycle phase)
Robot
Channel capacity
Adaptive behavior
Virtual machine
Planning
Annihilator (ring theory)
Device driver
Goodness of fit
Whiteboard
Autonomic computing
Energy level
Configuration space
Maize
output
Subtraction
Game theory
Robot
Game controller
System dynamics
Computer chess
Information
Machine vision
Model theory
State of matter
Mathematical analysis
Planning
Independence (probability theory)
Set (mathematics)
Device driver
Partial derivative
Computational visualistics
Identity management
18:21
Autonomous System (Internet)
Enterprise architecture
Thermodynamic system
Model theory
Model theory
Virtual machine
Reflection (mathematics)
Parameter (computer programming)
Mereology
Functional (mathematics)
Thermodynamic system
Virtual machine
Degree (graph theory)
Factor analysis
Systementwurf
Degree (graph theory)
Componentbased software engineering
Autonomic computing
Energy level
Energy level
Kolmogorov complexity
Object (grammar)
Data type
Identity management
20:17
Dynamical system
Graph (mathematics)
Texture mapping
Model theory
Decision theory
Finitary relation
Semantics (computer science)
Thermodynamic system
Formal language
Mathematics
Componentbased software engineering
Type theory
Synchronization
Object (grammar)
Repository (publishing)
Vertex (graph theory)
Information
Multiplication
Position operator
Enterprise architecture
Collaborationism
State transition system
Thermodynamic system
Electric generator
Process (computing)
Mapping
Organic computing
Reflection (mathematics)
Electronic mailing list
Automaton
Instance (computer science)
Functional (mathematics)
Arithmetic mean
Repository (publishing)
Telecommunication
System programming
Configuration space
Pattern language
Data type
Autonomous System (Internet)
Trail
Enterprise architecture
Motif (narrative)
Reflection (mathematics)
Distance
Rule of inference
Declarative programming
Goodness of fit
Computer hardware
Data structure
Address space
Form (programming)
Motif (narrative)
Mobile Web
Rule of inference
System dynamics
Information
Model theory
Physical law
Interactive television
Set (mathematics)
Semantics (computer science)
Componentbased software engineering
Word
Function (mathematics)
Atomic number
Synchronization
Vertex (graph theory)
Object (grammar)
Computational visualistics
Invariant (mathematics)
Address space
24:51
Autonomous System (Internet)
State observer
Spacetime
Implementation
Divisor
Model theory
Decision theory
Multiplizität <Mathematik>
Event horizon
Thermodynamic system
Factor analysis
Medical imaging
Systementwurf
Latent heat
Autonomic computing
Control theory
Process (computing)
Information
output
Information security
Mathematical optimization
System dynamics
Thermodynamic system
Information
Decision theory
Planning
Volume (thermodynamics)
Instance (computer science)
Prediction
Functional (mathematics)
Event horizon
Hausdorff dimension
Autonomic computing
Partial derivative
Right angle
Kolmogorov complexity
Information security
Data type
Mathematical optimization
Identity management
26:53
Systementwurf
Thermodynamic system
Componentbased software engineering
Divisor
Model theory
Autonomic computing
Interactive television
Coordinate system
Complex system
Kolmogorov complexity
Object (grammar)
27:36
Spacetime
Code
Length
Quality of service
Thermodynamic system
Independence (probability theory)
Componentbased software engineering
Object (grammar)
Dependent and independent variables
Repository (publishing)
Code
Information
Physical system
Cybersex
Enterprise architecture
Thermodynamic system
Physicalism
Streaming media
Functional (mathematics)
Fluid statics
Resultant
Spacetime
Autonomous System (Internet)
Domain name
Computational visualistics
Motif (narrative)
Transformation (genetics)
Parametrische Erregung
Cybersex
Streaming media
Coprocessor
Number
Architecture
Term (mathematics)
Configuration space
Game controller
Rule of inference
System dynamics
Information
Tape drive
Interactive television
Line (geometry)
Coprocessor
Group action
Singleprecision floatingpoint format
Embedded system
Personal Assistant
Kolmogorov complexity
Internet der Dinge
Object (grammar)
Identity management
30:00
Autonomous System (Internet)
Enterprise architecture
Service (economics)
Parametrische Erregung
Ring (mathematics)
Programmable readonly memory
Mobile Web
Verteiltes System
Thermodynamic system
Number
Usability
Architecture
Web service
Mathematics
Componentbased software engineering
Cache (computing)
Readonly memory
Computer hardware
Subtraction
Sanitary sewer
Mobile Web
Robot
Enterprise architecture
System dynamics
Thermodynamic system
Spacetime
Organic computing
File format
Forcing (mathematics)
Client (computing)
Ring (mathematics)
Instance (computer science)
Computational complexity theory
Componentbased software engineering
Web service
Telecommunication
System programming
Kolmogorov complexity
Data type
Asynchronous Transfer Mode
32:01
Constraint (mathematics)
Thermodynamic system
Web 2.0
Systementwurf
Video game
Type theory
Readonly memory
Data storage device
Renewal theory
Computing platform
Mathematical optimization
Selforganization
Standard deviation
Thermodynamic system
Sicherheitskritisches System
Software developer
Food energy
Instance (computer science)
Group action
Coprocessor
Subset
Word
Error message
Order (biology)
Authorization
Data type
Mathematical optimization
33:40
Autonomous System (Internet)
Model theory
Mountain pass
Time zone
Virtual machine
Volume (thermodynamics)
Thermodynamic system
P (complexity)
Systementwurf
Term (mathematics)
Formal verification
Authorization
Process (computing)
Category of being
Mathematical optimization
Proof theory
Machine learning
Zoom lens
Standard deviation
Thermodynamic system
Process (computing)
Trail
Model theory
Limit (category theory)
Logic synthesis
Number
System programming
Formal verification
Software testing
Kolmogorov complexity
Object (grammar)
Mathematical optimization
Address space
35:53
Point (geometry)
Autonomous System (Internet)
State of matter
Dot product
Data recovery
Auto mechanic
Mathematical analysis
Event horizon
Thermodynamic system
Predictability
Number
Systementwurf
Tablet computer
Causality
System identification
Subtraction
Covering space
Thermodynamic system
Graph (mathematics)
Hazard (2005 film)
Data recovery
Direction (geometry)
Model theory
State of matter
Open source
Mathematical analysis
Mechanism design
Database normalization
Embedded system
Event horizon
Network topology
Crash (computing)
System programming
Kolmogorov complexity
Data type
Electric current
37:58
Standard deviation
Building
Context awareness
Transportation theory (mathematics)
Model theory
Decision theory
File format
Public key certificate
Thermodynamic system
Predictability
Independence (probability theory)
Systementwurf
Machine learning
Personal digital assistant
Software framework
Machine learning
Product (category theory)
Decision theory
Building
Software developer
Term (mathematics)
Functional (mathematics)
Fluid statics
System programming
Software framework
Software testing
Mathematical optimization
Electric current
Ocean current
Point (geometry)
Autonomous System (Internet)
Product (category theory)
Asynchronous Transfer Mode
Robot
Rule of inference
Goodness of fit
Latent heat
Software
Software testing
Implementation
Subtraction
Game theory
Game controller
Robot
Rule of inference
Standard deviation
Event horizon
Function (mathematics)
Autonomic computing
Kolmogorov complexity
Game theory
Mathematical optimization
Systems engineering
40:52
State observer
Musical ensemble
Decision theory
Outlier
Real number
Virtual machine
1 (number)
Control flow
Revision control
Stochastic
Casting (performing arts)
Video game
Formal verification
Elasticity (physics)
Control theory
Software testing
Extension (kinesiology)
Subtraction
Condition number
Form (programming)
Addition
Standard deviation
Thermodynamic system
Process (computing)
Information
Model theory
Basis (linear algebra)
Perturbation theory
Line (geometry)
Instance (computer science)
Entire function
Internet forum
Arithmetic mean
Mathematical optimization
Resolvent formalism
Row (database)
00:01
[Music]
00:22
it's my pleasure to introduce the second speaker of the morning sessions or George C pockets he got the Turing award in 2007 for his role in developing model checking as highly efficient verification tool and in his talk today he will talk about I think a very hot topic naming about autonomous systems and what are the challenges and perhaps some of the overseen challenges in designing them so the title of his talk is autonomous system rigorous characterization of the architecture and
01:03
technology is working sir thank you q last year I talked about system design so it will continue the tradition and this year I will focus on autonomous systems the main motivation for parties a vision that we call the Internet of
01:22
Things and this a vision that promises marvelous a lot of marvelous achievements integration of services math services smart management of resources and last year he had explained that this vision involves in fact two
01:47
parts of an even difficulty something especially is called the human Internet of Things where it's a mere improvement of the internet you have the basic model is ok you have you submit a request and you get some answers and this new version of the Internet there will be semantically more each and the most challenging part is the industrial Internet of Things where here we are seeking full integration of services and systems without human intervention so we want to fully automate the processes and humans will only tune some parameters and this is really the challenging part of the Internet of Things there is
02:43
currently a an important industrial consortium that works towards developing infrastructure and solving all the problems at raises this vision frankly speaking I have not seen any interesting result so far the problems we are facing in order to achieve this are very very
03:04
hard because autonomous systems are critical so well understood difficulties that we have infrastructures that are not trans worth enough in particular they are not secure enough so security is a very big issue and we don't know how to cope with that also another problem is the impossibility to guarantee response times if you want to have control loops you should have guarantees that guarantee times and and protocols are synchronous you cannot do that in the with the existing network infrastructure and there are other problems that come from the fact that you tighten degrade have mixed with ecology system integration very critical systems with systems that are may be vulnerable and I hope you can imagine this now these are the limitations and I would like also to emphasize that despite these limitations and the repression of markets we have new practices you know that today functionality can change in systems so hardware is a kind of commodity and then you can create the new services my addin software software can be mobile a configurable customizable and this is a standard practice now in the internet and this is practice can become very dangerous going to apply to critical systems like autonomous systems you probably know that Tesla can software may be updated every month okay so this is a very dangerous practice I have worked on political systems for decades now and there are rules that require that we don't change once a product is put in the market and this is here some criticality it is certified and the product cannot be changed so what we are doing for Tesla cars and other devices when in a chance the software is against standard practice let me remind you that when an aircraft is certified you cannot change anything not only the software but you buy all the hardware that is needed for for a life of the aircraft so you have these new practices and this is a reality and quite soon we will have selfdriving cars and I believe that this is an inevitable evolution and the problem is how a systems engineering will change to to meet these needs we are moving from small sized centralized systems that are known evolve evolve up and are automated to autonomous systems that are distributed so in systems I have designed for aircraft the environment is very predictable and now for say selfdriving cars we have dynamically changing environments and also when we design critical systems we have the ambition to guarantee correctness to provide guarantees at design time and I think that we should break whooha it's not anymore possible to provide all guarantees of correctness design so these are you division challenges our capability to build trans coffee systems and I think that for autonomy we need a technical definition of autonomy based on some semantic model not just list of terms self prefix term so many papers in the literature about autonomy system you see that an autonomous system is selfhealing self of the myself protect itself everywhere whatever okay and this technical definition should allow us also to understand how we can enhance systems autonomy and for its enhancement what are the risk we take what are the technical difficulties and I would like also to remark that the technical discussion about the autonomous vehicles focuses mainly on AI and machine learning I don't doubt that these technologies are important or key but to achieving that autonomy nonetheless there are other issues related the design of the systems and if our design has flaws no matter how intelligent is this the machine learning techniques you use okay it does not help so we should be careful with that very important question I'd like to address here is whether it is possible to develop rigorous design methodologies and associated Transcorp fitness assessment techniques so this is an outline of my
08:16
talk I will introduce the concept of autonomy then I would like to show you how it is possible to have a modelbased approach why modelbased approaches are important because you need the models of the system you design to establish guarantees if we don't have model then you cannot have guarantees and then after that I will try to explain how the Guru's system design approaches that are located by standards can be applied to autonomous systems or hot lava difficulties to do that so to explain
08:57
the concept of autonomy we'll consider five exams thermostat trained shot a chessplaying robot a soccer playing robot and the robocar so all these are systems men consist of ages just introduce some terminology an agent is a computing system and the agents control objects these are physical systems and they interact in an environment and the purpose is to meet some global goals now as a designer if I have to design an autonomous system I have to determine that the behavior of the agents pursuing individual goals so that the collective behavior meets the system global goals I think that the problem at least is clearly stated now I would like to open a parenthesis here and explain what it means technically to meet goals so this again plays the agent for a given environment no the agent knows the goal so for me the environment model is an
10:05
automaton is a transition system so you have States you have actions and in this model you distinguish between green actions during actions are controllable actions that can be controlled vary by the agent and the reductions that are uncontrollable can be played by the environment so if I have a consider this example I have goals never it's bad so bad is a state of my model eventually reach target this is another state so I can where are methods that synthesize plans the plan is a tree in the most general relation a tree with alternating controller and controller action so generating plants is like solving game problem and okay I can say a lot about that this problem of plan generation does not admit algorithmic solutions unless the environment model is simple enough it is say finite state in that case you can synthesize a maximal controller that contains all the possible plans but in general this is a non computable problem so now if I go back to the set of exams
11:28
I have considered okay you see the differences regarding the environment very simple to understand the stimuli so for the thermostat the similar very simply just numerical information temperature for vision sharply it's a bit more more complicated it's dynamic configuration of cars and state of equipment for the chest robot stud configuration of bones you can read and then main difference are the way we deal with goals so for a thermostat it's trivial to have an explicit controller a single goal for the Sadler I have design controls for for shuttles of this type you have an explicit controller so I design time you can compute a controller and then you have some online adaptation techniques for or comfort for instance many fixed goals and then as you move downwards you see that you cannot have static plan you have online planning and for chess robot you store knowledge for sake robot also some stored and generated knowledge and have dynamically changing goals so I hope that through this set of exams you see that I can have automated systems and the difference between automation and and autonomy will become more clear when I
13:01
show you what I consider to be a good agent model so an agent interacts with an environment through our sensors and actuators or the sensors sent the sensory information this is the input to the agent and the agent sends commands that are sent to our actuators may change the state of the environment so let here this architecture involves five distinct functions and independent functions and one knowledge a repository knowledge is very important for agents and for autonomy so the knowledge repository in the knowledge repository store concepts what I call concepts are models of of the objects and of the agents of the environment and their properties and this is very important for instance for this function the perceptual function the perception function takes sensory information and identifies analyzes and says I seek a car a pedestrian a traffic light so concepts that are stored in the knowledge repository then the other function is the reflection function where all of the reflection function is to build a model of the environment so I build a model of the environment why I need the model of the environment because I want to make decisions based on this model and they make decisions by combining two functions goal management and planning what means goal management I have a set of goals and the goal management means boils down into a solving an optimization problem where I under some constraints that are constraints on available resources for instance have the best quality of computation under deadline constraints or under our constraints of memory or or of image and then the planner given a goal given a model of the environment generates plans and then what is very important for adaptation is this self adaptation function that handles knowledge where we find knowledge knowledge is generated either by using machine learning techniques or analysis techniques on the model the environment model and so the self adaptation self adaptation means that I am able to chance the goal management process either by changing some parameters that count in the choice of goals in the manual management process or by changing goals so this is a first description of the objectball model life I hope this is it's clear how you can combine these five basic functions and here I have a definition of autonomy that that comes from from what I said so far so if we consider this model we can now understand the difference between autonomous and automated it's clear that the thermostat is not an autonomous system why it's not an autonomous system because you don't need any one of these functions okay you have static control a design time you don't have goal management you don't have any planning faction self adaptation etc so another important question is that in some systems you have some part of the autonomy is ensured by automated systems and dollar by humans and this is a concept that is important in autonomous systems engineering is called the autonomy level of the system so here I am giving you an example of autonomy levels for self for cars and this is the the definition of autonomy levels by the Society of Automotive Engineers so you see that you go from level zero to zero five zero five is for full automation so no drivin wheel no steering wheel at all live level four is high automation so for instance selfdriving cars with a specific equipment in say what we call automated highways level three is an interesting level because here the car has direct direct the system has the control and upon systems request the human driver must be prepared to respond this is this is a situation how to manage so you have this concept of system autonomy levels and if I use the
18:25
the architecture I have proposed the concept I have the concepts I have proposed I can characterize autonomy levels so the part that is automated by machine and the part of these functions that is assisted by humans ok so I hope that what I said is clear and in practice I would like to emphasize that to determine the autonomy level of a system we have three types of parameters one is what I call autonomic complexities I will define this concept later the other is design complexity and the third is the degree of Trance worthiness that that you are seeking so typically if you have critical autonomous systems it's not realistic today to have full autonomy you will find some compromises where humans can intervene ok so now let me consider
19:34
something more technical how I can define models for systems and what are the minimal concepts to define models for autonomous systems now there is a problem with okay I don't know why some instability of the system okay so let me explain this idea that how I can define model sky can define behavioral models of autonomous systems why I need models as a designer I should understand how objects and components and agents interact and I ensured for each agent I should
20:20
insurgents should be able to define a model of its environment so I should explain what is a good model for for for autonomous systems I've been working for a few years on this idea now we have a language to describe autonomous systems and given some the basic concepts of this language in fact we used to describe autonomous system so a system I consider that it is composed it's a set of motives motives its ownership and architectural motif it's a hardware component instances live so component is that I suppose that I have types of agents and objects so these are instances of agents and objects and I suppose that each motif is equipped with a map a map is a reference structure that is the place a very very important role for the organization of the computation and the components of the motif that may belong to many different motives have addresses that are nodes of this map through matters function so this act function defines the positions of all the components in the map and then we can talk about interaction between components and interaction means strong synchronization between components collaboration between and configuration rules configuration has to do with the beginning of dynamism of the system so you can have mobile components what it means by the components the addresses of components may charge creation deletion of components or the even dynamic change of the map okay so we have defined the language and the meaning of these models can be defined by using operational semantics is an automaton is a transition system from on which I can
22:25
apply tracking generation of plants now this is an example I'm showing you we have say selfdriving cars on a road and I have here two maps two motifs one the road trunk map which is a geographic map of which I represent I show the addresses the nodes are coordinates geographic coordinates of the road I'm a communication map and then I can write in my language interaction rules of this form for instance that for any vehicles a and a prime if their distance is small enough then they exchange their speed or I can have mobility rules that are configuration rules so I have a language to do that and the SIS I have a language to do that and now I know how to define the behavior of agents so the behavioral variants of course when I design the system I should have a clear understanding about what is the environmental model and so each engine receives as I said sensor information and perceptor identifies agents and object types and give this information to the reflection function that builds the model and of course the problem is how to build dynamically a model and keep track of dynamically of the challenges of the environment and this is this is a very very important problem I would like also to say a few words about the use of knowledge because this is a very important aspect first of all you should have a formal definition of what knowledge is and knowledge is about the definition of types of agents and objects map patterns and also some declarative knowledge laws invariance of the system methods you use so all this you have so list of items here that characterize the knowledge of the repository and of course how efficient is this interaction between either the agent and the environment depends on how you use knowledge to make in the decision process okay now I would like
24:53
to discuss this issue that is very important how much hard is to build autonomous systems and I would like to show that this takes prospects one is autonomic complex is so issues that are strictly related to autonomy so to the realization of a function same I mentioned and there is another dimension that is completely independent that is design complex so there is a third type
25:23
of complexity I do not talk about is implementation complexity this is quite standard so let me say what is autonomic complexity autonomic complexity can be characterized by these factors here complexity of perception how difficult it is to interpret the stimuli to cope with ambiguity for instance of the images you get or the vagueness of the images you get also how what is the volume of data you have to analyze to extract some relevant information also another factor is the lack of observability and controllability how much you control and you observe in your environment the more you increase of the observability and and controllability the better to make the right decisions another factor is uncertainty uncertainties means the lack of predict ability lack of predictability and this may have various reasons and of course most critical events are a predictable failures at our consetta and finally complexity of goals there are goals that are easy to manage for instance safety goals reachability goals are much harder or security or the plan optimization of resources goals are much harder to to to to to achieve and okay so this is gives you an idea about what problems what of a difficulty specific to or autonomous behavior now
26:54
this should be emphasized and understood when you design a system an autonomous
27:01
system this is a complex system has components components are agents and objects and these should be coordinated so there are two factors in design complex one is reactive complexity of components I'm going to explain this and the other is architectural complexity how much it costs to coordinate components and reactive complexities how much involved is the interaction of components with with very very so
27:36
reactive complexity does not characterize the complexity in space and time or whatever the length the number of lines of code in the component characterizes the intricacy of its interaction with the environment so you may have very simple components in the sense that their interactions very simply understand the environment transformational components you may have streaming components streaming component you have streams of values and you produce streams of value so here you care about functionality and also about the latency about latency typical exams are say encoders signal processor signal processing systems and then you have embedded components and that these components are components that continues interact with an external environment so here the goals you have to maintain our much more complicated functional goals but we have a quite what we call quality of service goals and finally and this is very important for autonomous systems in the internet of things you may have cyber physical components so I said the physical component integrates I mean it's an embedded component with its own environment in terms of objects so it's abra physical component is receives not only inform information about about i meancan receives also information about the physical quantities and and that's very important so physical said the physical components you know if you want now to represent them as agents you should consider not only the external environment but the internal environment and I should say that cyber physical systems is a domain of results that is a
29:39
very active current now how about architectural complexity ok so you can figure out a way to go from simpler architectures to more involved architectures by considering this this classification that comes here from my architectural model and here is what I
30:02
propose as a classification you may have a static architectures so as a hardware architectures where you have components and all the interconnect is known and more difficult more involved these are parametric architectures like in a ring architecture in a ring architecture you may want an architecture that works for any kind and number of components in the ring and then dynamic architecture is a parametric architecture where you can create and delete agents mobile architecture like for instance architectures needing while telecommunications you have these are dynamic architectures where you have a mobility so so they're the others function changes and finally the most complicated architectures are selforganizing architecture we have many motives many modes of organization like for instance ease of driving cars so just to show you what you get if you
31:11
consider different types of systems in this space architecture complex theory are reactive complexity so you see that complexity increases as you go up and right it's interesting also to note this distinction between services and systems so services you use for service systems form a useless formation and streaming components and force for systems embedded and cider components so the place where you have increased autonomy is there so this these are all aspects also to be considered when you design autonomous systems ok so let me finish
32:02
by saying a few words about a renewal system design what this is a concept that rigorous system design is required by standards for political systems and you know that when you build systems one concern instance covenants that the system would behave correctly despite any problems that can encounter in its life and optimization goals of course because if you give me a lot of money I can give you a very fast quality system you cannot sell it so ok and you understand that the two types of goals are antagonistic so today we have a
32:46
situation where we have political systems this is the development of this systems cost a lot besteffort systems so these are very large systems and some intermediate situations now what you'd like to emphasize is this that when you build a flight controller very liabilities 10 to the minus nine failures per hour and the single failure is not catastrophic and the reliability of some platform is the web platforms say ten to the minus four why we don't have everywhere reliability ten to the minus nine okay the answer is simple even for rockets if you want to multiply by ten the reliability of a rocket you will have to pay a thousand times money more okay so when you tie and this is a problem we are facing today in order to
33:42
have large autonomous systems that may be safety critical when we apply rigorous design techniques they are based mostly on verification verification means that you try to get guarantees you have a model of the nominal behavior of the system the goals you try to formalize the goals and you reason about that and you try to provide guarantees I don't want to give details because most most people are familiar with the verification but you have somehow to formalize the goals and for for autonomous systems complex
34:23
autonomous systems I let you contemplate this is these are goals for selfdriving cars we have a 28 you cannot read but just to show you some of these goals these are very hard to interpret in terms of very concrete objectives and to formalize for reasons I hope you hopefully understand so verification as we have applied it today in standards suffers some limitations that I list here and ok probably I don't have time how much time I have left 10 minutes yeah seven another seven okay so you have wellknown limitations and also something else that should be remarked is that even if you prove your system correct truth is a social process okay even if I say I have a system it is perfect I have proven it with my program okay this should be confirmed by an authority and that's a problem furthermore I would like to emphasize that machine learning techniques cannot be verified cannot be formally verified why it cannot be firmly verified because you know to verify you need the requirements and we don't have the requirements about how I cut is different from a door so okay so let me
35:56
skip this and once now you provide guarantees then you have so when you provide guarantees you provide guarantees for a nominal behavior of our nominal model and then you have to take into account all the assumptions you made about the some ideal behavior of the environment so the nominal model says what are the transwarp Enda nontrans wealthy states and then you should consider hazards of different types you do what we call risk analysis and all of this risk analysis is to determine amount now the nontrans graph estates nonfertile states and fatal states the idea is that by using redundancy techniques when a harmful event occurs you will not go into a fad of state a further state is a state where the trans worthless is fully compromised okay you cannot go back and there are techniques to do that and we know how to apply this techniques for small systems I have done this for our UNIX systems so you have a recovery mechanism now these techniques just to summarize are not applicable to complex autonomous systems and you can imagine the reasons one reason is that these systems are to complex we cannot pray we cannot analyze just consider an example here
37:27
this is this is an example of failure topology cover in ninety nine point four percent of the light the clashes for some some number of cases and what means the vehicle failure the failure should be analyzed because has many different causes and according depending on the course we can have recovery techniques so these systems are extremely
38:00
complicated to apply the standard techniques and I would like to close with a discussion two aspects one I
38:07
think that the important question is whether we will have standards we will have guarantees current standards cannot be applied so today selfdriving cars and the other products are self certified which sounds like a joke so the manufacturer says my car is good enough just use it ok no independent certification so the important question is whether it's possible to guarantee safety just by testing so companies like Tesla or where you more say just testing if it's tested enough I don't think that this this this is technically acceptable there is an interesting paper by mobile I which is a subsidiary of Intel that advocates modelbased techniques that's a good point and of course the chaotic the question is whether it is possible to develop rigorous design techniques this is an open question now in what I said I think I clarified the concept of autonomy I think that autonomy should be associated with functionality and not with specific techniques as many papers do that so machine learning is essential but it's not the only way to build receptors and adaptive controllers and I consider that autonomy is a kind of broad intelligence it's not just decision so there is a big difference between an autonomous vehicle at a game playing a robot so if people tell me that alphago is the Summum is the height of intelligence I love okay it's ridiculous to say this because we know how it works and I think that it's not true because in that case you have a very good situation awareness you know perfectly the rules of the game and in autonomous systems you don't know the rules of the game cannot formalize them so building transporte and optimal autonomous systems goes far beyond the current ia challenge thank you [Applause] thank you very much for this very interesting talk showing us the challenges we face in our future so are there we have time for a few questions aren't they yes thank you for the amazing talk so I would like to know what a nice but is there any standard frameworks in which we can introduce Tok
40:53
asta city in the entire verification process because most of the real life environments for example the ones in which autonomous cars learn and act are highly stochastic so how do you make a model like model the stochasticity in its fullfledged form thank you okay yes okay so in the current standards regarding guarantees are considered some that very deterministic approach now I think that we should introduce the cast elastic approaches but this still should be modelbased okay not just saying I have tested for I don't know how many millions of hours my system and it behaves correctly without being modelbased okay so when you test for instance a program we are still modelbased you have a criterion of coverage for instance so models are very important we cannot ignore models and the problem with all all these machine learning techniques AI techniques is that they refuse I mean they don't know how to introduce models and models in many minutes of the basis in any scientific approach all scientific knowledge is based on the use of models okay so we should not I mean we should understand this now this can be perfectly stochastic whatever but based on models so it seems to me the the biggest problem with selfdriving car technology is what Alko outliers which is they depend on the white lane the white lines on the road being there when the city comes along and repave z the road then the white lines are gone for a couple days in huge snowstorms the the sensors fail and it seems to me you know if this solar glare then their sensors fail and it seems to me that to make things reliable you need to somehow figure out what all the outlying conditions are and figure out how to test against them and that seems completely impossible yes what do you do is yes you tell you what is for me a realistic approach that can be applied today provided we make the investments for that realistic approaches that we equip roads with the adequate infrastructure that is not affected by perturbations you mentioned okay and this is realistic to have in highways for instance to equip a highway I don't know San Francisco Los Angeles whatever or there are some interesting experiments in Europe this is the realistic approach I don't believe that it's realistic to have a mix of selfdriving and human driven cars in cities okay this may be a catastrophic so I I think that we should be careful about that but there are technical solutions if if we have enough observability and controllability and this is reliable then I think that we can solve the problem it it seems to me that at least conceptually there's a difference between trying to design autonomous vehicles that say look at all the other vehicles and look at the pedestrians and look at the environment and make decisions about what to do compared to a system where there might be a bigger agent that's looking at all the vehicles or at least all the vehicles in some region and is trying to optimize things like traffic control in addition to safety and I'm wondering is this really a fundamental difference or is there is one just a simple extension let me summarize if I understood your question one is the situation where you have autonomous vehicles each one making its decisions and pursuing its own goals and the other is to have some monitoring system global knows the global picture and resolve conflicts it's much simpler but the problem is that you have geographically distributed vehicles I think the solution will be to have some monitoring system to make things more predictable it's a matter also of realtime ok how much realtime can be a centralized system that centralizes the information analyzes it because decisions in some cases must be taken really on the spot I remember with very short reaction times but I think that the compromise is to have a solution so it kept centralized solution on fully distributed solution and decentralized solution so the centralized means that you have some arbitration processes that are supervising this this is a realistic solution but we see we'll see there is a difference yes questions for the lunch breaks or come back after the talk in front and the rear spoiler has an Altman to make but before his announcement let's thank rows of C pocket [Applause] [Music] [Music] you