A robotic platform for natural and effective human-robot interaction

Video thumbnail (Frame 0) Video thumbnail (Frame 5308) Video thumbnail (Frame 10041) Video thumbnail (Frame 11198) Video thumbnail (Frame 12269) Video thumbnail (Frame 13218) Video thumbnail (Frame 22974) Video thumbnail (Frame 24566) Video thumbnail (Frame 26252) Video thumbnail (Frame 28313) Video thumbnail (Frame 37670) Video thumbnail (Frame 39042) Video thumbnail (Frame 40328) Video thumbnail (Frame 41269) Video thumbnail (Frame 42237) Video thumbnail (Frame 54559) Video thumbnail (Frame 57197)
Video in TIB AV-Portal: A robotic platform for natural and effective human-robot interaction

Formal Metadata

A robotic platform for natural and effective human-robot interaction
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
A robotic platform for natural and effective human-robot interaction [EuroPython 2017 - Talk - 2017-07-14 - Anfiteatro 1] [Rimini, Italy] This talk deals with the usage of artificial intelligent techniques in humanoid robotics. The focus is on human–robot interaction with the goal of building a robotic platform which embodiments are able to interact in natural and effective way with humans through speech, gestures, and facial expressions. The system is fully implemented in Python and based on the Robotic Operating System (ROS). The talk will describe the hardware and software configuration of our currently NAO based humanoid platform. The strategy has been to use available high level Python libraries for spoken language processing, sentiment analysis, vision, interfacing with Artificial Intelligence applications in order to provide current edge technologies performances. The overall system architecture is based on finite state machines nodes interacting via the ROS communication layer. The main fields of applications that the platform is targeting are: - Entertainment - Education - Field robotics - Home and companion robotics - Hospitality - Robot Assisted Therapy (RAT) We will present the latest status of the platform together with a NAO based demo
Computer program Intel Code Mehrprozessorsystem Formal language Malware Component-based software engineering Process (computing) Endliche Modelltheorie Website Library (computing) Area Scripting language Computer font Flow separation Formal language Virtual machine Field (agriculture) Software testing Modul <Datentyp> Regular expression Data structure Electric current Mathematical analysis Infinity Computer Shareware Machine vision Architecture Machine vision Field (agriculture) Robotics Software Energy level Software testing Regular expression Data structure Implementation Computing platform Computer architecture Robot Artificial neural network Interactive television Mathematical analysis Code Field (computer science) Cartesian coordinate system Component-based software engineering Software Robotics Computing platform Speech synthesis Finite-state machine Videoconferencing Library (computing)
Pattern recognition Computer file Chatterbot Virtual machine Translation (relic) Drop (liquid) Infinity Formal language Architecture Telecommunication Robotics Natural number Computer configuration Interpreter (computing) Maschinelle Übersetzung Software Videoconferencing Energy level Implementation Library (computing) Computing platform Computer architecture Point cloud Robot Programming language Decision theory Moment (mathematics) Projective plane Interactive television Objekterkennung Computer network Translation (relic) Evolute Formal language Software Telecommunication Phase transition Computing platform Configuration space Finite-state machine
Scripting language Robotics Finite-state machine Right angle Process (computing) Evolutionarily stable strategy Virtual machine
Mathematics State of matter Robotics Multiplication sign Combinational logic Finite-state machine Website Mass Finite-state machine
Game controller Virtual machine Mass Control flow
Webcam Digital media State of matter Code Modal logic Multiplication sign Public domain Befehlsprozessor Repository (publishing) Videoconferencing God Point cloud Scripting language Plotter Arm Open source Shared memory Connected space Digital signal processing Skeleton (computer programming) Latent heat Internetworking Configuration space Convex hull PRINCE2 Task (computing) Resultant Reduction of order Laptop Addition Computer file Open source Computer Shareware Architecture Latent heat Skeleton (computer programming) Robotics Internetworking Computer hardware Directed set Computing platform Task (computing) Computer architecture Robot Projective plane Code Mathematics Personal digital assistant Computer hardware Robotics Computing platform Point cloud Object (grammar)
Finite-state machine Spacetime
Area Wiki Crash (computing) Moment (mathematics) File viewer Term (mathematics) Computer
Pattern recognition Word Multiplication sign Point cloud Speech synthesis
Laptop State of matter Multiplication sign 1 (number) Water vapor Mereology Frequency Component-based software engineering Energy level Pattern recognition Web page Projective plane Objekterkennung Sound effect Term (mathematics) Connected space Category of being Process (computing) Software Order (biology) Charge carrier Video game Website Regular expression Spacetime
Standard deviation Multiplication sign Projective plane Virtual machine Normal (geometry) Species
Joystick Normal (geometry) Maxima and minima
Web page Computer file Multiplication sign Image resolution Sheaf (mathematics) Virtual machine Disk read-and-write head Theory Formal language Mathematics Natural number Bus (computing) Endliche Modelltheorie Noise (electronics) Pattern recognition Focus (optics) Digitizing Moment (mathematics) Volume (thermodynamics) Term (mathematics) Proof theory Word Software Speech synthesis Configuration space Right angle Game theory Reading (process)
CAN bus Data management Virtual machine Term (mathematics)
Type theory Key (cryptography) Inheritance (object-oriented programming) Robotics Weight Workstation <Musikinstrument> Set (mathematics) Right angle Object (grammar) Quicksort Instance (computer science)
so high everyone my name is area when young and today I will show you how to create an architecture to create to woman a pup platform to create a matter of fact the the human robot interaction so
um I will start saying what are our
main goals what is the main architecture that we use then I will see OK how do we implement this architecture for the now in the field of application for the robot to I even video well Alieu doesn't work so I'll show you the street at the and without any out so we forgive me the shit out of 3 all the problems that there are uh news this year and next steps conclusions and then I'll show you uh them of the robot actually doing something so OK what are what our our main goals then Our main goal is to create a robot able to afford these communicate interact with human you to speech gestures and invited expressions because obviously that's the the most effective way to do it so what and the other thing is to actually build a 3 model softer infrastructure so that we have the possibility of integrating different robotic platforms integrates yourself to a really easy and also uh remove us all all software so um this is really useful expecially for for testing purposes and performance assessment also so we can test softer and change it whenever we want so I would it achieve so we use level Python libraries for spoken language processing the sentiment analysis and vision and also we use artificial intelligence applications and yeah so uh what they mean blurred demeanor architecture actually is pretty simple everything is based on addressing the got what white us indigo because the and this matter library that we use for a few States machines runs on the on indigo so we're stuck with that also that the state machine so actually on a have meant a separate bike breast prosthesis or separate Python scripts that and they communicate with the with the US to now what are the main benefits of these architecture so thanks to all of this and than to the state machines that actually run as different programs so we are actually able to to use a synchronous computing and also achieve multiprocessing it's not thriving is different they're just different programs using different costs so for the since it's um structure is modular because that was 1 of our main goal such then solve code can be added and removed the without problems also since since it's a idea ASR-based basically every component that was uses crossing 90 way can be added to without problems you so now why about where do we actually use the the now mainly in entertainment education field robotics home companion robotics hospitality and robot-assisted
therapy now now there's this would have been the video but to whether or or not robotic
platforms that these are like that the way that we designed the 2nd architecture then this can be used to be used on heavier about but based on what the rest days or not because in fact now we're not actually using the us to communicate with their were just using us to communicate between the state machines also if you want you could also use the the architecture to work on non uh robotic created the projects but it's up to you now why you did we perceive that to use that than our book well it's actually you are 1 of the best for my available you might what's there and actually not many other options the I D or the 1 is the bet pair which is it's it's evolution uh also since we actually need to interacting humans having their are you monitor abroad what makes them it would with the human drop what you money can actually connect and not to emotional level so the interaction is a whole lot more effective the now how did we actually implement them and our architecture for the for the now so I there is said the user rosters for the communication between the same machine and then we use their knowledge the librarian which is up by some these are very to oxidase and that all of the comments tool to the robot now and we use that quite high that the tool in further from natural human-robot dialog so what is I I'm basically is just some uh and um a programming language that permits to actually create chat bots this is at the moment kind of the only way to create an an effective um an effective way for the robot to communicate with the with the you that actually with and we done that's kind over there are you mind since you know you program them um also we actually heavily use Cosby solution especially for a voice-recognition and also language does translation because some they're del their local these 1 equivalent and not as effective and and also the Cosby 1 hour a lot faster and a lot more accurate but also why language translation because some basically if we add some kind of automatic translation then we can just go to the config files say OK and 1 to tell you anymore I want a Polish and whether the robot now can speak another language um I also use the narrow networks at the moment we use it for object recognition and the phase information the the
now this is the actual amount of the of the state machines so basically the main Python script is called the human right when you call humanoid it'll a deal this 5 state machines which are sensing inside thinking I can speak and acting so now you might ask their own uh why is acting speaker invective move connected to sensors that doesn't make any sense but basically you when the robot told so it's if we didn't do anything it'll still continue mean and will go crazy moderate less than do anything so we need to make it you with 2 1 when it told someone it moves the now this
is the that this is the inside of the of the
scientists of the site's state machine
basically what happens is there are different states and the and basically we you can just have but 1 state of the time you cannot have like chew on um chou watching at the same time just 1 at the same time this standard 1 is idle and basically when I talk to the robot to to alert changes tool was evident I ask him to for example ask me all way might it will go to face a combination and then when it's done if we go back to idle now do is the option of
getting not done before although this is not everything that they can do actually can do a lot more things but for the sake of new readability we cut them down otherwise you couldn't see anything it would be just a mass in it so that the MS right now imagine with 5 or 6 x the extra machines you can read and and it's pretty self-explanatory and
also we have a movement control which is the pretty simple against OK
um now 1 thing why did we actually decide that for a web-based solution it added and and you know um a local 1 so basically we can deduce a lot this if you load them so basically for use something like a laptop and then uh everything can run faster since it's on the cloud and who cares I also you actually need a fast Internet connection that might be a problem in some cases because if you don't have it then yes screwed basically and also obviously the tools provided by the companies are usually the the best available and the most reliable the case so now which extends for these it is for some reason decides the inverted the should not be so in OK um what wondered that the main problems with the with this so basically addressing the goal is an is supply is uses Python Chu stand there obviously if you have a piece of code that runs on Python 3 you will have a problem with compatibility basically to solve this problem what we do is just OK and I need to run a Python to escape OK very easy I just say OK when you're going to state just execute Python script that somewhere else and then at the result and make the robot speak or whatever whatever you needed to do also know right now when we change their but platform we ever really really big problem because and every time every every time you change it basically you need to vote um we need to go to search around on all the code that you wrote and said OK so here they're Robert speaks and use this kind of an this kind of common so I need to change a distant assign it to change it and you need to do this basically on everything that kind that interacts with the with the about also and then our robot for our needs is not really powerful expecially in the in the sensor Department like for example the microphone that he says it is the not powerful so basically we actually like need to use an external 1 tool to make it work and also the objects traitors department so basically it's silly names are not powerful enough so if you if you ask him tool than on um pick up something to so it was fading from most of the time but also of other and the grape is really basic is just like around and pinch being but basically just is just like if there were 2 fingers and if there are 3 so retarded the future steps that we will do to actually correct every at every problem with with this so um always your eyes you I think I could understand that this is actually actually project it here working on with the company so basically a lot of this stuff unfortunately is a close source so we can just show you what uh how it works but now we want to make them we want to make that the skeleton of domain architecture available to everyone so and on if you want to do is to make it a prince was so if I don't know if you want to start a new about it can project say OK just download them and the architecture puts on putting that on my my what the commands in the config file that we would create and basically you can just out cold really easily remove it and you could ever a robot working pretty easily also obviously since they're now is not enough Herrera needs we need to change in to something new and hopefully more powerful the and also yeah adding got config file so long as I said that when you and you need to change a something that under some all on the on the on the Robert specific task then you need to check that the softer so we just we use create complete 5 to say OK uh here is the web cam there is here for a is the island detonator for the left arm whatever you can do everything there you need to actually uh my modify all of the stuff are also in we might add some in a companion Robert to then our so do just a temporary solution so it to actually household of the art necessary hardware to be they could just media de desktop computer and also uh all of the sensors that we're increased that the now once because they just don't work really bad also I know it's kind of early tool for conclusion but don't worry we'll spend a lot of more stuff especially with this guy and also we have a video um so basically with this robotic platforms we have been able to to to achieve that the main goal to we initially achieved which is the thank thank god and also and we actually don't seem to prompt for maximizing results with the Madeira there now the embodiment and also obviously fightin is that the arts of everything and we exploiting the but their batteries included philosophy so we try to integrate everything that is already available and now it's enjoyed the you more and hopefully the the she has a the but share
in our let's make it start to actually you and the
the we journal so
can you see everything yes it was
expected that Don so basically as you can see we have it is kind of little space to actually make you see everything and but for every state machine that is like from a log on external so you can you can actually see what it's doing they also forgive the spelling mistake and here you can actually see what more what what's actually going on now on real-time now the problem is you you might
crash so within and doesn't of so forgive if everything is
really slow because this is just an area are not very powerful computers
for the speaker is not so important and
maybe with this the wikis
and something missing in the case of
of there is a clear if you can see what
actually is it today so 1 D by 1 small problem is that the the in the now for the moment can only speak in Italian if you're an English speaker and resource OK so for example we could see the 1st let me see if that fits it can actually recognize me you know the
and the of well without it they should do but doesn't so can do
so for example I could ask him for I don't know I could ask I about something so yes for the heard a lot of stuff so as you can see this trying to to understand everything that they said and it's nothing to recognize I said a lot of stuff and it's stating your private of time that's 1 of the actually it on of the main problems that those basic lecture a day um the speech recognition is is actually based on the cloud so to actually uh send all of the all of your and then to get the tantalizing dated back takes a lot of time especially if you if you have a really big words yeah and nest Gonthier the music through that he has school until on the misguided
the I the he also may go it alone or for example I could ask you if you like broccoli at the
catching properly the the
and Scanlan call it a land FIL obviously doesn't work any times that every every time so we so you actually need to to improve everything is still out pretty a pretty young for a project to catching up so I and you have a from the and and I think working times instrument as you can see the the be since they're actually and about using in English effects so um translated the English making him say something eventually to vector Italian so some sometimes as you can see doesn't work basically is just understood the often blocked diverse baroque we chemicals and we like properties as you can see it's and not to let not policy at work so earned Mr. Etruscan something else seem to much stuff commodities the under this because of the not the best we can basically you anything and she had your device the judges show us Finland collateral space so I mean the no doesn't really with the thing the space of all of the iterative come on the way and and I'm not and no no using this so be and that's the prominent Islamic frequency yelling small thing that all in all it effect so as you can see the performance is really is low because amusing on the current site we computed and reuse lawsuits a laptop usually reuse of a much more powerful back so so we have a lot of processes that were it's Levi's reusable also and carrier also the intimate connection it's not really the best uh pulsates hitting everything is a sense if a sense sensible microphone when it took the speaker of the 10 year everything was that best him for some object recognition maybe the peasantry work because as you can see that the camera is not really the best let's try anyway so and the let's try with the a water bottle because why not component of the think of this as here are the it and it there and I think I think and I can take the and I am 1 take the we as you can see it's actually work in the states I will all all all all all but there's a of all all all all all so basically as you can see here it tries to understand what it is and also there's like that life which is new reuse messing with his camera and obviously orders recognition expression it is a level of a few non-Arab networks is kind actually kind of hard I we could try something else if you want if you ask me OK and making recognize our glasses Matkin coin if you want the uh also at the show and not the answer maybe doesn't really and well the with that let's try it unity in a way a not it for actually let me do this so some of led to minister idea very legit meanest idea all in all they do not cure and so basically if you're given our books you can read them and that kind of actually it's pretty easy to do it's nothing there super-hard why is not is it it what working now it's the mutual knocking on q y the and there are a lot of problems as you can see binocular and and this is part of the 2 in there
and the machine and all the things you think about that and the reading the book from a just simple tricks you finally get all in all it because something very touching is that and that's that I you state she when you using the standard or not when the species now thing di t the diet of course patient yes and basically you can meet you or I probably truthful and so on
don't worry it's norm of the time and the end of the project and the the managed in a and of a the and but what it that you would that's that's something you funeral
so basically you know where this summer dummy and basically if it's understood the in interior to the following I want to the the minimum norm so all but and in this way are the joystick and can follow me the that's
the you know the the actuators and not the best and thank you the the that's in In more the do you think we on that and here is the creation of something
and the
moment you know and and this the 1 thing that this also as a model of the appearance of the and that is what we have it's really not powerful what the end of and I on the the the end of that on the of the I think there is now let me try this of this we and the the and you know it the whole we all all the people of the time you the theory that the noise to the volume of the obviously it seems that the more it must work or what actually we all saw it can actually also the Everything is big enough I don't know in a try with this because of you can this is on the laity the the and the the and the let the looking at all and the DNA In this means that we have same the land in the the the following sections so what we I'm I'm really let G for all the they the the words the digits the throughout plant like read that that's 1 out that's kind of the only problem and basically the 2 of big letters because the resolution of the camera and also the the focal points and kind of crap so if you have actually had this income enough you can actually use this with no even like reusable tax that you can just take a book put it there and say OK uh the the anyone read whatever it's on the page or before from and for the to bad but also of the could do for a lot more stuff for the moment of Ikerbasque in where and for example demand there and the demand there OK I believe we will for example the you will still the we've in there so the www the where you are on initially gave a lot of things in mind that you need in it selects all and all I don't know what all the all the all the all the all the all the all to the all the all the all for some reason you could grew will have no proof why you mediated by yet basically as you can see the excuse that the some really big problems think of it this is kind of 1 of the best the speech recognition software that you can actually use the right now it's not really the best because these all of the actual from the microphone and gets really confused the pretty easily so you know basically now you could ask everything and to and the we could also for example a desk and microphone and should ask encomio what the the head of the label in only eat all of the people they all they still going on all the and it works actually right rear criminal grow through from her own they all have they all along the in the detection of strong a change in the morning and I think the schools of thought these half and the true from you how to know any of the things you had all the money that can get in the extreme found to many of the things that I would be really bad and the strong the I freedom of us that we must look at here and all the way all of the time in the end of the game the information which you want to understand the nature of the walls of freedom of the Fremont Moscow and what the of so we did no an outside and go from if you went out to basically just make a lot of us machine use a bus stop them up but what everyone is like a config file for that and she it's goes back to idle uh I don't know we're still have some time so we want to see them so that more or not so but as a grandson of by the sequential and in the learned predetermined
and the and the united under the name of it and move it to the end of the best you you and so do all the the 4 and so on and so that it it it would do what they 3 9 should see the full manager that begin the so basically this is the machine that there's a character that's essentially as like the great and everything else that says OK it's fool them growing up and it's what is a can and the the and the and there you have it but there's new thank
I don't know if elected idemitsu recognize
something or communal tends to look at the 1st of its type a few questions OK so far stations are enough that then you what you want to do if you wanted to know that you need you want and continue showing stuff to the can do Delaney maybe 1 question of any of the and well like on the right on on the on the back of a lot of things and a the question is what happens if it's also the other side it used to go and the environment and on the on the right you can do it softly having you know this that I and the whole be well but the fact that the any other questions are shown just for and it would be good hi how had that actually this camera I was wondering about is really mean to call parents what you see the same thing that the 2 robots he why don't you for instance use that why some sort of infrared light so you're not dependent on to uh so the camera and the C is the same thing we see right now or you do some sort of looking while in the infrared maybe do not dependent on the environment alive and the and and and the the thing and did you and me and some of the work on the right and and on the set of all the same thing and some of the things that you find that you the size of the and of the and the and the other key challenge of questions on the the 1 question can you think of some objects can you think it will take on the ground and so on this feeling like the fact our so as it is right now it can find valve can take objects on the ground so basically that you can just create an animation for they're all booked up and say OK when you're in here in this bottom then you can just to run the animation to pick up the object but is not actually as marked the uh object begotten doesn't say what chaotic organized that is the object and then I will pick it up and it just says that I go here and then I executed this has to be the behavior that's good on because the thing didn't get back up we actually do it and not really that that kind if you want to uh can come to show you on just some things the growth that thing but on the and of then you have to move you the good the yes instance of a few dumb you good the and dynamic Tommy and and now I mean I got me and so so the you might sound but the that give me back the net in that it's not prominent so beautiful this this phenomenon yes and so the dummy ways not executing it yeah world yeah the and you're taking it out of his hands stealing from a if you just it I have basically you could just know it's kind of far cheap way to do it like a toward the in so I think you thank you very much that showing us the bubble sort of like broccoli and these kids in work warm-up lotteries for this movement thank you that and