Bestand wählen

Don't fear our new robot overlords!

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Erkannte Entitäten
high and scholarly that chair for this session and the 1st is from that select best left is iris professional handling and he's also the creative and you think of a and and his soul case don't fail on you know about the world idea of minus the
local and national operator prominent and I want to talk to you about a new way to test the moment advisors but 1st of all thank you for coming because I don't think this is like a very Europe Python talked a lot about back and something and you put in there net and something like that it's a very physical it's a robot arm and computer vision and stuff like that you can see the just setting up some video and you will see everything just fine I hope so at least at that hardware do with them all but there's probably a better idea anyway so and 4 show me that my little again so 1st of all going to talk about the problems and has a mobile devices is if you talk testing now that absolute apps on that side that our mobile radio or something like that I don't like the word that's what it's about and then the idea of why and what I was coming up with that the concept of how we do this and then I'm gonna show the hopefully a working at them on and some code and then some conclusion and learning because there was a lot of learning to actually and so talking about
problems in general and has a mobile devices and you just need to check because in that AP or ATM possesses is usually a front end for some data services you need to check how that looks if it detects a right if the localization is right stuff like that so you can do of course tests where you just check coordinates font names font sizes and colors and so on but in the end you probably never knows that's really good looking more of the coordinate system was that that's possible or other stuff that's just all so that in and you never know and so you just need the united to check and that OK if you're just doing that with like 4 4 5 Use of small that's like that's OK but if you have like a real our trip project with a lot of society you probably gonna miss something and if you just want to do a test after each deployment for instance and you need a lot of manpower to this as well so of course you can do some testing with non real devices like simulated on I don't know who
of you is mobile developer Western context of something like that OK was the 1 who sows simulators something I IOS uses for instance you run it on your Mac and and it only runs on the magnets 1st problem and the other problem is simulators because they're simulators with emulators and have too much competition power because they use here that I 7 something on on your device on your notebook or I at whatever and then you have no memory issues and that's 1 of the biggest things you have to keep in mind when developing mobile is how to deal with memory and working memory is issues mainly working with them and you don't have that much memory to store the image and so I need to find a way to something that and so that's an issue and because my local has like 8 gigabytes of RAM about friend that's like them and know 10 times more than my phone or even more so no memory issues and of course it would have a touch interface and if you see the ads that are developed by so vigorously and the only used simulated to test their interfaces the always see that the problems of quite timely and of course if you are running the simulator you can just use the mouse to well that that when that works because the mouse is quite executives have that little thing you touch and but this thing they're all 2 fingers are way larger than a mouse pointer so that's and assume and now that it's you and especially in I was about roles engines of slightly different as a simulated users directed that come from a guy last and the realize uses I and yes Bozlak that kids both nearly the same question but only nearly the same question and they have just different box and and some in the and the will of that G L for instance all our open GL is and slightly different or not available you contemplate to stuff and simulator and some other stuff as well and then
you say and you have an Android phone so and have an emulator and a great because they're emulating the hardware but they have other issues for instance they're incredibly so on if you ever use the Emirates simulator and in later probably have registered that it takes about 2 minutes to start up and it's hard to interactive interact with especially if you're doing CI testing and because you never know if that was started if the stimuli was something quite hard to so make group around that think so this problem and the Lakers and then of course every manufacturer of Marathon does their own roles interface and stuff like that and you never get that all the images you need an emulator to assess all those so and some zone uses a different browser engine all but engine then that's HTC they use different versions and even if you have let's say OK that's the and what Kitcat fallen and yeah it's Kat but only on the nexus you have the right that engine and on the next device running to head there's another engine another framework and and a little bit issue that can be fixed I heard is that money ready don't have people that play services and emulation and so you can't use Google Maps for instance of this because of this thing with so solutions were sold the best solution of course is on to test on real devices using real important so fingers and attacking eyes so you always need to know how it looks and the best solution right now is Media human and that well that's OK but I need some sleep and well so I
set off to new Frost and try to fix that is you can have an idea on this idea
what came up to me if I saw a talk about sort of talk about equally for automated testing of sites and that he was just to build the equally formal advises the and young light up OK what's Korea yeah is equally legally is known as the
see and that's an an act that you run on your PC it's based on drama and written in Ji firms so Python thin Java and and the quality of is was made of a tool for automating computer task with string so as can see this is that in which just around there and and it's very Python like and which is clear image type strain click the image and so on and so on and and that's very easy of course you can buy ads and stuff like find that part in that hard and stuff like that so great and was amended by the MIT is developed at the University of Colorado Boulder at these equally that's on it still maintained and stuff but it's made for their thoughts and not with the eyes so to get it running in a continuous and the testing something you need to write use the robot framework and do some editing and it's incredibly hard to run on tried it's not that good because it's not made for running catalysts and stuff like that that's possible and if you want to know on novel about it I am probably know some guy who said that they did that for much and it works so and he was making the
same from all analyzes insured makes we should use computer vision to detect icon and that in fact that I can and for that type in part use a robot on because robots never so 1st things
1st you do changes and doing screenshot is some quite easy if you all all connected over USB and then right you just use the ADD command that's the comment that came with that comes with the and and what is the k and is used to interact with the device and you just do some mathematics and longer terminals being you put your terminal and then you have a screenshot from works in 19 % of the cases of most devices sound of course to do some stuff different because I don't know why they just do it on and I ask you can do a screenshot using X. cold and iTunes and if you're having i functions using axioms you probably know that there are a screenshot of the interface of items and they're coming from the device so there is a channel to do screens of but there is officially no way to use with the kernel they also I from users humans using Linux and I don't know why sleep but they also reverse-engineer the protocol and called it the by mobile devices and with that you can interact from the terminal with a knife on device and you can do screenshots if the device has in developed a modal that's well if you haven't even it was this this property in the local not so quite easily and screenshot yeah now we need to
be taken can't and so find that needle in the haystack on and for this you can just use all the ones obvious right and you just to attend a recognition and that sense for finding an image of an image and that's totally easier to say I'm just if I start and stop the project is the 1st of this is probably the hardest part getting the computer vision stuff to work no it isn't so for those who have never heard of Old City it's those CC post has written library to do computer vision and this has great Python represent our so great that probably everyone who's using all cities using pattern so it's quite simple API for use of complex process actually while 4 lines or something to interact to find that in the limit a state and in the end you get X coordinates of that stringent strokes and size so that thing you wanna find is get a conic coordinate and the size and then you can just calculate the middle point where you need to still now we're talking about
robots and finding what what this is what's not that easy actually the problem was and he wrote that's quite fast and the the role that's not too expensive I don't want spent I don't know 2020 thousand euros for industrial robots something that that but great but I don't have the money for this and my c-MYC was like well have you need to cite find something cheaper so I found this maybe here on this if you can see it you see there it's called the text of what or truck test as you can see it's of and delta robots that as a robot is a robot where you have 3 fingers that has that actuators so that thing that interacts with the physical work is in the middle of the it's made with an area 3 several just like standard so as you get from that are see hobby store around down the next corner and weaken apart so if your own friends of this thing will cost you with the bolts and nuts and everything like 120 euros if you don't own a see the printer the it's it's around 2 and 50 euros printing on because I only 1 on the front of my own 1 at the pre must like cheap and we use a laser cutting the large parts so that is stronger but you if you just have a preprint like and ultimately the printed out and yet that's that now we come to point and do this with broke the drivers written in Algiers and and we come to that later in the day and so talking into
a lot and of course we need to interact with robot alone on the time what tap and stuff like that and to do this I decided to use much softer so why WebSockets and 1st of all the robots came before the driver that already had some that some that supports only only needs to extend it a and what of could could be a good solution for this because you can traffic the change of the robot without the changing their Python code so you can just use another reward so as I saw the robot is written a s and then there it's not that good and the not just at slightly undocumented and yet have come to that later as well that so effectively brought could be changed easily because you just have to implement the same but uh thing so yeah moral it's if you have another 1 of find better tells robot or not but the vector X Y X is robot you can just use another 1 so that now
where coming to that part all or put it together and bring it to work and and that's called protocol NY I don't know what's on another defense in and the they just called it that way so that the idea was to test the mobile web site on just written in standard Python unit test set on because if you're ever used like the has 1 and 2 you know that I'm quite easy to write such tests and of course the 1 good thing is that the usual CI so as like Jenkins sorting city on now using the city because I'm living in a job environments with all we want independently I so you can just you use unit tests as a standard you don't have to write your own parsers for the outputs of a graph on all the heavy lifting detecting the images in an image and making screenshot all
the heavy lifting is subtracted away you just the right that test that's quite easy In the end the uses right but test where you say that their point is that there find this in this region and stuff like that show you in a minute and right in such tasks as quite easy and even like to a guy can do this so you don't need that so if you want to do regular testing of large project on not is 10 give something to like this to query and they can write that as the cells greatest well and they even don't even need to make for this which is a prominent here and I doing I was at because they usually don't have an met so recommend to the part where I hope we don't face showing you they them as a I came back and he
sneaks into do some set more room so OK so we're
a so now we have a video feed from following very and now hope OK so there's a robot and down right side as correct the minute you can see the phone and I know just show you the test treatment and then we doing their thing whether all at once that yeah a chaotic is tried to do that with them microphone a case you can see here this is from our little example tests of quite cheap because it didn't have that much time to do a good test that's the working because of that robot and Kansas show you so there little set of it is a set of the test that because they're taken with and setting it to be but through truce just to show you some of the images of what the robot soul on and you just give us our command and then we have this little thing to find the popular speech of and and this is the test of course it's not that a good test of something just them all on so 1st thing is we want to find overlaps assert that if I give it a screenshot of the setting cycle that's the some in the conservative later should part of the screenshot and they I want to make sure that the taxes and if attacked and founded and headed I get a region object that certain OK attach it that that thing
from then on doing this kind of find is just look on the screen and find subject and returning the region where the loss of so this is what I did fear of just get the title of the settings in Germany and I've been and and that that title to users and strong up the table view of settings so we we can find the airplane mode settings you just see in a minute on so we can gonna look if there is in the airplane mode settings in the settings so maybe the use of all of the developer forgot to put it in there and and then we use still find and commands and if you using the finding commands but just given Of course the screen 1 find that I can actually give the original object as you can see that primal role that will go up back from that find and so I can just sort in a certain region because maybe and there are you know more than 1 switch so that's where you do of Boolean value but in their views so that I would just will find that would get multiple and never know if there is 1 in that region and then I just talked about the settings and it's just see if there is a the Bluetooth standing in the title a very simple task of course and then have a tear down and just tell the robot to stop what he's doing so that's the test now we
come to the point where we're going to run that I can so in so OK right and just see the screen of the phone and that's what just here place so that for testing just for the for showing you that there's a screen and working on the left-hand side see the role of the year and use see itself it is still present this point so I fail because on the flight notice on and and they're searching for another item so just let me just quickly fix that all mineral find of but again OK this that's the standard of Florida argument has growth that something so that's Self yea that's his friend if you come and no that's not the
protest but to work with them and
so what the role it gives you an output images if you want wanted to that the but the truth is I just showed you and then you can get the those the fancy of images in black and white that gives you the idea what the robot sure as this and try to show that to you on so it's this is just a screenshot of dust from that just the screen of the phone I see they're several of those of course
and you and then you have this on deeper and they can see OK on you can see that there is that square around that's settings but I can't and
that's the square on the settings title and then use it again then searching for that airplane things on it and then we searched for that Hutu but 1st of all we did not find an and and I don't know exactly where did this by Paul entered that on weights so that they have another name in the file and you don't find the consolidation concede that book to switch on in there and then we went to Bluetooth so that there the role of searching for and then there's the title of the Bluetooth setting cells on this is how that looks and of course you can still you would click on of the code of the unit itself that
library library looks and so it's no that's not a memory that's my tests the that's the right so it's so probably the worst Python code universal because I'm not present alternatives see guy an object by wealth and practices my I don't know my hobby and save some of that good scarce and it's probably not that's what you would I am that's going on and so the thing going to act as I just showed you that test the class and giving you those commands on this summer's thing of course untapped find find what's missing of course as swiping and double tapping so that's something we didn't implement but let's just see what happens if I do that tab on its cell tells find a class of that's when all the refund a fancy name and that gets this where you just get the device to a screenshot using an idealized so that's the idea I was part of yeah part I was is the forward but you just can't give a magic number of 1 to use light on so that we know locates and and and and then it doesn't screenshot finds its creation of and so on and so on and if you do this and we just the the moment the dividend the act right now and a on OK that support and evidence that actually so that's added suspension of find screen year OK now we do just that that and see the part where it is due out 10 plates imagery find that image and get the time do it getting it's getting needle height and weight and so on and then you got see that there's message see the 2 T and C C L E F eV along at actually don't know what that means but having their several types you can use and this is the 1 working message spread from on and then you
just do that and that matches a match you can just get a location back and so on and then because you could probably be more than 1 contact usually fond of city we define the average of those so as maybe Jesus to we which just do the move out some something there and then each other robots to tell on that the robot has its own thread would come communicates something so on
that was part OK come to my conclusion this could be useful actually if you're having a lot for Apple on that website want test animal device could be useful so I would probably keep up doing this and going much further with that of the problem was uh I don't have that much time to in the middle of Venus are needed and that's because of that robot but and come to this later and testing for existing projects especially on the thing where 1 is that of course tested in development is important as you probably heard that you it yesterday but on if you have a lot of project you always wanted to have a test of going on and on of all your stuff if and to just check if everything is right especially if you getting paid for certain content you wanna make sure it's in there in Roberts never get tired so they were the words of every time every day and nite at every time you do a check and they have about the same sleeping cycle as a developer probably not and of it Kuwaiti has and slightly different fitting cycle and the role of hardware to conceive working OK it's fine but uh As OK if you're looking away from that little dead on cables there but it's OK if it's working but this hostages is on that we're coming to the running on a function
in a function in in a function is normal but acknowledges that is so I don't know who came up with this thing which I and I don't know who wrote this code that I know it was discovered that but it's just terrible it's hard to divide what isn't documented that coat the specific code but the whole that whole layout to filter concentrating getting the function getting the function and its on non-blocking well so that and never just to get a project of of what maintainer never they probably never going march in your comments are your output requests and just never answer to your issues and probably never answer to him as well but that's all open source but you kind of opens if the finally we should probably users far more than the so we do right now if you're doing all the city then and it's not that hard to learn actually them very very nice documentation of a good example of stuff like that so all the CVA and that's
and thanks for coming of any questions want to is it other you want you want just see the demo again just to that no empirical test people and and we do something quite similar with a library of wrinkled guys again but it does visual summation of mainly of the moment kind of its strong points will be around windows but we we haven't covered for could be applied to the testing of mobile devices and stuff so I'm really interested in coming to talk to you about your book of but the yeah I just
thought it might be nice to the nearby and it tries to serve as a yes or no cost of and the only problem I have a spring to finish and I'm just needs a set of quite soon actually I'm just here for the day but you can just write may astronomy or just the catch me at that point and the actions of the robot is not my relatives from someone else on the yes and and but of course I started writing my own Python driver because I'm getting tired of that noticed on and the problem there is that the inverse kinematics that's a fancy telling the robot which removed and I so badly in that node yes teleporting into Python doesn't help so I'm probably need to do some of 1st but and the tube work quite better than now yeah the other questions on so you probably expect that question did you try to use it to get a really high scoring what do you think of that the I don't play from what I what I play is 2048 and so as I guess that's quite good actually that rewards forest that idea of that guy who built the 1st had the idea of playing and words that and so you can just model as manipulate the the end of angle where you're just to move their very precise and and that works and so on very but not just called works better than the rest but OK and expects the fun of it but no I never tried that we only use it for work purposes of course but because my boss paid for by the by the way I have a 2nd solution for upgrade from the rubble appear in the table and so you can also perspective yeah absolutely would be nice and I already both something violence and so on and what you can inject quite a lot of you can inject tests on the most phones but not at all and that's what I've been to the disk but you can inject location data rotations of so the screens prepared something that that on Android on speak on that accompanies on I so I already wrote a library that takes location I'm on the phone virus and that's on the top and and and the old and library to fake notifications there as well so and some stuff can think back but to the tilting concentrate so that's something you and needed probably tilting table tilting robot and well just 2 symbols right so the sake any more questions the so I think so actually have a group you questions so the first one is on when you're transitioning from 1 window to the ordered to use like a hard-coded delay or is it actually inference by the robot and is all kind of on the problem is the robot doesn't know when it taps and so there's a small little styles in there and you can probably see it's paying because that's the only call you get too small to fight isn't assault on and that's connected to ground so and it just needs electric running to that it works the problem is and that there isn't so much so much kernel to count just can't detect that because there is no just saw a tiny amount of current so not working on what I just do it and that they come from growing present telling robot all of that of indicate that and the robots described was just knows how long you could probably gonna take plus 10 % better attempts and then just make a test signal but 1 way to fix that and there is but the little rather on that styles and he could probably do a little switch in there was just some foyer and no metal and to get a real-time feedback from it right now that's and isn't in they're on because you probably needs to replace that side of so that's something that by but that probably he was could be used and the protocol this could use this could work this is a and I have like a small suggestions of how that could be used for the performance testing because you mention that 1 of the limitations of the mobile testing is that's when your emulated on on their laptop then you don't have the same resources now but then and instead of because externalized all the manipulation with the apple and what's not division 1 which is actually based on this national that is taken on and by the end of cold that is actually the always ruin device so I would say that if you will grow with the web cams that is actually mounted on top of the um mobile devices and is actually trained using the deep learning algorithms in recognizing the rather than the Snapchat itself with sequence of them and then and evolution of actually like clicking from 1 to the order of you you are actually training what is the the elements of and that could be it could be good solution and the problem is of course in that set up and if you put but come on top there's the common just and from the camp so that would work and there's so 2 solutions for this and 1st as you can see edges transfer the life image of that at of that phone by am play we tried that but to do that written at the recognition and of course you could just see if it's fast enough and stuff like that problem as that's to body so it's too late to flap is just gets a connection or something that even if you do it in our talk of a wireless such as a stored here and the other thing is and I was 8 so coming in September 1 on IRS and you can already do this on some inductive bias that you can just and use the screen as that can for the PC or your Mac so you just have to be life delighted you feed coming from the device on that somewhere quite hard when you're at sea and that could be a better solution than having that can and robot arm in front of the cell yes of course doing stranger is a little it's not that performance testing of course more like doesn't look right any more questions and no quite fast when I speak so and this kind of pressure and 0 did you experiment with the speed of the robot so we have get somewhere of course it's quite limited because of those services are not that fast and made for of flying nasty things and the robot can be fostered the problem is not the but the problem is that the matriculation and the transferring of that image an iPhone 5 like this 1 has an image of size 1 comes from the screenshot of its 1 thousand 100 36 times 640 something by latch onto Kinji uh and um this has a very high resolution so it's quite sold to screenshot that takes about a 2nd and of course the image recognition is not that fast because it's based of images model might be your feet if you would like to back to the video feed and they probably could be faster but with this set the officers all the good thing about this is you don't need so much computation port do something like this on PC C or on your Mac or something and you probably do this on they play actually tested as part of the it's a little flat piece and I don't want to do a damn on a device when all of this could break so yeah so you could probably set up uh some of those like 10 for sheep because you don't need such a powerful computer to do it or you just do it with 1 computer for like 10 robots this of course possible as well thank you but OK well thank you very much thank you for
Quelle <Physik>
Humanoider Roboter
Gewichtete Summe
Schreiben <Datenverarbeitung>
Radikal <Mathematik>
Statistische Analyse
Humanoider Roboter
Ordnung <Mathematik>
Tabelle <Informatik>
Objekt <Kategorie>
Folge <Mathematik>
Algorithmische Zahlentheorie
Virtuelles privates Netzwerk
Gewicht <Mathematik>
Reelle Zahl
Diskrete Simulation
Äußere Algebra eines Moduls
Mobiles Internet
Nabel <Mathematik>
Wort <Informatik>
Prozess <Physik>
Inferenz <Künstliche Intelligenz>
Komplex <Algebra>
Kernel <Informatik>
Statistischer Test
Regulärer Graph
Maschinelles Sehen
Lineares Funktional
Web Site
Projektive Ebene
Web Site
Zellularer Automat
Physikalisches System
Inverser Limes
Speicher <Informatik>
Bildgebendes Verfahren
Leistung <Physik>
Einfach zusammenhängender Raum
Wald <Graphentheorie>
Objekt <Kategorie>
Dreiecksfreier Graph
Offene Menge
Demo <Programm>
Statistischer Test
App <Programm>
Kategorie <Mathematik>
Güte der Anpassung
Mobiles Internet
Kontextbezogenes System
Dienst <Informatik>
Rechter Winkel
Klasse <Mathematik>
Dienst <Informatik>
Skript <Programm>
Inhalt <Mathematik>
Algorithmische Lerntheorie
Protokoll <Datenverarbeitungssystem>
Open Source
Elektronische Publikation
Einheit <Mathematik>
Prozess <Informatik>
Funktion <Mathematik>
Gewicht <Mathematik>
Interaktives Fernsehen
Framework <Informatik>
Zeiger <Informatik>
Matching <Graphentheorie>
Dämon <Informatik>
Automatische Differentiation
Manufacturing Execution System


Formale Metadaten

Titel Don't fear our new robot overlords!
Serientitel EuroPython 2014
Teil 119
Anzahl der Teile 120
Autor Brechler, Philip
Lizenz CC-Namensnennung 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/20019
Herausgeber EuroPython
Erscheinungsjahr 2014
Sprache Englisch
Produktionsort Berlin

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract plaetzchen - Don't fear our new robot overlords! This talk will show you GoldenEye. A setup of a robot, image recognition and Python. What could be better? GoldenEye uses computer vision to detect icons on an attached mobile device and then tells a robot to tap them. All off these tests are written in python's unittest module and the API is as easy as possible to make writing complex tests an ease. ----- GoldenEye is our solution for mobile front end tests. Testing on mobile devices can be quite devastating: On iOS you can write front test in JavaScript in Instruments but it is quite impossible to connect Instruments to you CI solution of choice. On Android the situation isn't much better. Other front end test frameworks can work with mobile devices (or simulators) but they lack the ability to see. Of course you can check if a color is set correctly, if a frame has the right x and y coordinates but in a world of different screen sizes writing these tests can be quite challenging as well. In the end you will always need to look on your screen again and again trying to spot any issues. GoldenEye takes a different approach. It does not need to run on your development computer, you don't need a Mac for running tests on iOS devices and you can have real touches on your controls. This is archived by using openCV and it's python bindings, Pythons's unittest module and the Tapsterbot, an OpenSource delta robot made with 3D printing and an Arduino controller. To write a test you just take some screenshots on your device, cut out the icons you need to tap or inspect and write a very simple unit test using a high-level API that takes away the hard parts. WARNING: This talk features a real robot. In case of machine world-domination: RUN!
Schlagwörter EuroPython Conference
EP 2014
EuroPython 2014

Ähnliche Filme