Keeping your tools safe IDA

Video in TIB AV-Portal: Keeping your tools safe IDA

Formal Metadata

Keeping your tools safe IDA
Lightning Talks
Title of Series
Part Number
Number of Parts
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Place

Content Metadata

Subject Area
Pixel Multiplication sign Water vapor Coma Berenices Mereology Formal language Software bug Hooking Computer configuration Bus (computing) Cuboid Software framework Position operator Social class Touchscreen Linear regression Cross-platform Software developer Moment (mathematics) Bit Category of being Process (computing) Repository (publishing) Order (biology) Website Resultant Row (database) Web page Server (computing) GUI widget Observational study Divisor Computer file Distance Coprocessor Product (business) Revision control Crash (computing) Operator (mathematics) Software testing Booting Metropolitan area network Euklidischer Raum Module (mathematics) User interface Pairwise comparison Projective plane Independence (probability theory) Word Kernel (computing) Computer animation Logic Network topology
Web page Discrete group Scripting language Pairwise comparison Server (computing) GUI widget Interface (computing) Cartesian coordinate system Particle system Word Mechanism design Pointer (computer programming) Computer animation Different (Kate Ryan album) Logic Personal digital assistant Fuzzy logic Website Speech synthesis Software testing Position operator Computer architecture
Email Electric generator Computer animation GUI widget Interface (computing) Source code Vertex (graph theory) Black box Rule of inference Formal language
Discrete group Axiom of choice Presentation of a group Group action Multiplication sign View (database) Plotter Function (mathematics) Computer font Usability Medical imaging Different (Kate Ryan album) Semiconductor memory Oval Videoconferencing Flag Endliche Modelltheorie Position operator Stability theory Exception handling Scripting language Area Curve Touchscreen Namespace Cross-platform Feedback Electronic mailing list Bit Type theory Category of being Data mining Curvature Process (computing) Hash function Phase transition Volumenvisualisierung output MiniDisc Right angle File viewer Quicksort Writing Resultant Asynchronous Transfer Mode Point (geometry) Server (computing) Service (economics) GUI widget Computer file Real number Motion capture Online help Mass Continuous integration Login Event horizon Number Latent heat Crash (computing) Causality Bridging (networking) Term (mathematics) Average Operator (mathematics) Software testing Code refactoring Computing platform Address space Pairwise comparison Graph (mathematics) Interface (computing) Surface Debugger Total S.A. Line (geometry) Cartesian coordinate system Limit (category theory) Computer animation Visualization (computer graphics) Personal digital assistant Speech synthesis Video game Object (grammar) Communications protocol Window
Point (geometry) Arithmetic mean Computer animation Multiplication sign Decision theory Software testing System call
Point (geometry) Scripting language Confidence interval State of matter Direction (geometry) Multiplication sign Structural load Moment (mathematics) 1 (number) Planning Bit Limit (category theory) Software bug Kernel (computing) Computer animation Personal digital assistant Natural number Software testing Pressure
I wrap add ham ham hammer man com mood Ombu who idra home I rolling along a path in 17 I
and so this this goes to this is to say so my over just a tiny bit bi-gram 1st so I'm man is on 0 I joined the bunch crazy people in 2011 and I were not so far from here on a project that some of you know uh when I joined the highest I had no idea exactly what we what I would be working on but something interesting happened is that and had just been released by just 1st versions in using Qt as the as the as a toolkit and unfortunately the guy who did the water then up a study whom some of you know I actually left the company at that time giving a lot of meaning to the bus factor thing because it was the you guy and nobody else that actually really knew how to handle the UI so uh there was maybe with a problem there so 1st of all just to to make it clear we have 2 big categories of bagging to could be categories of bugs either we have 1st a kernel bugs which typically can affect the loaders and processor modules erasing they're about the disassembly itself and we have another class I would say about which is EU-wide bugs which is everything user interface related happy crashes and things like that and this talk is about and the actual uh the UI that's not doing that what we call the kernel that's so inside not the OS kernel obviously so this is about you that and how we deal with those so do you why when I'm talking about you i is not about pages button is misplaced or anything like that we know they are right this is this is this is about basically rendering pipeline and how the that the distance that use your these actually rendered on a screen that's that's called what's what we call the rendering pipeline um this is about that I was not particularly inclined to go down that road but then again it's not as stupid as positioning buttons so it's actually fairly interesting and anyway we still have bugs coming in and somebody had to take over because well uh the situation was not exactly great especially we had the blanket problem where and when I fixed 1 thing in 1 place it exited the broke something the other place and of course then you rush the place you fix it and brought in the 1st place which you can do that forever if if if that's your thing but it's we decided we would try to do better so I whatever I wanted to do whatever I 1 where a fixed we wanted to make sure that this was fixed properly and did not happen in the future and the best way to do that obviously is not regression tests none regression tests for you however are not exactly trivial given that we actually wanted to have a platform independent and testing framework we obviously want them to ought to be automated and we don't want to have to we change all the tests every time the UI changes a little so we want to be able to access the parts that were interested in for a specific test and and deal with that and also yeah the results must be easily interpretable that's 88 text files uh so that we don't text files where we're interested in and that's good because then it's in it's in our servers and repositories so we know the history of tests we can do really relevant in operations on those that just a quick parenthesis here in my previous job I was working on something and uh literally a team of guys in Atlanta was uh where the HQ what the company was working for was they were literally just recording test you eat because a lot of you I testing is done through what they call you recorders so you just launched a petition Euclidian places and then it does typically screenshot comparisons um up unfortunately the problem the product was under heavy development which means 2 guys literally and I'm not making this up they were just re recording all the test every week that's all they were doing and the where a small team at X rays and that we we um we we don't want that and in in fact I don't think anybody should ever ever want that and this is indecent and job in my opinion but anyway so we we investigated a few options to 2 to 4 for tools that would help us achieve that goal we we set out to achieve the have a moment all of those that that the 3 that I used here 1 was less brain that than the other uh off the hook up out of the box it was school squished by a company called from logic is scriptable in Python among other things because it the it supports like 6 or 7 languages so you you get to choose whichever you want and the very important part is that it has an easy access to the qt which a tree which is can what we wanted the so there's a lot of buzz going around in squish if you go to their websites you may get notions at the at the at the end of the word salad that the stuff in there uh and um but never mind that duty which of 3 again this is what we wanted and we got it and the time so the but disagree granted that he's test recorder things don't use them and their stupid little stupid even squishes this recorder because they have they have to uh and if you if you start kicking places and they will it will actually have to put sleeps in a in order to make sure that the test will run again just fine and in fact it's not even a guarantee is just hopefully is going to run again on no no no way that this stuff is on this stuff independent of course that is if if you run the test on the west and obviously the widgets would be at different pixel coordinates that uh so my my advice would be just stay away from it it's completely done and another completely that and also a lot of you like testing is done by comparing screenshots which I let you guess if that
works well there is this whole page on the squish our uh the from logic website and explain like what the this is unknown reinforce creative differences obvious reason for speech of differences in other words screenshot comparisons don't work just don't do that because then what they start doing is but the comparison of screenshots and then of course there are some items that you don't want the these the day positives because of the fuzziness so this is exactly what you don't want to be doing in our opinion at least so and
just be very very brief on the uh on the speech architecture the uh it it basically and it runs test Biopython tests in our case we chose Python and it sense uh and it sends commands to a to a to a 3rd party to let's call this which server and that actually is in charge of launching a petition and then dispatch and according to the application um the they did this is not that interesting uh this is typically what happens if you have the test uh scripts on the left then you have the the discretion server and that close his eyes and so if you wait for a special widgets for example is going to say OK wait until this which it is available but is it's visible and active and then it's going to return a pointer so speaking if you call vertical for bore to get the scroll bar on the particle which have been is you send in Vol gets called Bob basically on the on on that which it to the server the server will do its thing in return another pointer and so on so forth so basically it's just it's an RPC mechanism really RPC by wrapping the and that the the Qt interfaces in our particular case so and that's great
uh the way squish itself is built is that when you want to build it from source and you need to provide a cutie uh source tree because what is going to do is that to portal the headers and generate ideas therefore for indefinite interface official language for all of the headers that it encounters and it's just gonna it generates automatically a lot of stubs for you like this the vertical scroll bar method called they showed above it knows that that thing has a vertical scroll bar method at all because of this idea of and therefore you have an easy access to a to the existing qt uh widgets which is great but I'd has a lot of custom widgets and we don't bills which against this in fact needed that that's the way the rule we should have taken that is injecting our own widgets in it before building it's where we chose a different approach the problem is that our on which it so what I call the rendering pipeline typically things such as uh this like this is a summary you uh deep did these these widgets are complete black boxes to switch which means what ever you do um near my doing here right
well and where is gonna where is my mother the the I'm sorry the the 3 treating
like to the right a much more left brilliance I did not see who you are but you will definitely get a free coffee after uh this is really cool ahead sh a date so again whatever operations you want to be doing in our own widgets I will not go back there because otherwise I have to do 3 fingers once again uh this is a course which will be able to do if we record that is to say OK click at this position and then dried for example not gonna work for us again because the font size differences or platform differences a lot of possible all really no you may say about stick to just 1 font size no don't do that because some tests anyway will need to use different font size is precisely because in some cases we wanna check upon sizes were fine so what uh and so uh what we introduced this just things to 1 thing that's called acute object protocol on our own widgets we have just 1 and point of this we call the squish facade of it's just again shorter returning uh um the QT object that is in trouble and that can be introspected and that's squish tend to so it cannot called the original method of or widgets but through this crucial side we don't have bridges basically to our methods that means now we have an entry point to calling stuff in our own widgets even though speech itself is not known actively about our own widgets and that's actually pretty cool and this is this is the uh this is this is the 1st other the over a type of which we call the customer and basically has this and uh in this it has 1 property like the renderer type whether in the flat model graph view for example and the and you as you can see that the the slot there what they call a slot is something introspective all uh and the differentia type would just return resource renderer type in fact so it's just literally just a bridge it's usually what it does is just it's just a trampoline so to speak and so this is a somewhat I move everything uh from the customer just because initially I started putting those discoverable properties and slots on our widgets with the quickly became a mass so that's why I extracted them into a facade that I could just retreat and work on them uh and so you have you have something like this on a widget now we define a public schools for property which expands to blah not so interesting but but it's basically just a bag of properties and accesses right um sort of a namespace if you will and we can call that could that if you want and they just do worry them very little they just query so there's just 1 hour of all use of squish with our own widgets and once you have that you would typically called it like this so you would wait for the for for the for the for for a specific kind of you in this case I have you ways of is something you and then you would you would retrieve the switch sad then call on this switch out and that cause into our application and retrieves our own custom widget specific stuff the uh and then of course you wrap it up into more or less well-defined API is because we don't exactly know where we're going 1st so are at the eyes of a are a bit of a mess in terms of coherency that doesn't really matter what matters is that they work so uh things acute object protocol this kind of thing is feasible and this is really really really cool so then we started writing tests that we wanted to have a pretty small scripts because otherwise it's down uh and we want the results to be of text files just as easy to to to to stuff into an ASEAN and then do comparisons over and uh this is an example tests for example the the 1st AI system areas but then I call on the viewer to say OK go flat mode then jump to a certain uh address and then go up and then go to graph and then make sure that the the current IEEE makes sense for example right because there might have been a problem there no this is a this was extracted from a test um so I it's that out at the at the end which is typically what will be end up into the listing file that we will actually submitted for so that would be the test results basically have so a few numbers uh and so we we started actually writing stuff in April 2013 and we've been added since then uh at the time of this writing which is the this is a small presentation idea some time ago actually um there were 400 test scripts but uh in all tests the descriptors are basically themes and in all of those tests we have multiple what what what I call phases so if you take into consideration all of the phases of all the test scripts religion to test for thousands of things which is not too bad and and get some pictures some numbers rather this this as you may guess is that and that the curve of the tests as we we started writing and so as you can see in the beginning we had a bit of a hard time setting it up setting it all up way with question everything so finding our Morris and then it started at some points in a in a in a note 2013 by the end of 2013 and then we have a huge curve going up there uh this is basic in these venues should lot of time writing tests and and then that fortunately and and the curve soft and some of its which means that paid there are probably less but now right and 1 thing that's probably worth mentioning is that around that time around here there was and this is a little bit of an achievement I'm proud of there was a massive refactoring in ITER between OK so that the red line here is the ITER really is
um 6 6 . 8 service back and the other red line is either 6 9 and in between there was a massive refactoring the why and when I say massive it's a massive and thanks to the test we hardly noticed real world problems 1 at during our beta testing and even more so after the release so it actually works right it's it's announcing it's a significant amount of work but it definitely is worth it in my opinion at least and I believe I'm not the only 1 thing in that so uh the typical the the the typical average output looks size again this must be readable by a human and the comparisons with the physical so we end up with logs that on overage are 1 that 1 about 200 lines the average test sizes to be if 74 alliance perscript which is not big and they work well I mean that is and they work use across platforms we have less of stability on our stand and windows in fact the my my platform of choice thing and this is where I focus the most and this is where tests are by far the most stable and the fastest and as he was he II getting at I have a certain advantage of by working on this and on this 1 but right so difficulties and I believe that you like this thing is actually fairly hard and other companies on even bother because it's it's pretty hard but I'd has a fairly large surface of actions and if you don't test that it's easy to not even notice that you broke something and then the reuse goes other people saw complaining because I am using that all the time now is broken yes sorry we denote so another companies I think that even bother and if they do they do it in the way that the recorded tests and quite frankly no uh um yeah so we have a it is the 1st we have squish and as the as I said I I introduce the notion of discrete facile over on just images and then on top of that we want above that we had to slap our own scripts helpers so as it this is this is the total amount of lines just will have a person that is all the tests were to be useful helpers I have about 11 thousand lines right there and just to have a cross platform kind of user friendly interface for writing tests and yet so that the tests are pretty darn stable 1 minutes and as I said last on the windows and and but they do run and a colleague of mine Troy uh added recently bench tests for I as the beginning and obviously those must be run from my platform because Apple uh so let us as a reference platform has 1 very very great advantages eggs with the suggested by muckily Ramiro here uh and EVA them if you know what it is but it's basic an X server and therefore the limits of visual interface um except it's in memory and it the inputs and outputs which is cool because then we can actually run the test disk which tests actually require input and output that is the new the mouse they will literally going and click on buttons and things so I um that's pretty annoying when you're working in you run a test and you have to have the everything happening on your screen this is a little bit of upsetting so uh exhaustive B is is super because not only does it let you run tests in its review and therefore does not interfere with your work that's the 1st bombers but you can parallelize it so now we run the test you by this of 16 and their ego so uh at the time we have a right now we have maybe a little less than 500 tests and run them all takes about half an hour I would say which is not that big a deal and we get a sense of usage and continuous integration now driven by a test driver I will mention later uh we did results every now and then I like multiple times a day of justly uh saying that hey this just broad and it's it's a great help and we can find any way to have an equivalent on Windows and on the west and we have a kind of a clutch analyst and it didn't really work uh but uh wrote running running on Exif the emperor lies and then just made our life so much better and so yeah uh this T 2 fingers I should have just driver uh which helps run a lot of things including the UI tests but not only um and and with this test driver I have added a few things such as the the flag look for trouble which means that we are underspecified tests and endlessly until it finds the failure or a crash or any any kind of trouble really uh that's J is the casinos as would make some multiple processes uh video captured that's cool because when you run an exhaustive B and something goes wrong high-volume have visual feedback but now with visual capture using ffmpeg fully capture and player typically for the playback I can actually in C frame-by-frame what's going on and that that is super helpful and then there is also a way for debugger which is very very cool because whenever um whenever something goes on at Disney ability together with the uh with that and with x of the Disney the ability to launch and lastly the tests and as soon as I starts the plots the gdb into it and and and so we would specific commands so that if the test crash well it stays right there but if the test succeeds and I DAGs it's gdb except as well so it's automated that again in UI testing because it's events base their thing some the hashes are very very very hard to reproduce that tends to this uh and I mean I could for example and opening engineer type this is
something that happened so I I I ran this particular test call tracing Mr. uh looking for trouble any and sj 1 meaning in its with the decision of the end but still and that then does the value which is short for wait for the better and so the test as you can see ran quite a few times here and then at some point eventually they receive the sex succeed and I have a backtrace uh proper badges and I don I should investigate all the stuff that he's um so that's like
massively helpful I mean yeah so the future directions of ego or Michael it's suggested may be doing some kind of funding but it needs to be heavily directed because otherwise is just a waste of time but I believe that the ones who are a little bit more comfortable with all that stuff uh yeah it's definitely definitely possible uh so yeah whenever you reported by depending on the nature of bugs is as I said in the beginning you will fall into 2 big buckets either it's what we call that kernel but it is to be deposited module unloaders known stuff what analyzes or to you in which case 1 of us uh will actually add UI test for its 1st before fixing it obviously uh and and yet you'd be surprised by the amount of things that we can test there are some limitations to you Whiteson but usually it's incredible the amount of things that we can test already and that we have we we have this it'll don't have a 100 per cent coverage but it gets better every day literally and uh this is uh this is very very helpful and and we have so much more confidence in what we're doing so um yes thank you very much for his and his the latter half lives there is again a
questions they have to answer them the the thank you little uh do you have any um API is our example so that the individuals could user this make this now we haven't published those uh as I said uh briefly before its time interval the little bit organically it's a bit of a mess they're not pretty they're not uh and at the moment we don't really have plans to publish them uh but uh maybe I mean it's among other things because we don't know if people would be interested so thank you for suggesting it I don't know firm if the pressure becomes unbearable we were just really is that at some point but uh there is no there is no plan at the moment and as I said it relies on it it yet relies on the other on a boatload of scripts that are in a state that I mean I wouldn't show my mother so um yeah thank thank you the no thank you yes