Automated Testing Laboratory for Embedded Linux Distributions

Video thumbnail (Frame 0) Video thumbnail (Frame 1608) Video thumbnail (Frame 2449) Video thumbnail (Frame 4959) Video thumbnail (Frame 8870) Video thumbnail (Frame 9792) Video thumbnail (Frame 11181) Video thumbnail (Frame 13618) Video thumbnail (Frame 14615) Video thumbnail (Frame 16215) Video thumbnail (Frame 21277) Video thumbnail (Frame 24270) Video thumbnail (Frame 26856) Video thumbnail (Frame 30192) Video thumbnail (Frame 33151) Video thumbnail (Frame 35740) Video thumbnail (Frame 39409) Video thumbnail (Frame 43612) Video thumbnail (Frame 45743) Video thumbnail (Frame 47993) Video thumbnail (Frame 51979) Video thumbnail (Frame 54532) Video thumbnail (Frame 55620) Video thumbnail (Frame 61544) Video thumbnail (Frame 62963)
Video in TIB AV-Portal: Automated Testing Laboratory for Embedded Linux Distributions

Formal Metadata

Automated Testing Laboratory for Embedded Linux Distributions
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Shipping quality software always involves in-depth testing. In order to minimize time spent on repetitive actions, this task should be fully automated. During this talk Paweł will discuss key problems faced while building automatic testing infrastructure for Tizen operating system images. He will also present how Tizen release team dealt with this task. Both hardware and software presented in this talk is open-sourced and can be easily used to build embedded software testing lab. Paweł Wieczorek
Ocean current Default (computer science) Distribution (mathematics) Process (computing) Computer animation Profil (magazine) Code Projective plane Mereology
Distribution (mathematics) Group action Computer animation Computer hardware Execution unit Phase transition Shared memory Virtual machine Single-board computer Software testing Resultant
Key (cryptography) Projective plane Bit Water vapor TDMA System call Revision control Arithmetic mean Computer animation Computer hardware Cuboid Software testing Whiteboard Computer architecture
Computer animation Plastikkarte Software testing
Distribution (mathematics) Server (computing) Game controller Service (economics) Information INTEGRAL Software developer Patch (Unix) Revision control Medical imaging Mathematics Computer animation Software Repository (publishing) Software testing
Server (computing) Mathematics Service (economics) Computer animation Linear regression Order (biology) Software testing
Multiplication Server (computing) Distribution (mathematics) Information Linear regression Multiplication sign 1 (number) Variance Sound effect Set (mathematics) Volume (thermodynamics) Price index Shareware Medical imaging Mathematics Inclusion map Process (computing) Computer animation Software Personal digital assistant Different (Kate Ryan album) Order (biology) Software testing Resultant
Server (computing) Service (economics) Forcing (mathematics) Virtual machine Bit Disk read-and-write head Event horizon Medical imaging Mathematics Process (computing) Computer animation Software Personal digital assistant Software testing Resultant Task (computing)
Server (computing) Scheduling (computing) Service (economics) Computer file Archaeological field survey Heat transfer Event horizon Revision control Medical imaging Mechanism design Bit rate Software testing Task (computing) Computer architecture Default (computer science) Multiplication Inheritance (object-oriented programming) Software developer Variance Connected space Digital rights management Process (computing) Computer animation Software Personal digital assistant Telecommunication Order (biology) Video game console Communications protocol
Server (computing) Statistics Distribution (mathematics) Service (economics) Information Multiplication sign Interface (computing) Virtual machine Set (mathematics) Configuration management Mathematics Voting Computer animation Strategy game Integrated development environment Robotics Network topology Cuboid Configuration space Software testing Resultant Reading (process) Social class
Ocean current Standard deviation Key (cryptography) Software developer Flash memory Interactive television Shared memory Set (mathematics) Single-board computer Medical imaging Computer animation Semiconductor memory Different (Kate Ryan album) Whiteboard Procedural programming Computing platform Computer architecture
Scripting language Game controller Projective plane Memory card Virtual machine Plastikkarte Connected space Revision control Film editing Computer animation Semiconductor memory Computer hardware Whiteboard Task (computing)
Ocean current Medical imaging Twin prime Server (computing) Computer animation Memory card Computer hardware Thermal expansion Whiteboard Resultant Number
Server (computing) Information Open source Structural load Electronic mailing list Counting Planning Basis <Mathematik> Total S.A. Mereology Limit (category theory) Number Medical imaging Digital rights management Computer animation Software Different (Kate Ryan album) Profil (magazine) Order (biology) Cuboid Software testing Condition number Physical system
Distribution (mathematics) Scheduling (computing) Multiplication sign Shared memory Lattice (order) Limit (category theory) Configuration management Process (computing) Computer animation Repository (publishing) Computer hardware Renewal theory Task (computing)
Group action Service (economics) Multiplication sign Execution unit Set (mathematics) Online help Theory Different (Kate Ryan album) Ontology Software testing Physical system Focus (optics) Information Physical law Projective plane Lattice (order) System call Word Kernel (computing) Computer animation Grand Unified Theory Addressing mode Tower Personal digital assistant Whiteboard Resultant Spacetime
Computer animation Information Link (knot theory) Physical law Content (media) Whiteboard Power (physics) Wave packet
Computer animation
we but I'm sorry for the delay a article everybody my name is part of a toric I
work at some some current Dean Institute bond that currently I am involved in release engineering process for pleasant things distribution created specially for embedded Linux devices I work with that of of the common profile which is targeted at the only and its target is still provides and simple real distribution for as many target devices as possible today I would like to talk to you about and our automatic distinct laboratory the project which we started to almost a year ago and so the code defaults what it is that right now I will start
with a short introduction of our solution then I will describe the to what drove us to 0 creating such a laboratory the are also presented the what we did optimize and how we do that the and then I'll share uh you our to the least uh what is still had to be done and I will uh and this with a short summary and conclusions but
so we have to introduce and our solution this is how it the hold the basic unit of our automated testing laboratory looks like the it consists of multiple and a single board computers which we use as target devices for ties in common and the distribution the it also has some had supporting them the other which I will describe in depth later as a casino all the has this is a single-board computers are and in and groups also if you are the multiple mainly because the 2 have them redundant if any hardware failure occurs there's always a possibility that the other 1 and is still working also it allows us to allow the perform all the tests in parallel tool may even faster to get the results faster and also if uh there is a failure in tests we are always read over and on the other hand as machine just make sure that the In the test phase where wasn't and the of but it's is that it's not the hardware so fault unfortunately the wants to the whole unit of testing laboratory that was hooked up it doesn't look that's pretty and
it's loose and more like this uh 1 so we put all the cables in in the right places but there for now let's get back to the pretty picture for
and let me describe and that's what we've got here no told test on Intel-based architectures we use the mean water to a box that back when we started the project we use to mean award Marx's which are almost in any way compatible uh between the school no versions and they are compatible with the the the mounting calls with the and imports so the placement and so of course uh with the architecture we test on they also allow us to will perform tests on both the uh 64 bits in the architecture as and thirtytwo its is changing it is this is the as replacing a unified the tool test on the on the 7 architecture we useful droits and you'll 3 plus that they are pretty common unfortunately no longer manufactured but the we've got a few pieces and so there small and they have also for the commonly used sports and they perform and greats in our testing laboratory the as I mentioned before uh our target is tool provide support for as many architectures as possible so we also added on 64 target devices and so they currently height keys are in our the reference devices for an 64 we but as for supporting hardware we of course go to and USB hub so these are just stock and bonds and the to to fall and the whole set up up also on the top and the bottom we've got 6 the boards of our custom-designed these are at a micro as the current demultiplexers that the and so I will describe them in depth uh shortly In this
devices up close look like this and allow us to share and access to the micro the cards between testing costs and the target device being on the test at yes the well
assume and all and how it's it looks like let me tell you what drove us to uh creating such and such
solution and so too will make it entirely clear I would like to
describe to you shortly how does that really engineer uh so work look like at the tightening of distribution so 1st of all and software developer that constitutes represent 3 from our and carrot the developer makes changes and pushes them back to to a server for of the review once the review is dominant the reviewer uh both reviewer and to develop a is happy are happy with a given patch the uh integrate or uh takes his overall entered marriages and given patch to of the repository and that's not all of that integrators and work is all about because then integrator
should create a submitter request which is then processed by ASR by release engineers the submit requests and is the and any information that we should include a new version of software and in releases of ties distribution for all the target devices and such submit to request is then processed by our drinking server the the to the open build service for ties and also our G where all the software that dispatched and not only it but also all its dependencies are being revealed a and so the him packages are created once this is done a new image uh although the distribution is created and and and is available for further testing and this is where release engineers and take a control and this is where we have but it starts in our testing and the as you can see most of the work workflow is already and fully automated and ingenuous server and obvious that work a fully independently the images that are being produced and they required the some manual work in
order tool be sure the new change but also submitted there can be accepted or or if it should be rejected and because it's the we introduce some regression or tossing clean fails to build and for for a given release the Our work is based mainly on on tool and pulls uh which I already mentioned on an open do services server and on Jenkins and Our of
automated testing laboratory had to be and created and based on those tool technologies which tried to alternative solutions above uh these uh worked best so far that they are well known they opted to stable and the various you most of our needs that's how we use
them and 1st of all if the obvious the shows up the variance of failure which should be investigated and to check for possible solution and we also check uh if new images work properly on all kinds of devices or whether it's introduce some regressions also the ones so we can gather all the information the without my approved inclusion of the effect changes but the this takes
a lot of times when 1 when it is only when it is done by hand by release engineers and it's mainly because the volume of the multiplicity of target devices that we test on and we currently tests only on a single example also given architecture and since we test on 4 of them there are already already have for can before past that each new or changed has so in the past in order to be included in the present release as you can imagine this workers pretty monotonous and it requires repeating over and over again in the same set of instructions to the same set of commands to be executed but still it requires a lot of Fox because all the now test results look to pretty similar and today it acquires a good I still captures some differences between them so we ask ourselves there maybe Inc not all the images created on submit requests should being uh tested as thoroughly or maybe just a few test cases and have to be checked on each 1 the and we also uh briefly considered the the assumption that there may be in the software builds and it's it's pretty good indicator of that it's that it's that works properly unfortunately this is not how we would like to create a Linux distributions and especially for embedded devices that we always try to all results and all the defects and that we find as soon as possible and it is much harder to find the reason of such failure and if we checked it might have on the occasions and we also can tool to look for a solution not just for the foreground and workarounds from our experiences do but fire and so this is this is not the way we have try to create a coming of distribution the and most importantly uh we do not release software which was never tested on the actual device on the and images that are publicly available at the our demo server there are some who we're being flashed onto a target devices and our safety to use so the only way tool made to this whole process uh easier for release engineers and tool uh release actually from all these mine your work was to optimize the
whole process and that's what that's actually what would it be the the
what was the be automated could be categorized as
in in uh and fuel 1st of all we had tool for poll and servers and for and any change unfortunately the obvious server does not provide any events the system and for for us to be notified was change occurs we also have to and actually optimize getting those images from the service but the the software uh tasks were pretty is imprescriptible so it didn't have to take much time as for controlling uh all the horse head machine and target devices and this is required a little bit more what but the moral that's very shortly as for publishing test results which we didn't want to keep for ourselves not that we want to them to be shared with all those uh in case someone does not believe that our the results and wants to reproduce them the we decided that they should be publicly available to everyone and so the probably the hardest in task force was tool provide a unified way for a flashing target devices but with the new images prior to their publication and the and as I
mentioned before the obvious unfortunately In the version that is being deployed on the ties and also servers and does not provide any events and so because of that because of its currently and is not best suited for uh fully automated so that mechanism it's it provides a more human readable naming conventions which have to be passed in order uh uh for a full automation of the process then what is more yeah discovery of all the new images have to be made on multiple levels as and currently images are not published all in 1 place but are categorized them through and architectures and this is the manager and the variance and target devices and so on fortunately with the technology that we already and now and so that in most cases like we could make Jenkins a tool to be able to work for us hence the tasks as such as scheduling for our queuing new new test was already made for us by the Jenkinson the as
for having uh to communicate with all the devices we consider there are several possibilities 1st of all we thought about SSH because it's the in it is already 10 over popular it provides a and many other features and it is a flexible solution however it depends on multiple services and networking and for example at and 2 we didn't want such a dependency therefore our communication funded devices we wanted to as little the as possible and since some of target devices do not provide an annotation port are but some of them are only equipped with a wireless communication and we felt that it would be better to find another solution we also considered the serial console the the which is the most popular at embedded development however in the rate for that up transfer also to all fall for our needs and parents unfortunately this solution is not as flexible as others the since neither all of them could be chosen uh we decided to fall back to the default uh communication and protocol used in ties and development which is smart to debug breach and the then or as the B for sure as the B and provides us with the best of both worlds and if the requires only a USB connection and 8 depends on single surveys which is already uh available at all dies and images and it is so pretty flexible and did provide social access to a remote command execution and file transfer the the but
once we have the votes in place but we also took care about the configuration management for our test service for our host machines we started with uh just providing a handbook in Tesla and will propose a tree box and here we soon found out that that this was not enough uh changes might still the a server the wearing a we're putting much faster pace uh that we could updates our handbook this is why we decided to provide danceable that playbooks for storing consideration and also our test services and in the slot horse addressed this way and we were able to improve our deployments to have robot strategy and and the since it is on defined and providing new nodes until and laboratory there is much faster and much easier and much more reliable reliable than repeating set of instructions over and over again in this way we also a goal to reach all servers which were configured by some person and the had some in different In 1 set of configurations and and 2 that's how uh snowflakes which should it is the name for those services and most common where removed from the hours of not having or internal infrastructure and place wasn't the last class we also had to work with our
external infrastructure on ties and all technologies and services mostly we wanted to use it for the publishing the and so it seems that we King and is already available at certain times in the foraging with side of that 8 7 the might become a place for publishing our test results it is easily available tool everyone can it provides the possibility of future use of the test results we provide and since we didn't want to introduce that any new services therefore are sharing our results uh would so that fit into the way to go In this way we can and provides information about our environment and so what we actually did with everyone and that all the test results have the same set placed will be published and we also hope that in this and it can be used in future for example for and statistical analysis on hold the in of distribution was improved uh to do so we use that as great tool provided by which is headed by which a bolt by Wikibooks provides easy the interface uh in Python for both reading and writing to uh to lead in which a server and
the last and we also have to to overcome the problem of
flashing and target devices and so it was not the entities you on but unfortunately the current wastes current set of tools uh which is used as a tool of flash new images on and target devices that is focused on user interaction it is also designed therefore uh not for platform developers but rather for other developers and to it assumes that I'm developer only require a single and bought and he set up so it is not a and not well-suited for having multiple devices connected all at once the complete also that the procedure in the for flashing devices is pretty architecture-specific there are different sets of and tools of commands to execute their for each of the architecture we test on this is why we had decided that it should be uh the way of flashing devices should be unified once and for all 4 every architecture we support and currently and so that we have found out that when all these little hand devices the share the unique feature but they are a bootable from uh as micros the cuts and so it is a currently and I believe that all of the a standard since not only and devices I I mentioned before and high keys or drawings and certainly no boards our and suited to do and many of the more popular platforms such as the president Pi smaller and then but in different examples also all draw its and other single-board computers also use in this this standard full 10 memory access this is why we
came up with our design for my credit card the multiplexer and so what you will see here is actually but our 2nd attempt that's creating such device this but for the 1st 1 there was a little bigger in the first one that we have thought that and it was only in the beginning of the project and we saw that there may be as the the regular the uh cards will be better standard but we soon found out that it is a dead end and that we if we wanted to be as close a total of the actual hardware that will be the used in diving devices we have to switch to a miniaturized version of the markers the cuts the for but and in design and as the marks provides the but but for and many of connections 1st of all bought control which we use to and and to manipulate with the board itself then the it was since we have already and it's more that and based on USB connections inside that it is the right way to go for as the marks as well in and then of course we got the reader for memory cards and we use actual memory cards not some uh flashdrive builds Spontal but as the marks board but actually memory cut it to the proved itself to be useful essays in we can now measure and where on a memory devices and so this way we are as close to the actual hardware as possible and of course there are also connections uh to the target device for and about breach and tool a connect that the and memory to all target and so and the same for the host machine people that but 10 for there is also and that it has the power switch tools and be able to remotely turned off or turned on and target device a together with our micro as the card to demultiplexed there comes with a
little uh show tool which we use in our scripts and provides the access to all the time and and on the features of offered as the marks board and but that tasks or which we can put it the on this device the
this way to our former work for which still require it among your work from release engineer and could be uh fully automated there we know longer had tool uh number of images and at at then ample them on like in memory cards for the cars back to the target devices from the desk other results and compare them Little is said however places that are the this little board of our design allowed us and to make this whole workflow a fully automatic the twin sided that since it's provided so such an improvement in our workflow maybe it could be useful for others as well and this is why to schematics
therefore the boards and for the expansion of the course to collect to but the current solution actual hardware they are all available at server at Tyson adults or G the we also thought that's the
all proprietary software and should being and necessary tool to access a little schematics this is why we provide them and in the to the informants readable by an open source software count for designing and the 2nd box the all all the information and all these cannot expenses and some to condition is available at the gets server at the ties and total the last we got it said on the basis and we when we began a tool think what should be done next and the since we are part of the the whole workflow therefore the selected there we
decided that we should focus on having and the handled on our set up and then in a proper way and we decided that since we expect already at a higher load on our test servers we should also provide a solution for determining which images uh will not be useful for us to reject them as soon as possible to and we also known as there no needs and more detailed way also determining the differences between the tested images unfortunately not just the difference and returned from packages and from the list of packages installed onto the uh image for certain kinds of is not enough then and so having just the name and passion from the server that is not really and best solution to compare and all images then so there is a need for improving in this can this matter but also will we plan tool improve profile failure management we want to get as much not as possible even from affairs test and what is more and entities connected with it we intense tool have a better management over our resources over our and target devices since it is so limited and number this is there's a limited number of targets we test on we also known currently I think of distributing uh this system and in order to will be able total and that's the only way much more and target devices on other examples for all of those architectures and so the not to be required to but all the test in a single laboratory the the but this is still yet to come yeah and to
summarize uh shortly what we've been working on I would like to
share with you that there are currently no automation and does not need the new is there is no real there is no need for reinventing them uh with the currently available tools such as tools for configuration management for and and advanced scheduling and queuing tasks or for Ed 2 the having of the packages for uh Linux distribution repository repealed the there is not much need for for renewal that tools what is more then and forced limitations which the currently uh we stumbled upon in an external infrastructure and I can be quickly and easily overcome with some short scripts in and so and custom hardware which we were which we are not convinced that we will prove itself and useful a longer on uh can be in designing such custom hardware can be intimidating but the it's the pays off in a long time just like ultimate meeting and manual processes
that that is all about what I've got for you today if you have guts and question I'm more than happy to answer them at the the of at the the researchers of the early peas in the bulls or you could have led to the group so you just do the kind took about any other related parties have 1 1 in mind which is just I remember going to tool to be difficult lost you all know where they had to look out that's place on out where they they had looking units the laws of answers and so on we have found
on words please let me know house done and the whole word as a meeting you not concealed title and and so is 1 of the come across the reasons and we will work with you with you to that was the Unites States agree with all of these and so would you like to and help Tolman differences between kernel CI and our solution our just uh in this year I think it's just you just you just let it be the outlines of these approach 1 so that the you uh so on as for custom-designed also the board thing I wanted to a different approach that tool to share their own memory and between the target device and the costs of they required to different connections and on the only collection and they uh took into account was different that was not the case for us that we decided that simpler a solution in the minds of being easier to use as for kernel CI and it is focused on on the set of tests than we are a provides a book tests the answers are not the test of all the user space of the system the seal services the tower had been around that that for which we currently don't and kernel CIA also focuses on and said that all the time and ontology it focuses on gathering results from many different laboratories uh whereas we focus on creating such laboratory full for our purposes and the and their social and is is great it provides much useful not much useful the information the I hope that our distributed and erotic what would be as helpful as but our key focus that is far from this theory and I am not so sure if you're aware but there are the to all and 2 more then projects which currently involved automated testing on embedded systems uh 1 is for the automotive great Linux the Eigil for short and the 1 is the the then it was once the call to DDA Jenkins test automation and if I recall correctly it was recently renamed tool fled on the if it if it brings about there were way and compare our solution to those as well ants and we hope that this will be reliable we will be able to come up with a thank you maybe more standardized solution the tool to combine all the needs that in in the in the project that but but a case of someone who would like to the
contact for further information all this law an a more
links tool to ourselves forward all tool schematics but also the power uh boards and he is not a MRSA for me that I'm afraid I won't be able to answer detailed questions about the current design of our and board that but I will be able to to point you to the actual designer thank you for a contention and so on training however and goods and pretend the whole the I think