Test Software On Emulated Hardware In Containers... In The Cloud
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47266 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Software testingEmulatorSoftwareComputer hardwareComputer hardwareBitSoftwareSoftware testingComputer animationJSON
00:32
Software testingSoftwareComputer hardwareComa BerenicesContext awarenessEnterprise architectureComputer fontSoftware testingEnterprise architectureMultiplication signLevel (video gaming)Computer animation
01:08
Focus (optics)Artistic renderingInformationPhysical systemDigital rights managementInformationOperator (mathematics)AreaWave packetServer (computing)Digital rights managementOpen sourceType theoryPhysical systemElectronic visual displayAlgebraic varietyComputer animation
02:32
Physical systemSystem programmingInformationOpen sourceSingle-precision floating-point formatData typeEnterprise architecturePublic domainOperating systemInformationComputer hardwareSoftwareOpen sourceWave packetSingle-precision floating-point formatService-oriented architecturePublic domainQueue (abstract data type)Algebraic varietyPhysical systemBlock (periodic table)TrailOperator (mathematics)Scheduling (computing)Information systemsPlanningRevision controlDataflowClient (computing)Analytic continuationMereologyWorkstation <Musikinstrument>Variety (linguistics)Instance (computer science)Interface (computing)Different (Kate Ryan album)Type theoryOrder (biology)BuildingEnterprise architectureSoftware testingBit1 (number)RootFront and back endsWaveComputer animation
06:43
NumberSoftware testingDisintegrationSoftware testingRow (database)Order (biology)DataflowPoint cloudPhysical systemEnterprise architectureDigital rights managementReal numberPoint (geometry)Client (computing)CompilerExecution unitSoftware bugLimit setUnit testingSoftware developerINTEGRALMultiplication signPublic domainOperator (mathematics)Computer animationDiagram
08:36
Software testingVariety (linguistics)Computer hardwarePeripheralMedical imagingBefehlsprozessorForm (programming)1 (number)Different (Kate Ryan album)Function (mathematics)Power (physics)Variety (linguistics)Topological vector spaceCombinational logicSerial portSoftware bugElectronic visual displayOrder (biology)PeripheralVirtual machineNumberOpen sourceSoftwareStack (abstract data type)Software developerPhysical systemInterface (computing)Real numberComputer fileLine (geometry)Computer hardwareMechanism designBootingPixelWave packetReverse engineeringGraphics processing unitMulti-core processorClosed setArithmetic meanComputer animation
12:56
Bus (computing)Computer configurationService (economics)Different (Kate Ryan album)Game controllerComputerComputer hardwareSemiconductor memoryMereologyLatent heatOpen sourceTable (information)Electronic visual displayConnected spaceOrder (biology)MotherboardSerial portComputer animation
14:05
Computer hardwareMiniDiscComputer-generated imageryState of matterSoftware testingComputer hardwareClient (computing)Service-oriented architectureInterface (computing)MiddlewareInstallation artMedical imagingMiniDiscMultiplication signNetwork socketStructural loadMessage passingStress (mechanics)Communications protocolIdentifiabilityFront and back endsAlgebraic varietyLink (knot theory)SharewareContext awarenessDirection (geometry)Physical systemMereologyBootingCartesian coordinate systemVirtual machineGame controllerCASE <Informatik>Computer animation
17:32
SharewareTemplate (C++)View (database)Computer filePresentation of a groupSlide ruleVideoconferencing2 (number)Computer animation
18:03
Slide ruleView (database)Template (C++)VideoconferencingMaß <Mathematik>Reading (process)Computer fileLattice (order)Medical imagingMiniDiscPerfect groupComputer animation
18:29
Slide ruleTemplate (C++)VideoconferencingView (database)Computer fileReading (process)Physical systemServer (computing)Network socketSerial portCache (computing)BootingComputer animation
18:58
Socket-SchnittstelleProxy serverServer (computing)Uniform resource locatorPhysical systemCommunications protocolRootComputer fileView (database)Scale (map)Computer hardwareSoftware testingNetwork socketSerial portModule (mathematics)Source codeComputer animation
19:27
View (database)Computer fileServer (computing)Network socketCommunications protocolModule (mathematics)RootUniform resource locatorNavigationWindowGoogolOrdinary differential equationSoftwareSoftware testingComputer hardwareWeb browserGraphical user interfaceFirefox <Programm>Google OfficeLocal ringSlide ruleOffice suiteGoogle ChromeSpecial unitary groupLoginACIDKernel (computing)BootingComputer animation
20:06
Local ringComputer clusterComputer fileView (database)ACIDParameter (computer programming)Kernel (computing)LoginDirectory serviceSystem programmingSystem administratorInformationNavigationPhysical systemMessage passingServer (computing)RootBootingServer (computing)Function (mathematics)Service (economics)Computer animation
20:54
GoogolComputer fileServer (computing)View (database)Parameter (computer programming)Kernel (computing)LoginDirectory serviceComputer networkWeb browserNavigationBit rateService (economics)Computer animation
22:03
Codierung <Programmierung>NavigationView (database)Computer fileServer (computing)Web browserLocal ringError messageError messageComputer programSoftware testingComputer animation
22:34
Local ringView (database)Computer fileWeb browserCodierung <Programmierung>NavigationFirefox <Programm>Slide ruleGoogolScale (map)Software testingComputer hardwareConfiguration spaceSimulationComponent-based software engineeringData typeLattice (order)MathematicsInstance (computer science)Physical systemMetric systemComputer animation
23:03
Error messageElectronic visual displayScale (map)Software testingComputer hardwareSimulationComponent-based software engineeringConfiguration spaceComputer-generated imageryMiniDiscSpecial unitary groupSoftwareRoundness (object)Service (economics)Computer animation
23:50
Computer hardwareSoftware testingSimulationComponent-based software engineeringScale (map)Configuration spaceAlgebraic varietyInternetworkingSoftware testingSoftware developerRepository (publishing)CodeOffice suiteConfiguration spaceMoment (mathematics)Different (Kate Ryan album)Point (geometry)Client (computing)MetadataHookingVariety (linguistics)Computer hardwareSampling (statistics)CASE <Informatik>Multiplication signCycle (graph theory)Right angleoutputPhysical systemUtility softwareCore dumpComputer configurationMedical imagingRoutingBuildingSoftware development kitModel theoryRange (statistics)PlastikkarteInformation securitySoftwareReverse engineeringDecision theoryLibrary (computing)Operating systemStreamlines, streaklines, and pathlinesMulti-core processorGraphics processing unitEnterprise architectureVirtual machinePower (physics)Video gameView (database)MereologyInstallation artWorkstation <Musikinstrument>
31:21
Video gameChainVirtual machineRoutingVariety (linguistics)Metric systemMultiplication signMereologyCache (computing)Field (computer science)Latent heatMedical imagingComputer hardwareComputer animation
33:54
Point cloudFacebookOpen source
Transcript: English(auto-generated)
00:06
Why not? This is surprisingly crowded. We were originally told that there might be like one, two, three people, of course a little bit more. Okay. Hey, this is Sean Parker and I'm Paul Schroeder
00:24
and we're here and tell you something about testing software on emulated hardware in containers, in the cloud, and I hope you all like that. We think it's quite fancy.
00:42
We tell you something about the project, why we're doing this. It's high level architecture. It's high level architecture. The test pyramid where our stuff is located in the test pyramid about the target devices and the challenges related to them.
01:02
A little demo and in the end, Q and A and recap if there's still time left for. So we're working for an international train company and it's not about cargo, it's about travel information like the thing below and you might all have experienced the delays
01:23
and so like five minutes delay on the display and five minutes later it's 10 minutes delay and 10 minutes later it's 15 minutes delay and the reason for that is the federated decentralized system of servers serving all these displays
01:40
and if a train did not arrive in the visible area of this server, the operator doesn't know when it will arrive and so he's just starting to type five minutes and five minutes later if the train's still not arrived, he's increasing that. So the goal of our project is to centralize that.
02:07
We have a partner project in another city which is doing this consolidation of information. If you have conflicting sources and conflicting information, which source do you trust more and this is a tough problem on its own
02:21
but it's thankfully not ours and we're more focused on devices itself, the management of them and the rendering for the different display types. Yeah and this is where I hand over to Sean. Cool. Yeah so I mean Paul already mentioned
02:42
that like the old system was a federated system where depending on where the information display is or perhaps where the train is, the information might not propagate properly like to the end station. If you have a train starting I mean in, I don't know,
03:00
it's like eight hours early like in a totally different part of the country, the information displays are currently with our client perhaps not timely updated because yeah the flow of information is just not giving that information. So this now is basically a simplified abstract version of what we're doing.
03:21
So basically Paul just mentioned the system that is basically at our other project consolidating the data. So and the planning basically delays and updating the information continuously. So in the background we have something like a, well a Revit and queue broker where you can basically subscribe to a certain train track
03:41
and so forth and then you can basically get the information and you continuously update it. Sounds easy, it's actually quite complex and I mean it's improving from year to year but what we are then doing, I mean the other project is basically the external domain up there. So there's single source of through block
04:00
of travel information, that's basically where all the information goes in. So train delays, when the train start, when the train arrived, scheduling and all that kind of stuff. And it's also basically interfacing in the background with information systems from other countries or other operators. So that's where all the information goes in. And our project is basically interfacing with that
04:21
and you might see train tracks, different devices, basically voice devices. So basically if a voice announcement comes, that's from one of those devices. If you have those displays, LCD of TFT, the old ones and the new ones, that's one of those devices and they might still have a variety of sensors. So you could think of our system
04:41
a little bit like Netflix for travel information. So basically we have a big Microsoft's architecture in the background and depending on demand, I mean you can imagine during the night there is not that much going on. I mean there are cargo trains, there are night trains, but really rather during the rush hour we have like most of the information
05:01
going through our system. So it's growing. It's really pretty much like a sine wave over the day and falling down in the night. Yeah, indeed. So and what we are focusing on, I mean this backend block without getting too much in the nitty gritty parts is already complex enough. So that's the Netflix bit.
05:21
And then still we have quite a lot of different devices we have to interface with. So those devices are quite often more paid by public money so you can just, can't just replace them. So you have to basically retrofit them with yeah a new operating system. That's what we're doing
05:42
because I mean there was no clean standard basically in the past. So for instance we had a lot of different device types with different interfaces and that of course doesn't make it that easy to integrate with them. So we're building a new operating system and hopefully it will work quite well in the future.
06:06
So and that's what this talk is about in its core. So this little OS box, we're building the operating system based on build root. Who knows build root? Quite a few, good, cool. So it's basically a toolbox really an abstract way.
06:22
It's something you can use in order to build a Linux operating system and customize it to your needs. Yeah and we're doing that but that's of course quite difficult because testing hardware or testing software for hardware with different configurations, it gets complex quite quickly due to combinatorics.
06:41
Cool, yeah. So who of you has seen a test permit so far? So probably you have experience in the enterprise domain or somewhere else or you have made some nice PowerPoint slides about that to convince your manager
07:01
that it might be reasonable to write a few tests. Yes, that's a good idea. So integration test and unit test that usually happens with software but at some point you have to integrate it with the real world and that's usually where things blow up and you actually want to find bugs and problems as early as possible because I mean in the beginning
07:22
if you have a unit test, right, it's like you find a bug, it's just like you fix something and recompile and it's done and it's good and you're happy. But if the client finds the bug, it's like you're getting called, operation ops is unhappy, you have to roll out everything, you might have to wait quite a long time in order to get your fix out
07:40
and it's just not making me as a developer happy because basically my development flow is broken but yeah, long story short, you just want to find bugs as early as possible and that's what we're doing with those operating systems which we put in containers in order to run them in the cloud because then we can do that at scale,
08:00
more automated in the CI instead of flashing it or automatically flashing it to a limited set of devices and testing it manually because there we are basically limited to what we have hardware wise and you have to be there physically. The stuff we want to present you
08:29
fits in this column of the, in this row of the test pyramid and the target devices are pretty, it's, they are, there's a huge number of them,
08:42
I guess 6,000 at least, a lot of different types, most of them are the old ones like the LCDs over there are PC 104s, the form factor, but they vary in CPU, in RAM size,
09:00
in CPU types, in the interfaces, you have these LCDs and the TFTs which are like better TVs but they are powerful quad cores with gigabytes of RAM and the outputs are different, we have different boot mechanisms like some devices only support WEFI
09:21
but the older ones only support MBR and our goal is to have like one image that runs on every single device, this is not possible, we have a variety but still we want to have it reduced and not extended to have the main,
09:40
to have it still maintainable and testable. There is different, this is just the devices itself, there's different external peripherals like it's called Sukkalzen 1, train stop sensors, let's go like this, the displays are sometimes controlled
10:02
by serial protocols, UART or RS485, some sensors are connected by I2C, sometimes there's special translational hardware which presents itself to the system
10:21
as a graphics card and the whole thing gets translated into pixels down to the LCDs, so proprietary old legacy stuff, no source and there's sometimes a lot of reverse engineering involved and because some of the suppliers
10:43
of these devices are not bankrupt. Yeah, they're bankrupt or they're being bought by a different company and they're not, say, willing on cooperating, like basically helping us, I mean it's like yeah, it's all a closed source software
11:01
so why should we help you because then in the future you might do everything yourself, maybe. Sometimes they even don't have to source themselves anymore, so sometimes they go down in the basement and have a look in some file folders and like real paper and stuff, anyway.
11:22
It's quite interesting, sometimes you basically log on to those old machines and you have a look at the history and people jumping from one machine to another in order to fix a bug because they don't know. Yeah, so that's something we really want to fix because that's something you can, well, you probably know yourself,
11:40
that's something that is not maintainable. So we really want to push a standard, standardized interfaces so it will be maintainable and will make us, I'm a DevOps guy and I use more developer guy but that will make me personally really happy if that works. Yeah, the standard work for all the suppliers
12:01
was have one DVM image, put their software stack on it, put it in the device, fiddle around till it works and then never touch it again and what we're trying to do is like automated updates in the field and on 6,000 devices with thin network lines
12:23
with sometimes only 10 KB bandwidth, you really don't want to push that often an image through it, especially if it's like 200, 300 megabytes. So we would like to test the stuff before we roll it out and make sure that it's working
12:40
like we want it to. So the problem now arises that we have like a huge hardware variety and how can we simulate that hardware in combination with the software we want to run it on and we're trying to use QEMO.
13:01
The QEMO is pretty awesome because you can really find details, specify how the hardware should behave, you can select different CPUs, you can connect different serial devices and different options like is it connected by an ISA bus or a USB
13:22
and you can change the details of the DMI table which is like a specific part of memory on the hardware motherboard itself. We heavily rely on that to identify the different devices.
13:40
So in order that we know what services we have to start and which not. So on a TFT you need, we use there an electron to display stuff and on an LCD we use like custom written controller service which is communicating with the LCD displays.
14:00
So yeah, we choose QEMO and it's open source. So this is the container, how it looks like. It's not what we're using on the project but it basically reflects the state of the composition.
14:21
You have the outer container, it's an Alpine whereas QEMO installed. You have an inner QEMO running which gets the disk image via a volume mount and some ports are expressed in the same way exposed, for example the SSH to remotely control it. In the future we would like to get rid of that
14:40
and do everything by AMQP commands but this is still far future. Does everybody know what AMQP is? Basically the protocol used by RabbitMQ which is our middleware, middleware basically, message broker.
15:02
So the interesting stuff for QEMO is the DEV TTYS0 which gets exposed on a socket 9000 and can interact with external hardware, in this case with a mocked hardware and the control port of the mocked hardware is also exposed.
15:20
We have an OVNC which lets us see how the boot process is but really the yellow stuff is only optional. It's not really necessary for testing the image itself. So it's quite nice. So Nova, who knows? NoFouncy, it's basically HTML Founcy VNC applet basically
15:46
so you can browse to port 6080 and basically see what is going on on that machine and you don't need one of those applications. So it's tightly coupled in there.
16:00
So what we have been doing at our project, we first did that actually with graphical applications with a NoFouncy in a container so we were running up to 600 of those devices and you could just click a link with a unique identifier of the simulated device and you could see what was going on on one of those devices. So that's really helpful if you wanna stress test
16:21
your system with real devices. Especially the back end, not the devices itself to generate load on the back end. What you're gonna see is something like this setup. It's not much different but the mocked, you see like the mocking part is deleted
16:41
and the TTYS0 port is just like exposed by the Docker container so we can interact directly. So we kind of doing the manual mocking. Yeah, so we're gonna show you basically how that thing works. We're gonna do that unfortunately locally
17:02
because we can't interface with our clients CICD system at this point because we didn't get approval but that's fine. I mean, we didn't get an answer so far. So that's why we're showing you it basically
17:20
and then we're telling you how we're doing that currently in the CICD context of our project and how we're deploying that. Okay, demo time.
17:42
Yeah, give me a second.
18:18
Perfect.
18:20
So there's the Dockerfile which builds like the container you've seen so far. There's this QEMO disk image. I was trying something that's not so interesting. We have a makefile which should make it easy to do stuff right now but we can make runcocker.
18:54
So we're still attached to the container which boots up right now. So we have a little delay between the QEMO start
19:02
and the novnc start because if you start them right after each other, sometimes novnc says it can't connect to the VNC port of QEMO so detail is not so important.
19:31
The next interesting thing is we can have a look
19:43
at the booting container which looks like this. So this is what you would see if you would start QEMO manually and you could interact but I guess here it's disabled.
20:06
We can now connect to the running session inside.
20:22
The service running inside runs in tmux so I can attach to the output again but on boot it starts as root so we have to sudo tmux, we can ls first, we see a session running. We can attach to the session with this main
20:45
so there we see that the HTTP server started and the sensor handler started on the devtty as zero with the given baud rate. So now we have a look at the service
21:04
we want to test which is on local port 8000 which looks like this. So far it hasn't received anything. Make this a little smaller that you see. Okay and then we can send some data.
21:29
We're not waiting for a response. Q0, connect to local host and the exposed port
21:41
and let's try this. So you see the sensor received some raw bytes, decoded them, stripped them, parsed them to int
22:03
and if we plug something in which is like
22:22
non-parseable as an int, we see that an error happens and test scenario would be put garbage in and see if it's got parsed by the program itself so we'd see it didn't change.
22:41
Yeah so what we're for instance doing with yeah sensor data perhaps is we all our devices basically have a exporter endpoint basically so Prometheus is basically scraping metrics from that system and well it then basically and Prometheus digests all those metrics and we can do later on something with that.
23:03
Quite interesting to see in the morning where the sun is going up or how bright it is or whatever but it's also for diagnostics quite interesting. So I just go back a little.
23:24
Too comfortable. So we pushed some data in here. It got processed by the custom service
23:40
and got exposed on the 888 port again. So this is like a round trip through the software to test. Yeah so since it's dockerized you can put it on Kubernetes and put it in the CI and run tests on it
24:02
do it automatically yeah. So what we're doing we have our project still help Charles which is the configuration layer you basically tie around one of those docker containers and then you can basically mount the news build
24:21
or OS image you want to put in there then basically run it and you're also through Kubernetes can tag it put all the metadata into it and run it at scale. That's actually quite yeah helpful for us at least because now we can basically show the client what we're doing and what kind of different configurations is retestable which is usually not the case
24:43
with hardware if I mean probably a few of you have tried replicating things with hardware it just takes a lot of time. As you see it's not silver bullet because like this special hardware is really hard to mock a special graphic cards
25:05
you can't really mock and clear new and this is not feasible but the standard cases are doable and it's not possible to have a sample of every hardware configuration in the lab.
25:20
First we would need like a warehouse to store them all on the other hand you would have to plug in like the images or like deploy them via updates but still so and you're not really controlling the input on the test you're just testing that it's doing something but you don't know if it's doing the right thing for example.
25:42
If you can't control the temperature sensor right and so this approach makes it kind of possible to get rid of all these problems and not all of them for sure but get rid of some of these problems and make the workflow a little more streamlined.
26:06
Okay cool that was it basically so we're open for questions. I guess we have plenty time for questions. Oh go for it. Do we have a microphone?
26:21
No we don't have a microphone. I have to repeat the question so. I have to repeat the question. You guys have a lot of powerful legacy machines so these are quad core machines. No. This. Yeah go on. This to me sounds like a botnet in the making. Did you guys ever run into any problems with that?
26:40
Did you know of any of these machines that were hacked in the past? Please repeat the question. The question was if these old legacy systems were hacked. If that's correct.
27:00
Yeah. Yes. Okay. Was it even possible to install custom OS's on these machines? Most of these legacy systems typically don't allow you to do a lot of stuff with them. So do you guys have like another part of hardware or installing the stations that you can do anything with?
27:25
We actually have a lot of those devices shipped to our office so we have a quite big laboratory where we can test quite a certain range of models but our office is quite stuffed with it so I mean that's our bottleneck.
27:40
We can test a few of those devices live at the stations but from a security standpoint of view we try to limit the access to the network for once because of past happenings and yeah security is also a reason why we're doing all of this so we can basically close down the operating systems.
28:03
Yeah. The workflow, how the suppliers deployed a new image was like a guy was going there, opening the device, putting a CF card in there with the new image and fiddling around and so if that was a question.
28:24
You basically do it by experimenting with the library in your lab right? And then just coding the behavior that you see so you're actually just reverse engineering. Yes, the mocks are getting developed by reverse engineering. Sometimes there's a spec but usually it's wide
28:41
as a door and every vendor made its own solution to that. You first. Have you considered Linux kit as an option to? There was a, if we have considered Linux kit,
29:05
I guess so. The decisions were made before I came but we have good architects and I guess they did so but I can't tell you why the picked build route. I think because you can choose pretty specific what driver, what core utility you want to have in there
29:23
and if you really need it and I guess that was the reason for that. So back in the time build route basically seemed like a good option. So when we started, well when the project was started was not quite sure what variety of hardware is out there so with I mean of course you test against what you know
29:41
or you basically build the architecture against what you might expect and at that time we had a certain amount of knowledge and build route was basically fitting for that. I mean it's a long running project so we haven't started like yesterday so yeah. Linux kit is quite old also. Yeah cool. I was interested about the options.
30:02
I'll check it out, yeah. Ah yeah, if you are interested just drop us a note and. Yeah we have the code basically in a public repository so we probably gonna push it, publish that, is that possible? Cool yeah, so you can have a look at it.
30:21
Okay. So you talked about using Helen to deploy these and you showed us the developer experience. Pretty nice. Do you also have some CI of the internet so I can push things up, are things tested or is it very much aimed at the developer? It's like our project is completely built in the CI
30:43
so we have a GitLab CI system and as soon as you push basically it's through the hook basically then builds all the code and deploys it automatically. If that was your, so again. Yeah so you were talking about exposing those ports and then does your GitLab runner,
31:00
the thing that runs in your GitLab runner then point at those ports or is the GitLab, does that thing sit inside your run? So at the moment it actually just deploys it so the life cycle of the deployment is there independent from the, yeah, let's say run time of the CI or deployment basically
31:22
and we haven't integrated that at this one so far because the requirement was, we just want to see what is running on there and if it's crashing and that's something you can see over time quite well in Kubernetes through logs, metrics and so forth. Another thing like compiling build route
31:42
is like a huge deal. It takes really long even in the CI even with having machines so it's not likely, we reduced it down to one and a half hours I guess like with the whole tool chain but still it's, you want to have it optimized. You can't wait, click a button and wait for it to compile.
32:17
If we build separate images for the target hardware and the QEMU, no, we don't want to do that.
32:24
So we want to test the real image which goes into the fields, not a specific image for QEMU. We are having some kind of a variety for some in-house monitoring but this is not going ever in the field.
32:43
Down there. If we're using caching for the build route part.
33:05
Yes, we're caching the tool chain so usually build route builds the tool chain as well especially if it has a blank system
33:21
and but what things would really speed up is the CC cache but we didn't get it working till now but this would be a really boost like I guess we could make it in half an hour if we do so. Someone else?
33:43
I guess then we're spot on. Perfect, thank you very much for your time. Thank you. Thank you.