is titled, Dissecting Modern 3G, 4G cellular modems.

This is by Harald Welt and Holger Freiter. And that was totally mispronounced, sorry about that. Freiter. We saw in the previous presentations on smart cities that there's a lot of IoT that just needs to communicate. And while there's Zigbee and LoRaWAN and other protocols, most likely they're gonna fall back

on what is already there, tried and proven, and those are the 3G, 4G modems. I don't really need to introduce our speakers today. They're well known for years and years here at the Congress. So I'm just gonna pass it right over to them and have a great talk.

So today we're going to talk about cellular modems. Just to differentiate it, it's not about baseband or baseband exploitation, it's really about a GSM module or 3G or 4G module.

Our talk is going to be structured in a couple of phases. First of all, our motivation. Why are we looking into it? Where are we coming from? The second part will lead to the history. What have we done before in terms of modems? Looking at how we picked the modems we actually looked at.

Then something that we actually didn't expect when looking at it. Also, then looking at the firmware upgrade mechanism, if there's one, how it works, what has been done, and we will finish with our recommendations and wishes. First of all, we're implementing GSM specifications

for more than a decade now. It started with humble beginnings of sending AT commands to modems on mobile Linux devices, to actually working on a free software smartphone at OpenMoco and then working on OpenBSC and Osmocom to implement radio area network software

and core network software. And has been eight years since La Forge presented about how our Zen modern smartphone hardware looks like. It's seven years since we worked on Osmocom BB to run our own baseband software on our commercial grade hardware.

And professionally we have worked with M2M devices and have built M2M devices ourselves using 2G modems. And from this point we started to explore how, which kind of device would we use in modern embedded devices or M2M or IoT devices these days.

For our implementation of 3G and 4G network software, if you send messages over the air and you don't get a response, it's always difficult, like did you encode it correctly, was it sent? So we looked into having a device that allows us to get logging,

to see if the message arrived or throws an error, even able to extract traces from it. And what's also important for us and got us into Osmocom and OpenBSC is to build tools to allow others to understand how cellular technology works. So while TCP IP might be well known to many of us,

how the IP is actually transmitted back to the core net and routed to the internet is not that clear and we want to make it more visible and having technology and tools helps for it. For a brief moment, picture yourself trying to build a classic M2M device.

You might pick a modem because it's already certified and easy to use, but you need to run some application code on it. And the traditional approach would be to get a microcontroller or a bigger processor and connect these two devices using USB or serial. But it means that you need to have a bigger PCB,

more power consumption. So it would be nice if you can run application software on the modem itself already. And one of the driving factors for it is to reduce the PCB space, to have a lower power consumption, to save on the bill of material to have fewer components.

And back then we found something that's called OpenAT. It's done by Sierra Wireless and mostly it allows you to write C software, which is then compiled with GCC and uploaded to the modem and you can start running it. It will be loaded into the real time operating system. It runs as a normal process.

There's no MMU, no privilege separation. So if your application crashes, the entire modem will crash. To ease debugging, Sierra Wireless has nice tools to get output and send IT commands and logging debugging. But the problem with this approach is that if you build an application and make it stable,

you know so much about this OpenAT stuff that you're mostly locked into this API. So your architecture and software is following what an OpenAT application will look like. And then you've spent years developing an application and suddenly there's no path for 4G. So you're locked in and even your 2G modems will be discontinued.

So it's a nice platform to get started but it's kind of a dead end. And this brings us to like our modern requirements. So what does a good modem look like? And one is we still want to be able to run our own code in it and not like some Python script or some limited Java

but like our real C application with access to the device and no artificial control. We don't want to be locked in by a single modem vendor. So it might be okay to use a specific ship set but we don't want to be forced to follow whatever the modem supplier wants us to do.

For debugging purposes, we want to be able to get lock messages to see what the modem is doing, see if it's throwing away stuff, debug output, be able to control it. And for 3G and 4G development, we want to be able to see the radio messages.

And you might know a tool called XGoldmon has been written by Tobias Engel. It allows you to get tracing information from Infineon basement, but it has some limitations. So we want something like XGoldmon but for GPRS, Edge, UMTS and LTE.

The modem market or general, the cellular market is kind of dominated by Qualcomm and it's kind of set by itself but it also means that if you pick a modem, most likely it runs or it's based on a Qualcomm ship set and Qualcomm is very close to what we want from a modem

because they expose something called the Diag protocol. And it's also used in many different Qualcomm products from DVB-H to GSM to FEM to cells. The first time I personally heard about it was at the 28C3 by a talk from Guillaume

that looked into the baseband Qualcomm stack and this Diag protocol, the framing is very simple. It's like classic HDLC with a start and an end marker with a comment bit, some payload and a checksum and it's used for events like if your modem is switching a network, you get an event, you can enable logging

and then you get a lot of textual output but also for comments and response. So you can send a comment to read a memory address and to get the value from this memory address back. And literally there are thousands of different messages that can be sent or received and in terms of free software implementations

like the modem manager or GSM parser only use a fraction of these available messages. But it means like this Diag protocol is something you want to have direct access to because it allows you to see what the modem is doing. And it brings us to the point of selecting a device that is exposing Diag.

And in the past, the option I can stick might have been a very good device to use because it's exposing Diag on USB out of the box but it's kind of old, it's using old Qualcomm software, it's limited to 2G and 3G. So we look into something more modern and one of the devices we found is from a Chinese modem manufacturer called Quactel.

It's a UC20, it exposes Diag out of the box, it's even documented in their hardware interface that it's a diagnostic interface but sadly it doesn't support LTE. So if you look at a modern device, it makes sense to not go for 2G and 3G only

but also have an option to go for 4G which brought us to the EC20. It looks like the UC20 but it has LTE so that sounded quite nice for building a product that can be soldered to your BCP but for development purposes,

they also offer it in a mini PCI express form factor so you can just plug it into one of your devices that might already have mini PCI express. So we picked the EC20 as like a module we want to look at it and once we started to look, it has like the Qualcomm MDM9615 chipset

which surprisingly is also used in the iPhone 5 but beyond that there's not a lot of documentation of what it has which brings us to the unexpected surprise. So after I got the modem, I used the AT command interface just to play with it to see if my application could do what it wants

and was hanging like after a day and I got involved with the supplier of the modem and they gave me a firmware update which was a zip file and I unpacked it and the file names looked awfully like a Linux system, which is why would my modem have a Linux system in it

but maybe I'm just mistaking it and then they have a flash tool and it looks like the flash tool of Android was fast food but like maybe it's coincidences or it was, it was just convenient to use it but actually other people have already seen that like the MDM9615 runs Linux on it

like at the DEFCON 26 by Miki Schalkotov. So apparently it runs Linux on your modem and the question is why? Why would it even run Linux? Because Qualcomm is known to have put their IP stack into the modem, IP stack, zip,

why would they stop doing it? It didn't really make sense so we started to look at it and also there was no information that Linux actually runs on this device, which means no written offer. Then I used the Rmines GPL tools

to unpack the flash file system. It really looked like Linux and I started to look at some of the binaries that are not standard but is a correct Qualcomm specific and you see funny strings that look like AT commands but like AT plus Linux or Q Linux command,

what would it do? And at this point we started to explore both technically, so what is this platform really looking like, but also from a legal point of view like can we please get the source codes, can you please put a written offer in it and at this point I hand over to LaForge. Yeah, so as a lucky coincidence,

I have been doing some GPL enforcement in the past, so well, okay, but first we started to look a bit at the hardware and well, if you're doing hardware anyway,

professionally and you have all the tools and the process and your partners for assembly and so on and so on, why not just do a couple of boards to help you with the task at hand? So many of the mini PCI Express modules that you can find for cellular modems, they have additional signals on undocumented pins

of the mini PCI Express connector like PCM audio or even URs and so on and well, of course, if you just put the modem in a regular PC main board or an embedded device, those signals, they don't terminate anywhere, they're just not used on the slot side and soldering wires to the pitch

of a mini PCI Express board is not very convenient, so I created what we call a breakout board and you can see a picture of that, it's an open hardware project, schematics and everything, design files have been published, so you have this connector on the right hand side which exposes all these extra signals so you can actually easily access the various undocumented signals.

So the EC20 solder module documents, there are debug UART pins which are not the normal UART pins on which you speak AT commands but it's additional UART but it seems like not all modules have it enabled so we bought modules from three different suppliers, they all have different firmware versions and different configurations

and some of them have it enabled, some not but those that had it enabled have it at 1.8 volts but how can I say, in their wisdom, the designers decided not to expose the 1.8 volt voltage rail on any of the other pins of the modem so you have external voltages of 3.6 volts,

3.3 volt and so on but not 1.8 volt and since in previous projects whenever I needed to attach to a UART of an embedded device, I built another level shifter for a specific voltage, I already had built for 2.8, 2.5, 2.3 volts before, I decided okay, let's do a board that is a multi-voltage UART

so you can actually select the voltage of your UART with a small rotary switch and connect to a serial ports of various devices and that's what's here, it's also another open hardware project at OsmoComm. Yeah, and then you attach to the serial port and you get a Linux login prompt and interestingly, if you look at the firmware update file

that Holger just mentioned, it contains of course an ETC pass with the file and there is only a DES hash and if you do a little bit of password cracking, you see it's oelinux123, is the root password of the device. So by now we were 250% sure

that there is Linux in the device and yeah, this is by the way how the full setup looks like, you can see I sold the small additional three pin header to get access to the UART on this module and this way we could start to explore it further. So okay, brings us to the topic of

well, where can we find the source code? The module didn't ship with a written offer, it didn't ship with a license text, it didn't include source code, it didn't even mention that Linux or other GPL license software was used. So while digging around the official supplier of the manufacturer didn't release anything or didn't provide any such information, we were looking a bit around

because I mean, it's likely the other devices do the same and we found that actually Qualcomm is publishing a complete open embedded build system for building Linux for those modem chipsets. So this includes the open embedded meta layers, the kernel, the bootloader and various other bits and pieces.

And it's almost completely undocumented, there are basically some Git raffles and you can peek around them but there's not really a documentation how to use it and well, if you look at the entry level website of this open source project from Qualcomm which is called Aurora, well, there's like literally hundreds of branches and tags

and you don't really know what to use for what and they say, well, this is one example version that you can build and then somebody I think years ago posted, well, I tried your instructions but well, it doesn't compile, I'm facing some issues and of course nobody ever responds because well, you know, why would they ever respond? But anyway, it's public

and on that website you can find it and you can start building. So we started building that, of course, we also ran into the issues that didn't build but anyway in the end we managed to build some of the code and meanwhile we were talking to the manufacturer of the modem and asking them for the complete and corresponding source code and then what they send is the complete and corresponding source code

to the firmware update tool that you run on your host PC which we didn't ask for and it was not GPL license, we still received the source code, okay, nice. It's good, at that point we could understand how the firmware update protocol works towards the modem

and then well, you ask again for the complete and corresponding source code and I said, oh, we never been, by the way, all the typos and grammar issues are not introduced by us so we never been in legal dispute and we always make sure to understand intellectual property rights ahead of using technology belonging to a third party. Well, clearly they did not in the Linux case and then they sent us this nice little letter,

this is an excerpt from it, it's like, oh, we always respect the importance of intellectual property rights and laws and we actively engage with known essential intellectual property right owners, apparently the copyright owners of Linux and other free software are not essential and so on in order to be compliant with the rights.

So then you ask again and you ask again, so you see basically we always ask the same question and it's like, oh, we appreciate the efforts, this was to the lawyer of the NetFilter IP table project, they missed the fact that it's multiple tables, was singular but anyway, well then your client, me,

doesn't have the right to empower copyright, it's like, oh, okay, that's new to me and they claimed that I had transferred my copyright to the Free Software Foundation, which I never did and which nobody, I think, or at least it's highly unusual if you work on Linux kernel code to do so. So I did not have copyright.

Okay, anyway, and Qualcomm, oh, sorry, sorry, my mistake, QuickTail always respects intellectual property rights, of course. Okay, so still no source code, then we ask again and ask again. Well then thank you for your detailed explanations, da-da-da-da, we will provide a tarball and then we're always willing to achieve GPL compliance and so on and so on

and then another month or so passes and then I finally, well, we do some legal enforcement, we do some warning notices and so on and then yeah, we are not perfect and we cannot construct a perfect website, which I never requested. We just wanted a source code and then well, you get some source code

and it doesn't build and then say, oh, there is a header file missing and by coincidence, it's a header file that I wrote. I don't know if they did this on intention. I mean, how many header files did I write? It's like. And if you know IP tables, there is a match for the differentiated services

code point for the TCP field in the IP header and that has a header file, which is like eight lines of boilerplate, eight definitions and this was missing and I said, oh, we don't have this file and Qualcomm also doesn't have this file and Qualcomm never provided this file to us. I mean, it's in the public repositories that Qualcomm holds on code Aurora

and the kernel doesn't build without those files and then by the way, we will not discuss compiling issues by email anymore. Okay, then some more time, I get more and more, then you get individual header files and individual emails and you put those header files in your kernel source tree and then you see that the scripts in the kernel

have missing executable bits, but certain C files suddenly have executable bits and then another header file is missing and so on and so on, but by now we have received various source code tarballs. They interestingly contain not only the GPL-L-GPL code, but also other code with like BSD type

or Apache type licenses where they wouldn't have to release it, which is good and they did this intentionally and I think it's a very nice sign of them that they don't release only what they have to, but they release more. For the EC20, it's still not license compliant. There is no busy box source code included, for example, not that busy books would ever have done any GPL enforcement in the past.

And well, but that's not my primary concern and I think it's getting there and it's workable and you can use the source code that they release. Interestingly, there's other modem manufacturers like Sierra Wireless, which are also building modems on such Qualcomm systems

and they release not only the source code, but extensive documentation. So this is a small excerpt from a screenshot where you can actually, oh, they describe how to build it with open embedded to describe how to use fast boot to install the firmware on the module and so on and so on. However, as good as they are in the open source side, they try to lure customers into a proprietary framework

like they did with OpenAT in the past and that, well, again, would result in vendor lock-in. So it's not really recommended or I think a smart move to go ahead that way. And with that, I'm going to hand back to Holger before returning later.

We're going to very briefly look at the hardware. It's a Qualcomm MDM chipset I've already mentioned also in iPhones and it turns out maybe in your future car, at least right now it's a modem from Quacktel and Sierra Wireless,

but sadly from a free software point of view, like it runs Linux, it talks to the hardware, but there is absolutely no documentation about the hardware in the internet. I like even on other websites, not nothing because it brings us to the hardware overview. So we know there's a ARM processor inside,

probably hexagon, they're connected somehow, nothing. So that's very frustrating to have spent many years on free software to see Linux winning, even getting into the modem devices, but no hardware documentation being available. Not even a block diagram.

You have to repeat it? Not even a block diagram, yeah. Yeah, so nothing. Let's look at the software part and explore the system from a software point of view. LaForge has shown how to get a serial console on it, but not every modem has it enabled and we didn't look at what it takes to enable it

and also soldering is not that nice. So after exploring the unpacked root file system, so we saw it runs an Android debug bridge. So if you have used Android, so ADB shell should give you a shell and nicely we have already seen the AT plus Q Linux comments to execute something on the device

and we found a script that is reconfiguring the Android USB gadgets to actually put ADB in it as well. So let's try it, you execute the AT command and then suddenly the serials on your host Linux don't work anymore because the way the QC serial module is written,

it's matching a device based on the number of interfaces and if you added ADB to it, you suddenly don't have four interfaces, but five interfaces and your driver cannot identify the device. So we had to first take it a bit and LaForge has made a more clean patch to actually get it.

But after this small experience, we have ADB shell on it. It works on any module. But yeah, and the ADB shell is root of course. I mean, so you don't even need to root the device. It's like you get the root shell immediately. So there's no lockdown as well. It's root, there's no SE Linux and just a very nice and open Linux system.

To build it, it's a bit odd that it has a bootloader that seems to be proprietary. Then it has the Android bootloader and Android Linux kernel, the Android debug bridge we've mentioned. But surprisingly, it's not using the rest of the Android system, but it has a GNU libc with BusyBox,

Userland, System 5 in it. So it's a very classic open embedded build and it's actively developed and maintained by Qualcomm, but they make so many releases you actually don't know which one to pick. It's a bit of a zoom. Yeah, and then you start to look a bit at the Linux kernel that is released

and luckily and interestingly, and to my pleasant surprise, there's no binary only kernel modules. So everything in the kernel is released in source code. Nevertheless, of course, it's a lot of source code. So if you look a little bit at the number of lines a diff shows up between the closest mainline version and the kernel that's used in those modules,

you end up with 1.5 to 1.9 million lines of diff. I mean, this is not actual code lines. This is counting all the lines of a diff output, including the context, but still it gives you an idea about the size of the differences compared to mainline. And of course you expect on those kernels that there's all the CPU specific stuff and lots of driver code and so on and so on.

But then if you look at it in more detail, and as a disclaimer, I haven't looked at Qualcomm, Android, Linux kernels during the past 10 years or so. And we'll just say eight years, no, six years, whatever, a long time. And I know there's a lot of code in there, but I didn't expect all the different things that I found in there. I mean, they have their own shared memory

based logging infrastructure and shared memory is not shared memory with the modem processor. It's only shared in the Linux system. You have an interprocessor communication logging process. IPC is not interprocess communication like you would know it's interprocessor communication. And you have something which is completely flabbergasted me. It's called remote spin locks.

I mean, you know, spin lock, you basically, you have a mutually exclusion mechanisms are only one thread or one CPU can enter a critical part of the code on your multiprocessor Linux system. But then here you can actually also block the modem processor, the hexagon on the other side from entering a particular section.

What could possibly go wrong? If you hold, if you keep the real-time operating system in busy waiting, but okay. Yeah, then you look at the source code and I've actually, since I haven't looked at Qualcomm kernel source code for quite some time, I was expecting, well, this has been in all these Linux Android phones that have Qualcomm trip sets and lots of, since it's open source,

plenty of people must have analyzed it. And there's certainly some documentations and high level overview about all these individual subsystems, how they glue together, how this works. And I could just look this documentation or look at that documentation. But interestingly, it doesn't exist. So I had to start to write that documentation. There's now a lot of information in our Wiki and some of the interesting parts

that you find is in there is the shared memory device, which is the core of all the interaction between different CPU cores. You have interprocessor communications, you have remote network, you have a BAM, that's also a BAM to BAM. It's the bus access manager. And we have IPA. This is not your favorite beverage,

but the internet packet accelerator. And it does some diagnostics forwarding and so on. And if you look at these subsystems, I draw a graph, you know, if you normally know how rare that is, I draw a picture. And to symbolize you what's happening in this modem. So if you look at this picture,

you will see basically the large square at the top is the application processor. The bottom small part is the modem processor, about which we don't know much, of course. But then on the Linux side, due to the availability of source code, we can see what's happening. So you have the shared memory device and then you have channels implemented by the shared memory device

and individual different subsystem binding to those channels. There's two 480 commands, which attach to the serial function gadget of the USB gadget code in Linux. So basically, the important part is to see that the USB you speak, you don't speak to the modem actually, to the modem processor, to the basement processor,

but you speak USB to the Linux gadget inside the Linux ARM core on those devices. And then that Linux ARM core forwards or handles different interfaces on your USB configuration in different ways. And you have this small box symbolizing the user space and you can see how the different paths go. It's quite interesting.

If you look at the serial ports, you have a serial port for GPS and you have a serial port for AT commands. And you think, well, okay, these are both serial port devices. They must be handled quite similarly. But no, the GPS part is actually handled here, over here. It goes into user space, it's bridged. It goes into a virtual serial port here again and it goes into the serial gadgets

whereas the AT commands go straight here inside the kernel and never end up in user space. So I don't know why, but it's quite sophisticated to say. If we look at the diagnostic subsystem, which is particularly of interest to us, and now you can see, I didn't draw graphs anymore. I just wrote a little bit of dotty.

We have the modem DSP on the left-hand side. Then we have the Linux kernel where we have the shared memory device. On this, a diag forwarding module in the kernel binds. On that, we have a connection to the diagnostic function gadget of the USB gadget driver and that just goes to your host. So if you talk the diag protocol to the modem,

it actually goes this way through Linux, through the shared memory device in the modem DSP. But what's even more interesting is that there is a diagnostics character device on Linux called DevDiag, which, well, for example, to QMuxD or other processes, they basically attach to this diagnostics device and all the logging that you find

in those Linux user space processes. They don't use syslog, they don't use an Android logging framework, but they log through the Qualcomm diagnostic subsystem and you get the log messages of those processes through this kernel device over to the function device, over to USB into the host. So if you manage to figure out which logging flags and so on to enable,

you get the log output of those Linux user land processes over diag. Well, very Qualcomm-like, not so surprising, but still sort of unusual in the Linux world. If we look at the networking, the QMI, which controls your, basically the modem, which network you attach to, whether you activate PDP contexts, your QOS parameters, and so on,

you might have used QMI, CLI, or other tools on your Linux laptop to talk to such modems. And in this specific case of those Linux-based Qualcomm modems, well, you again have the modem DSP, it goes through the shared memory device, talks to the RmNet device, the USB gadget,

the USB to host, and then you have your host PC somewhere on the right-hand side over there. So this is basically the path your QMI takes. But then you have also QMI in the user land on the modem itself, which is what's presented here. And they have what's called a QMuxD, the QMI multiplexer daemon, which then offers Unix domain sockets

to various different client programs. So basically all these user space programs, by using this Unix domain socket, they can talk QMI to the modem as well, and all of them can basically configure, get status reports, and so on, all the different paths and all the different services on the modem, which is interesting, and is of course something that we want to have

coming from our initial motivation. We want to run our own applications in there, and we want to talk to this QMuxD and talk QMI in there. So we created a couple of tools to help the analysis. On the one hand side, we use the OpenEmbedded that Qualcomm released to build a matching OPKG and the packages for tools that you need,

like SoCAD, STrace, LSOF, and so on, for some exploration. We also have written a couple of C programs for testing, basically for accessing the QMI from code inside the modem that's successful. And then we have a couple of, those still linked to the proprietary libraries

that are provided in the modem. And then we have started with some entirely free open source programs, like the QMuxD wrapper, which is an LD preload library, so you can trace this QMuxD communication. There is ongoing work for libqmiglib transport for this QMuxD,

which then would enable you to run a program that's linked against the free software libqmiglib inside the modem. So you can develop it like you run it on a laptop, but you can run it transparently inside the modem after cross-compilation. There's also a tool which we call OsmoQCDiag, which is basically a host tool

for obtaining these diag-based logs from the modem. So you can run this on your laptop, attach it to the modem, and then you get all kinds of traces, not only the interface traces, but also QMI protocol traces, which we then again decode using libqmi, so you get a textural representation. There's ongoing work here to basically

move all of that into Wireshark, so you get the full decode of that in Wireshark, but that's not yet there. So what kind of user-based programs do we find? Okay, ADBD, we know what that does. We have an ATFWD daemon. Well, AT Forwarding daemon. Well, what does it do? It implements those things like an AT plus Q Linux CMD.

And other AT commands. So basically, a user-based program on Linux can register like a callback within the modem to forward certain AT commands into the Linux user space, where you can then implement them. So you can basically implement custom AT commands in user-based programs. There is all kinds of other software,

which we haven't really figured out yet a lot what they do. There's one monster called the QC Map Connection Manager, which basically allows you to run a Linux-based Wi-Fi access point with LTE backhaul, excuse me for the typo. So basically, you have to attach a Wi-Fi chip

to the SDIO interface of your modem module, and then you have a full personal access point device that has an LTE backhaul. And then the Wi-Fi, like the parameters, for example, the key and the SSID and the channel and so on, you configure all of that through what? AT commands, of course.

If you ever wanted to look at software that receives AT commands and then generates textual config files for WPA supplicant and for host AP daemon, then you can look at this code. I prefer not to. So we have the Quectail Bridge, which, well, is a very simple device.

It reads from one device and it writes it to another device. And apparently, this is such a complex task that you need to schedule a different process, and I think it has three threads or something obscure like that. So, okay, well. Which brings us to the funny bits and pieces that you find in those modems. Well, the first thing is AT plus Qlinus command.

We spoke about that. You can run arbitrary shell commands as root in a read-write-rule file system in there. So basically, you can do anything. I mean, you can send an RM-RF slash and then, well, it's gone. It's dead, Jim. So we also have commands to switch into fast boot mode so you can update the firmware. You have a special AT command to print a DMESC,

and you have also all kinds of other AT commands. And when you do a strings, just a strings call on those executables, it looks like shell scripts in many cases. And one of the most funny things I found in this modem is how many processes and threads does it take to reboot a system?

It's apparently a very complex question. How do you reboot your system? Apparently, this was the easiest method they could find. So there's one process, the reboot-diag app, which registers a diag command with command code 0x29 with the diag infrastructure. And it spawns a thread which executes another executable

called QMI simple real test with an input FIFO. And then it calls system echo modem reset to write modem reset into the FIFO of the input of the process it just spawned, which then causes this to send a QMI message to the modem, which will reboot the baseband processor. And then, of course, you clean up.

You remove that temporary file because slash TMP is not a tempfs, but it's read-write file system. And then it goes on to write the string reboot into slash def slash rebootedef using F write, not using echo this time. Right, this is a C program, not a shell script. So apparently, they discovered that and used F write.

And then we have a reboot daemon, a second process, again with two or three threads, which reads this rebootedevice. Actually, they published the source code. So this is the actual source code. So you read from this def rebootedevice. Then there's a nice comment documenting what it does. You do a string compare. You have the first printf going for reboot.

You have the second printf initiating reboot. And then you call system on the reviewed executable. So this is apparently the most simple method. So if you ever were wondering how we would reboot your Linux system, this is the new reference.

Yeah, then we have C programs that look like shell scripts. So this is an actual, of course, condensed output of strings on the Quectl daemon, right? You see like echo in sysfs files. You see copying files, even with semicolon in there. You see echo into the duty cycle of some full width modulation.

No idea what that does. And then they even grab for the process and they kill processes, you know, with the most obscure things. And they even, they don't use reader, but they do like ls and then parse the output of that rather than use open the reader and the usual calls you would do to get a list of files.

It's quite amusing. Yeah, which brings us to the topic of firmware updates. But you can, I actually missed this example. Sorry, I have to make a quick interruption here. All these machine-to-machine modems, they typically have an LED and the blinking rhythm of the LED indicates to you whether it's registered to the network,

whether the data connection is open, whether it's searching for networks, that's all different blinking patterns of the LED. And how do you implement this on this modem? Well, you run a user space daemon that calls system echo one to the def GPIO, which controls the LED all the time, you know, to blink the LED. It's not like the kernel wouldn't have infrastructure

for LED blinking patterns and so on and so on. But okay, so you have a daemon that does nothing else, but basically toggling your GPIO by spawning shell processes using the system syscall. Okay, with that, I hand over to Olga.

Now the question is, do you expect anything after the topic of firmware upgrade or is it going to be an empty slide? And the answer is they know that they have to offer firmware upgrades through over-the-air updates, making it as small as possible. And actually, it's something that Qualcomm

is preparing for modem vendors. So it's based on the Android from around 4.0 Recovery Git, not using Android myself. I haven't looked at how it worked before. Many of you might be more familiar with it, but it's mostly a zip file and it contains delta updates and surprisingly, they are somehow hashed to SHA-1

and then the SHA-1 is being signed with a private key and the result is being put into a comment of the zip file. That's probably pretty standard, but looked a bit odd and it's nice that they prepare probably secure firmware upgrades

with minimal delta function. So what has QuackTel done to this code? They still use it, that's quite nice, but they have removed or actually not really removed, but they're not using the RSA code to verify a signature. And instead of using the standard Android way to patch these systems,

they use a proprietary component from a company that used to be called Redband, but now it's Harman and probably soon Samsung. And this Redband component is nothing that QuackTel has written, but it's a commercial product and it's used in the QuackTel UC20 module as well.

And also in other automotive projects, we have seen Redband updates being used. So at this point, I started to look into how does this update format look like? And instead of presenting a very complex form of the format, I have this slice. So other people like Matthew Solnyk at,

I think Black Hat presented something called attacks on OMA device management. These can be remotely triggered, but the update mechanism used here cannot be remotely triggered. The modem needs to be asked to start an update. So that's already a bit more secure,

but I started to look at the hex thumbs of a specific Delta update and with a lot of help from DITOSPA, we actually managed to understand how the updates binary looks like. And we have created a small tool to take an existing update and put it into smaller parts and also be able to create our own diff files.

The format itself has many different pointers and offsets. So in the example, you might already see the offsets here and compressed size. So it starts with a common header and then it's after the header, you have a LZMA compressed table of contents

or however you want to call it. And outside in the header, you have an offset into the decompressed version of this table of contents where actually file updates starts. And when you start playing or creating your own file, you might get the offset wrong and the update binary just crashes

with your malformed update file. So it's like not very robust code, a very complicated file format and nothing is cryptographically signed. So when you use strings on the binary, you see the word signature, but it only refers to a CSC32 checksum.

The next part, like now we understand how the update format works. We can create our own update files. The question is how do they end up on the device? And it's something that is implemented in the AT-Forward daemon that Harald or Lefort just mentioned.

And if you scrap for some specific strings, like Wget, this QC map connection manager or photo or update.zip, you already kind of guess how this application works. So you issue an AT command with a URL for updating it. In turn, it will disable your normal IP connection

that you have might established on your host and opens our PDP context on the device itself using the QC map connection manager. Then it will spawn Wget to download the file. Then it will use system to move it to the right directory.

It will remember what it wants to do with this file and it will reboot into the recovery partitioning system. And at this point, the update zip file will just be applied and the system will reboot once again. Again, without any cryptographically signature or checks. So if you manage to hijack the update process,

you can install any binary on the device or on the modem or anywhere else on the system as you want. But instead of just seeing how bad it is, we want to say like, what do we expect them to do? And I'm handing over to Harald again. Yeah, so rather than saying, oh, this is all bad,

it's all unlocked and it's insecure and so on. Well, it's fun for us, of course, because that's what we wanted, right? We wanted a modem device where we could do basically whatever we want to and where we don't have to break sophisticated security mechanisms that are designed to keep the user or the customer, the owner of the product out. So yes, there are security issues and those security issues must be fixed,

but we need security mechanisms that work without locking out the user or the owner of the device, of course. So this is our public call to the manufacturers. If you fix those issues, keep in mind that the openness of the platform is interesting for all kinds of legitimate use cases.

And while you want to protect against malicious attackers, you of course still want to enable the actual owners of the device and the users of those devices to use the flexibility they provide, because there's a lot of, yeah, thanks.

Yeah, so what's the status today in Outlook? Well, we have just opened the Wiki on the, it's on the last slide, there are all the links. So on the OsmoComm project, we now have a Wiki for Qectel, Qualcomm, LE modems, where all the information that I gathered from reading the thousands and thousands

of lines of source code is in there. We have released the debug tools presented in this talk in a Git repository and source code. The hardware boards are released as open hardware and available. And what's unfortunately still ongoing is the libqmi-glib integration,

which is, well, to some extent, to the fact that I've never written anything against glib before or anything inside a program that uses glib. It's like all this infrastructure. I'm usually more low level than that. And while we hope to grow this documentation and we kindly invite all of you with an interest in, well, understanding those platforms better

to help us out, you don't actually need to necessarily reverse engineer and disassemble things. It's just read the source code, understand what it does, and then play a bit with the device. We are planning an open embedded package feed, so we can actually easily install additional, everyone can easily install additional packages on those modems.

There's plenty of flashes, I think it's like 20, 30 megabytes of free flash, so you can install quite a number of additional packages in there. And our aim is to have free software only user land on this Cortex-A5 CPU. So to do away with all those proprietary processes that run in user space and the libraries and run the open source kernel and have basically

this libqmiglib integration and all other bits needed to run our own standard Linux user land code in there and have custom images that we can run on modems in for all kinds of use cases. Okay, now before we go for Q&A in a minute,

there's an unrelated announcement that we would like to present here. The OsmoCon project has gained support for running your own 3G 3.5G network during the last year. Not sure who has noticed that. Unfortunately it suffers a bit of a lack of contributions. So we want to motivate people to contribute more. And we have just started an accelerate 3.5G program

which provides 50 free 3.5G femtocells to people who can convince us that they would contribute something reasonable to our project. So these femtocells are already supported by the OsmoCon code. So using the femtocell and the OsmoCon code, you can run your own 3.5G network.

And we're giving those away for free. So if you're interested in any of that, please submit a proposal until the end of January and then you will hopefully receive your free femtocell until the end of February.

Which brings us to questions. Yes, it's Q&A time. We have a total of eight microphones here. One, two, three, four, five, six, seven, eight. I can do math. Please step up to the microphones to ask your question. Meanwhile we have from the signals angel, we have a question.

Yes, the internet wants to know if there will be in quotes, next open moco. Not from us, no. We're not working on a mobile phone. We're looking at modems, not at mobile phones at the moment. So, microphone two.

So just to clarify, does that mean that Linux runs on the iPhone 5? We don't know. We don't know. We'd be happy to hear from you if you can find it. But I think you can run the chip with something else on this Cortex processor. But we don't know. Signals angel. Yeah.

The internet wants to know why there are no mini PCX press slash M2 to USB 3 adapters because they think LTE is capable of at least 300 megabits and USB 2 could be a bottleneck. Well, I'm not really in the business of manufacturing

or selling adapters, so, I mean, yes we did this open hardware device out of a specific need, but you would have to ask the hardware manufacturers that, I'm sorry. Could I ask the people who are leaving to leave quietly, please? Microphone two. Can you buy the entire EC20 with the breakout board

and the UART as a single kit somewhere? Well, you can contact us, but it's not really something that we have prepared for, but yes, it's certainly an option. But I mean, we're not here to sell you anything. We're here to invite you to help us learn about modems. Thanks.

And with the Android debug shell, you actually don't really need the serial. So you can reflash the device using USB. Even if you break it, it can be reflashed. And with ADB shell, you have your root shell, so. That works, and with ATQ Linux command, you can actually start a lock-in on the NMEA console as well, so even if ADB doesn't work,

you can get a lock-in on it. So it's very easy to get started without serial. Signals, Angel? Hello, okay. Do you have, have you tried to get source code from other manufacturers? Well, we know at this point of,

I think three different manufacturers that use this MDM9615 plus Linux combination. From Qectel, we have just described how it went. Sierra has published all this already by themselves in a very good way, so there's no need to ask them. It's out there, you can just download it

and it's documented. And the third one is an old end-of-life Huawei module where we have also asked, but this is still ongoing. They are now, it's interestingly that from a Chinese supplier, you get the excuse, oh, there is Christmas coming up. It's like, okay, so many Christians in China, but okay. Well, I think I'm quite sure if I ask in January, you can say, oh, there is Chinese New Year coming up.

But we'll see about that. Microphone four. You mentioned the Qualcomm chip used in the iPhone 5. Does this mean, well, can I, is it likely that this, the iPhone application processor

talks AT commands to this Qualcomm chip? Is this still state of the art? No, it's not state of the art and I don't think it's happening. I mean, also on these modem modules, the AT command is there for legacy purposes. You have the QMI exported over USB and normally, like your modem manager

or whatever you use, or phono or whatever infrastructure talks QMI to those devices. I'm gonna go back to the signals engine again. Yeah, the question is, what is the total size of the flash or the root FS on there and how much RAM has this thing got?

I don't know, I don't remember the flash size, but RAM is 32 megabytes. Yeah, that's 32 megabytes of RAM. The successor, the EC25 has 128 megs of RAM, but a flash size I also don't remember right now. But check out the wiki, we have boot logs and all kinds of like DMSK outputs and so on all on the wiki,

so I'm quite sure it's somewhere in there. Microphone four. You said that you tried to check Legato Linux from Sierra. Wouldn't it be possible to completely build your own kernel there and your applications? Yes, it's certainly possible, but then, of course, if you do this,

you buy a lot of things with your money that you don't use in the end. And you support a vendor that tries to lock people into this proprietary framework, so yes, it's possible. But do you have to use a framework? I don't think so. No, you don't have to, but I mean, they're developing this and part of what you pay for

is this framework and that's why their products are more expensive, so. Microphone two. Yes, do you have some ballpark figures about the power consumption in the lowest power quiescent current mode? No, I'm quite sure it's going to be high. I think Holger has some comment on that. Yeah, so when you look at this device,

you use S trace against the QMuxT and it's like mostly waking up every couple of milliseconds, so it's not power efficient C code. It's really, really, really annoying to see processes that run all the time. And from the previous slide, this echo mem to sysfs,

that's actually the advanced power management. So you can trigger an AT command to have the device sleep and it writes echo mem into it to go to sleep. So it can be tuned and with a free software user space, it can probably be better than what it is right now. Signals angel again. How accessible is the baseband?

Like memory and like emi and stuff. We haven't really investigated this in detail, but we also haven't seen any signature verification and so on there, so I think it's completely open.

So yeah, the DSP firmware is in three separate partitions and there's no RSA signature at the end of these partitions, so it seems that you can modify it. If they have locked down like the poke command, that's something we haven't tried or looked at it.

Microphone one. Are the modules readily available in the LGA package rather than the mini PCI dev board, like if people want to start trying to use these in open hardware? They should be available again, yes. I mean, they were temporarily not available due to the GPR enforcement, but I think now they should be available again.

And what are the costs of those modules like? I think it's like 47 something euros around that region or maybe more than 50, but anyway, somewhere in that region, yeah. Okay, thanks. So one more question from the internet. Yeah, and the internet wants to know

if you can capture layer two network stuff with this. Yes, yes, you can. So I think that's the end of our Q&A session. Please join me in thanking LaForge and Holger on this fantastic talk about things

that I really didn't want to know about 3G modems.