and I'll show you a number of examples and how you work around those. So, if you see some of these bugs, it's because you have a lot of users, so you should be happy if you see these bugs, and also because I provided the solutions for them. So, why does it matter?

We are, when we have Android, we are actually seeing a lot of old devices. 25 per cent of the active devices last month were from pre-5.0 Android, so that is a huge user base, so if you say I don't care about those old devices, no-one has them, yes, they have,

and if you want to exclude 25 per cent of potential users, that's of course your business, but we try to use in as many apps as possible. Of course, as a developer, Android developer, you always have the new shiny device. You probably buy a new phone every second year,

or every year, you have the top-of-the-line phones. People in rural countries, in areas in China, India, South America, they don't have the newest shiny phone, so you have to,

if you want to have those people as users, you have to take care of that as well. So we will see these, some of these bugs I'm showing will only show up on very old devices that are not sold any more, but people are still having them, they're not upgrading them, because they cannot afford buying a new one, and they're not upgrading because it's

just a phone. Do you upgrade a phone? Because we are software developers, we upgrade our phones all the time, but common people, they don't do that. Our users don't do that. So that's why. So, what I'm going to cover, well, the first one is that if you have

native code in your app, like C++ code, then you have a shared object, and you have to load that at some point. If you're using, for example, Realm, you have SO files in there. It might not be you who wrote them, it might be other people who wrote them, the library developers have written for you, but if you cannot load them, your app will crash.

So I will go through that and what you can do about that. Then we have a case where our encryption, Realm is supporting encryption, and our encryption was failing for some reason on a few devices. I will go through what was why that happened, and why I can already

give you a little hint that Google fixed it really quickly for us, because they made a bug. Then, actually, when you have a context, and you want to find the app's directory, so you can write files, there is a method for that, of course, on the context, but

sometimes, an app cannot find that directory, and I will go through when that happens and what you can do about it. And then we also have some amusing, in the beginning they were not amusing, but we have random crashes on a few devices, and there was a talk yesterday

where Samsung was mentioned not in a good way a few times, I think, and we also have in our horror cabinet, we have this Samsung device that we have crashes on. It's a very old device. You cannot buy it any more. But I will go through that, and that is a really

amusing story, and especially when you see the solution, you will say, wow, I didn't know that you could do that, but you can, and I will show you that. So let's start with you cannot load your ISO file. So the joy of having native code or C++ code

is that you have this shared object, a library file. So what basically you have to load that at some point, and in Realm, we do that through our init method, so when you use Realm in your system, in your app, you call this in the beginning, and that will kind of set up Realm, and what it basically does is it loads the ISO file. It happens

that, underneath, Android has this system load library function that does that. It is calling the dynamic linger, and it does the trick there. And APK contains the ISO file for all the architecture you support, and doing installation, it will pick the one

that is specific for your device, and copy that ISO file in, and when it's there, when it's there, you would think. But sometimes, you will see we had a bug report, you can

see there's a GitHub issue for it. We have that. It seems that the package manager does not always install the ISO file, and we couldn't really understand that. That happens sometimes. Google hasn't really been very responsive to that. It seems that the package

manager in some cases does not copy the ISO file. APK files, and zip files, and copy the text files, and all the resource files, and asset files, and also the ISO files, but sometimes it doesn't do it. And no-one really understands why. So, what we did,

our solution is that we are using re-linker. It's done by keep safe, and they contribute to that. What you will see, you can see the down here, you see this unsatisfied link error. When you see that, then you know that something is wrong with the ISO file. Just use re-linker. It solves most of the cases. What re-linker actually does is that, if you cannot find

the ISO file, it actually finds the old APK files, unzip it, and finds the ISO file in the unzipped file, and copy it in and load it. So it actually does the re-do the installation once again. I went also through the, before we settled on re-linker, I also

went through and see what Facebook is doing. If they cannot find the ISO file, it fails. It actually loads the ISO file from one of their servers. So they're actually doing it really complex as well. Just to be sure that you can load your ISO file. So just use re-linker. If you're doing native code, use re-linker. That's a good option.

But you might still have problems. We have another issue on that. If you're mixing 32-bit and 64-bit shared objects, you're really in trouble. Because the dynamic linger, well, if you install on a 64-bit device, of course, it can still run 32-bit code. So if you

only have two only ISO files in 32-bit, everything is fine. But if you have a mixture, it will try to load the 32-bit, and then it will try to load a 64-bit version of the ISO file,

and it will say, oh, there's a mismatch of architectures here, so I just crash. Of course, Realm is supporting both 32-bit and 64-bit. Because we want to support as many platforms as possible, and also for performance reasons. But things like Unity only support

32-bit. There's also parallel space. Parallel space is very much popular in China, because it allows people to share their phones so they can have different users on the same phone. So they live in parallel worlds, so to speak, or parallel spaces. And they only support 32-bit. Renterscript is also one of them. I think they're working on a 64-bit

version at the moment, but it's not there yet. But if you're using Unity in your app together with Realm, you will be in trouble. The trick is, basically, that you exclude the 64-bit version and runs a 32-bit version only, and then it works fine. So, if you

see something like one of these unsatisfied link errors there, and that was from the visual client, the VLC, they had that issue as well. And that was actually reported, because they were only compiling the 32-bit, and someone was trying to combine VLC in the

app and Realm in the app, and so they got into trouble for that. So, just exclude 64-bit if you see this. A lot of these old third-party libraries are only using, only

supporting 32-bit architectures, so you see this very often. Even though you don't know that you're actually using SO files or having native code in your app. Crashlytics are fine. They are supporting 64-bit, so we can easily work together with them. I checked that, of course. So, that is some of the issues you can run into.

So, just be warned. We saw that when we started supporting 64-bit, and everything was fine for a while, and then someone was using parallel space in an app, and

we have to dig into that. Anyway, that was some of the easy ones. Well, we also, as I said, Realm support encryption, and it was failing at some point. It was working fine for a long while. Everybody was happy. What you do in Realm, you just

supply an encryption key, and then it just encrypts your data, all your database, everything is fine. No worries. But, at some point, we started crashing. We couldn't really understand why. It was reported in April 2015, and it seems that

it was a segmentation fault. Segmentation fault you only get in native code. There is something done in C++ code that when you reference going into some memory, you are not supposed to look into. So, there's a pointer pointing in the wrong direction, pointing into memory that you are not allowed to read or write. So we are seeing those.

And it seems to be the cookie manager and encrypted ROMs, and it seems to affect only two versions of Android, and we couldn't really understand why. Then we realised, after digging into it very closely, that it was related to using signals. How many are old-fashioned

Unix programmers that are using signals ever? There's a few. Signals is a very old technology in Unix, and Android uses Unix, basically. So a process can send a signal to another process, or internally in the same process, saying something. One of the very often you

probably try to kill a process where you type kill on command line, then you're sending the kill signal to the process so it dies. You can send all sort of signals to them and you can define your own ones, and we're using that so when we are writing, trying

to access something that was encrypted, we will get a page fault, and that was a signal from the operating system that we are accessing this page, and then we knew, so we had a be encrypted, or if we are trying to write it, we have to encrypt it, so we are relying

on these signal handlers to do the encryption. It was a very smart way of doing it. The problem is that you have to be very careful when you are working with signal handlers. The user that had the problem, he was able to write a small test, and he called

it make ROM crash, and there was actually the code that could crash ROM. So, the user could do that easily. And so we just had to look into that. There was a very simple test and we could use that to debug it. Using signals requires a lot of discipline.

Because when you have these signal handlers, you have to pass on, if you have multiple signal handlers, one signal handler has to resend the signal to the next one so everybody kind of gets the signal. So if you break the chain of signal handlers, only the first one will get the signal, and the rest will not. Unfortunately, WebView 40 didn't send

the signals. It was receiving signals, they have a signal handler, but they kind of forgot to resend it. So we never received it, because we were later in the chain, they put themselves in front, so the first signal handler, and they didn't resend it. So, we never received

it. Yeah, well, we reported to Google, there's a link to the bug report, and they fixed it within a month. They were really quickly fixing it. They could see that they were not playing well with the community, so they had to fix it really quickly. And we

kind of see, could see that if Google could do this wrong, other libraries could do this wrong, so we might end up not using signal for our encryption. So we rewrote our encryption layer, basically, because of that. So we're not using signal anymore, but if you have native code, using signal, you might have to rethink about it, because

people are breaking it. We've seen that. Crestlytics is also using signals because they are receiving the signals and making a report and sending you the bug report. That's what they're doing. Basically, that's their business. And they're doing it correctly. They're resending it. But they state in the documentation that if you're including

their library or their system into your app, don't use signals. They say that. Because you never know, as I say. So, be careful about signals. Because when people are making

mistakes in their code, you will be affected. Even when Google are involved. Well, this one is kind of amusing. When you have this get files here, the documentation says it can never return null. It can never return null. But it can. And you see something

like unable to open a round, blah, blah, blah. Because it couldn't we were assuming that the documentation was correct, so we were getting our files here saying, okay, we make a directory where we can store the round files for you. So now, but if

you get a null back and try to create a directory within null, that will not work. Because that is a null pointer exception. So, we have this problem. So, you can see that people are running into that. We never seen it. As you can see, we had Realm out

for two and a half years at that point. And we have never seen this before as an issue. That we couldn't create that. And we realized that get files here could return null and we never seen that before. So, we are apparently hitting some devices that we never seen before.

It turns out it was actually a known bug. It's a condition in how when the app starts, it has to settle all the things within the app. And there's a condition. And it was fixed in Android 4.4. Okay. So, it was apparently a device older than 4.4 we are

seeing it on. I mean, it was fixed in 2010. And we saw it in this spring. I mean, that is something we see. Also, the issue tracker for Google, they fixed it. So, they

closed the issue. For them it was gone. Not that problem. They fixed it. The race condition. But since we have these old devices, remember, we are talking about 25% of active devices that might have this bug. That's quite a few. So, we are thinking, what should we

do? And since it was a race condition, it's just a matter of waiting, basically. So, we ended up doing a workaround that we tried to create that directory for us, and tried to get a number of times. We had to try up to 200 milliseconds to do that, and eventually

we have to give up, because we cannot just keep going and say, oh, let's try to wait, and then we are blocking the app and nothing will happen, right? So, we are trying and after ten attempts or something, we just give up, basically. But that's our workaround.

That is if get files to return null, wait a little while and try again, and again, and again, that's the workaround. It's not a solution, it's a workaround, basically. But there's that bug out there, and just so you're warned about that. Have any of

you seen that? Ever? There's one down there. Wow. It was a known bug, but it wasn't known to us to begin with. Anyway, we also have some random crashes, and I mean really

random crashes. They came out of the blue. This is the bug that I spend most time trying to solve ever in my entire career. It was reported in October last year, and in April, we were able to send out the fix for it. So from October to April, I wasn't working

full-time on it. I returned to it quite frequently, and I think my team lead is here, and I'm pretty sure he was kind of being annoyed that I spent time on trying to fix something that was only seen on a few devices. I still said, yeah, but, remember, we had

to be a good citizen in the community. We had to fix our users' problems. So I keep trying to do that. And it's also not only me that was spending time, but we have this in Realm, we have our core database, which is written in C++, and we have a team

of dedicated C++ programmers, they have no clue about Android, they have no clue about OS, they are C++ programmers. Eating templates at breakfast. I mean, they also spend time

on it. One of our lead in the core team has been building CPUs when he was younger. He was working for MIPS building CPUs. He knows quite a lot of these things. He was also spending time on it. Anyway, what was it? Well, we got these

stack trees. We got a segmentation fold in a method called array strings set internally in our storage engine. Which is really strange because this is the core of our technology when you store a string in our database, you call this method. Well, you don't, neither

do we in the Android team, but internally, underneath, this is the last bit part of storing a string deep, deep, deep in our storage engine. So, if there was a bug in

there, that was kind of strange because we have been testing that method on iOS, on Android, on Linux, on Windows, everywhere for years because it was part of every unit test we have. Every time we write a unit test, we store a string somewhere. Even when we create a table or a class, we have to use that to store the information

about the name of that. So, we're using it always. It's probably been tested billions of times. So, how could there be a bug in there? And we have this, as you can see, it's just that that was the last one. It's crashing there. So, there's an address, 176. Please

remember that. That will be important in a few slides. Remember that address. Anyway, so we have this. And what should we do about it? One of the things is that I couldn't

I was trying in the emulator to run our unit test. It couldn't crash in our emulator. It couldn't crash on my phone. I have a OnePlus One and it couldn't crash there. It seems to only be a few devices. It was one of them was a Samsung tablet, which is this one. We managed to find one used buying on what is the same as eBay in Denmark.

We figured we could buy one. It's an old device. It reboots randomly. It's hard to debug something if it reboots during a debug session. Believe me. I'm pretty sure it wasn't expensive.

I only took it here just so you can see the horror. Only that. The problem was that none of our unit tests was failing. None. And still people have problems. Even on this

device, our unit test was not failing. Both the core unit tests which are for the engine, the storage engine alone was not our Android unit test. None of it was failing. Only the we have and in the previous talk, it would mention that you should do when

you're doing the SDK, you should do some demo apps. We have one, the intro example. It's a very simple one. And that was crashing. Only that was crashing. So, it's kind of hard to debug using your app. So, you have to don't have unit tests that are failing. So, only your demo apps. Luckily we have a demo app, one could say.

And as some of you if you are working on C++ on Android, you know that the debugging capabilities are not that great. And they have improved in the last year. But I mean, if you go two years ago, back, the only thing you could do is you could write something

to the log and hope that you get some information out of that. So, it was very hard. I mean, I have no clue. So, I was just playing around. The message that is called in the just before this set array set was we are creating two meter tables. We are creating two tables

to take care of some of the information, the meter data about the file. There's one called meter data which is storing the schema version basically. And the other one is called PK which tracks the which columns of which properties are primary keys. Do we

have some internal tables for that? They're not visible to any users. They're hidden. But we have those. And it was crashing. Actually it was crashing. So, that was when we were setting the first string for the first time. Calling this method for the first time. I was crashing there. So, I was just I had no clue what should I do. So,

what I tried was I just swapped the order I was doing creating those two tables. And then it worked! It's not very nice. Then you know that what there's and I'll come to that. There's two branches in that array set method. And apparently one of them was

crashing and not the other. So, I quickly since it was kind of solving the problem, I was we have no clue why. We released it. So, people could at least work. But still

people were kind of saying, oh, I still see the crash on another device. So, there was not really a solution. But it solved some of them. The way Realm stores data is that it tries to store as little as possible. So, what it does, in the original code we had the PK first and then the meter data. And what it does is that it stores

PK first. And then when it has to store something that is longer, it actually expands the old strings. So, there's only one string. So, it's expanded and puts zero characters in there. So, it just expands it. So, it expands when it has to. Of course, if you do it in the other way around, then you have a length of 8 and then

you don't have to expand. Then you just have to add zero characters and store that. So, there was two different branches in how you store the strings. At least that we learned from that. So, I was digging further.

See, I was really desperate. I was looking through the code and it do a lot of pointer arithmetic. And if you're not really sharp in C++, which I'm not that good, doing a lot of pointer things and adding things to a pointer and dereference it, it's really hard to read.

So, I asked my colleagues in the core team to help me there. And I also tried to disassemble it, to look at the assembler code. Because it was only affecting ARM. 32-bit ARM. All the devices were 32-bit ARM devices only. So, I decided just to disassemble it and see

how does it look. And then I realized at this address, 176, it seems to be calling something called map move. It was around that. Map move. Map move is one of the old, very old Unix technology. It has a signature of this. It's back from 1990. It's really old

technology. My good old friend was so kind to send me the link to the commits in the FreeBSD 3 source code where it was added for the first time. And it has a sibling

called map copy. And what it does is that underneath it, it copies memory, basically. A map move and a map copy, it depends on you have overlaps or not overlaps. If you

copy a chunk of memory from one place to another, or it can actually overlap with itself, basically. So, I started digging into that. I could see that there was a guy who actually figured out there was a map copy bug. And it was also reported by QT and Unity. They also

have the same bug. And it was, as he said, I spoke to him at some point and he said, it's probably a race condition in the Linux kernel. And it was fixed in 2013. It was

old devices, as you can see. These old bugs. And it was fixed in the kernel. Okay. I cannot ask people to upgrade the kernel. So, what can I do? Well, the work around is that you roll your own map move. Map move is highly optimised by vendors.

They write them as simpler. They do all sorts of things. Every operating system is doing that. If you look into the FreeBSD kernel, the Linux kernel, all of them have their own specialised version. Of course, Samsung has their own specialised version of it as well. Everybody has that. So, what we decided to do is that we took a reference implementation of

map move. It's slow as hell. But it works. Actually, the source code is so short, you can just read it and say, yeah, it works. You don't even have to test it. You can see it works. Because it's only a few, maybe 20 lines of C code. It's so easy to read

and you can see, oh, this will work. So, we took that. And then, GCC has a very nice option. You can take all the building. They have a lot of buildings like map move. There's a lot of these buildings in the C library. You can swap out with your own version if you want to. So, there's an option in the GCC. I never heard about it. I've been

using GCC for almost 30 years. And I never seen that option ever before. So, you can actually swap out and use your own. So, you just what you do, basically, is that you check if you have the problem here as you can see in check map move. And what

you do is that you just swap the pointer to the function. So, you take the one that are supplied by the compiler and say, it's not working. So, I just put in my own version instead. So, that's the way you do it. So, what we do is that on devices, especially

32 with ARM, because we know they are the only ones that are affected, we check that when RAM starts, when you call this RAM init. So, we check do we have a move issue? Do we have a bug in there? If we have, we just roll our own version. It's slow,

but it works. So, we have this slow device that are now even slower. Maybe that would get people to upgrade it. But at least it works. And everybody's happy. So, we're able to close that bug in that way. So, even though it was a bug that was fixed

four years ago in the Linux kernel, it hadn't reached devices yet. Think about that. Which is actually my conclusion. If you have enough users, you will literally run on every

device ever made, basically. And you will see weird things happen. Because you will still be facing the old bugs. And Google, for example, they will just close them. When they fix them, they will close them and say, oh, it fixed an Android. So, you also have to look for closed bugs when you're doing these searches. Because that was what I was doing wrong in the beginning. I was just looking for the open box. No, you also

have to figure out if, okay, it has been closed, but it hasn't been on the device you are actually debugging on. Yeah. So, that's it. I didn't do all this alone. So, for this move, the last one, we had this Taiwanese developer who did a lot of work.

He analyzed it. He was in a completely different context. He was doing it. Analyzing it was something different. But he was doing a very good job. And he actually got me to understand what it was. Then we had two users. I never figure out who this

GitHub user is. Probably not his real name. Diego is something else. And then we have I think he's a German guy, Jonas Bach. When we built the new stuff, he was trying in his app to see if it was actually working. It was a company here in Germany that was

actually using this device on their CI system. So, it was very good. So, we just supplied him with new versions and he was testing on his CI system as well. So, we were sure that it actually worked for users as well. And, of course, internally, Christian

and Thomas was doing this encryption thing, my colleagues. Christian is based in San Francisco and in Beijing. So, they worked together on that one. The work around for the get files here is Christian who is in the audience. He has a talk later today. So, yeah. That

was it. Any questions? Thank you, Kenneth. If you have questions, go to the microphone. We have two here in the room. Yes? Looking at the box you were telling us about, three out of four were due to CC++

code. Yeah. But if you're you might have seen it anyway because other people are using C++ code. For example, I could see the QT had the same problem with this move and was reported by a user that was using QT originally. The same for Unity. They also

have this move problem and it was also reported by a user saying, oh, I see this in this stack trace. So, it was reported by users originally. Yes. But it was only CC++ that could fix it. Yeah. Just a second before the next question. Is Jason Feinstein in the room?

Yeah, okay. Hi. Thanks for the nice talk. I'm also programming native C++ for Android and came across some crazy things with the NDK and also hit some bugs. And I just saw that there's much going on in the development of the NDK.

Yes. The switching to ceiling and all this stuff. Yeah. So, my hope is that this will get better some in the future. Oh, yeah. Yeah. What's your thought on that? And also the NDK team has also been expanded recently. They have hired a few extra guys. Yeah. Okay. We are in close contact with them. Okay. Thanks. Believe me. One more question.

It's very fascinating stuff that you did there. I can imagine how hard was it sometimes to find the solution. And obviously you also addressed some bugs that affected very small amount of devices. Yes.

Also probably devices that the amount of those, the amount was diminishing. And the bugs that were sleeping for years there, right? So, you obviously didn't go for 80-20 rule. Was it because you are a library, you

want to serve as many cases as possible and want to get the best publicity possible? Yes. As an application developer, I definitely wouldn't go for that 1%. If you have as a goal that ROM should be the default database for mobile developers in general, iOS, Android, then you have to take care of those

problems if you want to be the default. Okay. And that's a goal, right? Thank you. We want world domination. Thank you. Thanks, Kenneth, again. Can I have one more? Yes, of course. Because you mentioned, you know exactly which Android version is used when the bugs occur. So, you have some tracking like crashlytics-like

solution. Did you develop it on your own or you use crashlytics or other? Which one? I wonder, because you specifically know this bug is on this device. Very often the users, the people are using ROM in the apps are reporting that they're seeing on which device.

So, it was only from the bug reports? You don't have tracking in the ROM? No, we specifically ask for that when people are sending a bug report. Because we know that we might have to go out and buy a device. And that happens. We do that on a regular basis for that.

Thank you for your talk. Very interesting.