Embedded FreeBSD on a five-core RISC-V processor using LLVM How hard can it be?
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 561 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44252 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
CoprocessorMulti-core processorReduced instruction set computingSystem softwareFunctional (mathematics)Reduced instruction set computingMulti-core processorCoprocessorSystem programmingComputing platformCompilerSpacetimeMultiplication signIntegerSoftware developerData managementProjective planeFloating pointFreeware32-bitComputer hardwareComputer animationLecture/Conference
01:10
Read-only memoryComputer hardwareComputing platformPrototypeComputer hardwareWebsiteWhiteboardMulti-core processorComputer animation
01:29
Computer-generated imageryPrototypeMusical ensembleComputer hardwareReduced instruction set computingWhiteboardMulti-core processorIntegrated development environmentPoint (geometry)MetreFloating pointCASE <Informatik>CausalityXMLProgram flowchart
02:12
BootingFingerprintKernel (computing)Wrapper (data mining)BuildingComputer-generated imageryNetwork topologyMiniDiscCompilerChainGDB <Programm>DebuggerReduced instruction set computingDisk read-and-write headVirtual machineDatabaseMulti-core processorBefehlsprozessorPatch (Unix)Address spaceThread (computing)Data storage deviceLocal ringReduced instruction set computingMedical imagingVirtual machineStructural loadKernel (computing)Workstation <Musikinstrument>Multiplication signPrototypeBootingLevel (video gaming)Physical systemMulti-core processorWhiteboardMiniDiscOrder (biology)ResultantAbstractionSource codeBefehlsprozessorBuildingSystem programmingNetwork topologyProcess (computing)CompilerServer (computing)Computer fileCompilerSingle-precision floating-point formatDisk read-and-write headWrapper (data mining)Library (computing)CausalityDebuggerOperating systemPoint (geometry)Stability theoryArithmetic progressionProjective planePatch (Unix)Game controllerDatabaseInterrupt <Informatik>Connected spaceUtility softwareChainComputer hardwareAuditory maskingDefault (computer science)Loop (music)GDB <Programm>32-bitComputer animation
08:01
Patch (Unix)Address spaceData storage deviceLocal ringThread (computing)CompilerEmulatorKernel (computing)Computer hardwarePhysical systemReduced instruction set computingSoftware testingServer (computing)Multi-core processorState observerCodeGeneric programmingServer (computing)Reduced instruction set computingFloating pointVirtual machineData storage deviceLocal ringMereologyCodeProcess (computing)Workstation <Musikinstrument>Computer programmingMultiplication signEmulatorCompilerThread (computing)Software testingCode1 (number)Level (video gaming)Independence (probability theory)Computer hardwareLibrary (computing)Position operatorPatch (Unix)Link (knot theory)ChainSoftware bugSoftware developerPhysical systemSystem programmingQuicksortMoment (mathematics)WhiteboardMulti-core processorCartesian coordinate systemVideo game consoleKernel (computing)BitCausalityBinary codeSerial portProjective planeError messageGcc <Compiler>BootingOpen sourceCompilerBuildingScaling (geometry)Point (geometry)Proper mapTrailComputer animation
13:50
ImplementationLatent heatMulti-core processorCompilerElectric currentLibrary (computing)Address spaceSpacetimeComputer hardwareComputing platformTerm (mathematics)32-bitReduced instruction set computingNumberType theoryBitCompilerInterior (topology)Software bugMulti-core processorLimit (category theory)View (database)ImplementationLecture/ConferenceComputer animation
15:42
Arithmetic progressionComputing platformReduced instruction set computingRow (database)Software bugRevision controlComputer hardwareCore dump32-bitSpacetimeLecture/Conference
17:32
Server (computing)BitOpen setPhysical systemSet (mathematics)Extension (kinesiology)Computer architectureCommunications protocolQuicksortOperating systemProjective planeArmMulti-core processorProduct (business)Focus (optics)Link (knot theory)Multiplication signEquivalence relationFunctional (mathematics)Finite differenceFeedbackService (economics)CausalitySystem programmingWritingReal-time operating systemCartesian coordinate systemSemiconductor memoryComputer programming1 (number)Computer networkInformation securityBoiling pointProcess (computing)Goodness of fitRemote procedure callReading (process)Lecture/Conference
20:28
Point cloudComputer animation
Transcript: English(auto-generated)
00:08
And away we go! Okay, thank you all very much for coming along. My name is Jeremy Bennett. I work for a company called Embercosm. You may well know us as a compiler company. More recently, we've started to move into the development of operating system software
00:25
for embedded systems, which is our speciality. I'm joined in the audience by my colleague Mark Corbin, who heads up that initiative. And this is the first time we've actually dipped into FreeBSD. We're more Linux, FreeRTOS space.
00:41
And this is me with a project manager's hat, talking about our experience. So this is the platform we're looking at. It's a quad 64-bit RISC-V processor. No floating point, it's an integer platform. And sitting alongside, it's got a 32-bit integer RISC-V platform,
01:06
providing some general housekeeping functions. This is proprietary hardware. The hardware is not yet available. So we've started work with the Hi5 Unleashed board, which is a five-core RISC-V board that you can get from Si5 Corporation.
01:31
And there's its schematic. It has essentially four cores, the U54s. And it's got a housekeeping core, the E51. In this case, it's a 64-bit core.
01:43
It's also got floating point on the core, which we're not going to use. So that's our hardware for prototyping this work. And before we even get to hardware, we're using QEMU, which supports RISC-V. So we've got a soft environment. And that's quite good. The Hi5 boards cost about $1,000 each.
02:02
They're scarcer than hen's teeth. So we've got one of them. And Marc there operates with full anti-static RISC bands when he's within 25 meters of it. So this is only a short talk. I'm going to whistle through things. We'll take questions at the end.
02:23
Broadly, if you want to build FreeBSD, you build your user world. And from that, create a RAM disk. You then build the kernel and tell it where to find the RAM disk so it knows where to jump to. And then, in the RISC-V world, we use the Berkeley bootloader wrapper, which adds in the device tree.
02:41
And the result is you have an image to load. And you can then put that on your SD card, plug it into your Hi5 Unleashed, and in principle, it all works. And to build that, we have a compiler toolchain, Clang LLVM, which is the default compiler, usually for FreeBSD, typically sitting on GNU binutils and the GNU debugger.
03:03
And if you're deeply embedded, you really want it to use it as a remote debugger running GDB server. And you need, at the very least, C libraries, probably C++, but certainly C libraries, those come from FreeBSD and we'll see why there are some issues around that in a minute.
03:22
And we took this project in incremental stages. The first thing was to make sure we can reproduce what the suppliers do with the board. So if you get the Hi5 Unleashed, it comes with a GNU Linux system on it and built with GCC. And the first thing is to make sure you can rebuild that and do that.
03:41
That's nothing to do with FreeBSD, that's just checking that what you've got on the board works. And there's a certain amount of hoo-ha, just getting that to work, but we found fairly straightforwardly the first day, we were able to get Linux built from source and put back on the board and it still behaved like when it arrived with us. The next stage then is to try to bring up FreeBSD.
04:01
But we did that with GCC and the reason for that is that the LLVM for RISC-V 64-bit is very, very new and we didn't want to be fighting compiler issues while we're trying to bring up the kernel. And so we first of all started with GCC and once we've got the kernel working, at least on QEMU, then we could switch over to LLVM and check it all works.
04:24
And so that's the loop we did for 64-bit RISC-V and then when we're going around it all again for 32-bit. So we want 32-bit RISC-V LLVM and so forth. So, what could possibly go wrong? 3BSD has already been ported to the 64-bit RISC-V.
04:43
Thank you to Ruslan Bukin who kicked off the project at Cambridge University and others who since contributed. And Clang LLVM is the FreeBSD default chain, so that ought to be fine and thank you to the people who've worked on that for RISC-V. But, what could possibly go wrong?
05:02
Well, like all things, you develop something, you document it, Ruslan did a good job and then someone picks that up and finds all the things you didn't quite write in the documentation. The documentation, as you'd expect for something that's developing, works with head, but quite often head isn't going to work straight away.
05:20
So, actually, if you're going to start from the first sketch, start from a stable point. There was another thing, it validated the users against the build machine and our build machine and the actual machine we're building for are different and that didn't work. And it used the machine user base database rather than the target database to build the file image. And those, when those were corrected, and those are fairly trivial things,
05:43
fires up runs multi-core FreeBSD on QEMU. And that was early stage, an early win, we'd got FreeBSD running on RISC-V on QEMU. But then we put it on the Hi-5 Unleashed board and it didn't work. And the reason is those five cores, the CPU0 is this funny controller core,
06:04
the E51 and not the U54s. And that starts up the system, then goes and masks itself out and goes away. So we had to modify the FreeBSD system to not assume it was running on CPU0. And with that fixed, up the Hi-5 Unleashed board came and it booted single core.
06:26
So now we've got our prototype system working. But then we ran into problems and this is where we're still working on. Multi-core boot doesn't work. When we try and bring up four cores, it usually doesn't work, except sometimes it does.
06:41
Sometimes it boots and it works. And the reason is there are four cores on the Hi-5 Unleashed and the order in which they decide to come up is random. And it's only if they come up in one particular order does it seem to work. That seems to be best if you've left the machine switched off for a while and then it starts.
07:06
So we're in the process of working on an abstraction layer to try and map the CPUs to come up in a consistent order, at least as far as the operating system sees them. And as far as we've done the debugging, and this is where it becomes clear,
07:20
this is the first time we've tried to do FreeBSD, there seems to be a problem with the enabling of interrupts and its connection when the interrupts come live, that's when things go horribly wrong. That's a work in progress. I had hoped that I would be announcing it's all working by the time we got here today, but we haven't quite got there yet. But we've still got a prototype hardware working on a single core
07:42
and that's quite good for a lot of things. But the actual bigger thing was, and why we originally got involved with this project, because our reputation as a compiler company for Clang LLVM. And we do GCC as well, we do both of them. And the problem we had is that we were asked to start on this project at the end of October,
08:00
which was when the first patches for Clang LLVM for 64-bit RISC-V were actually published. So we were trying to not just work with a Clang that wasn't ready, it was brand new. And we actually had to submit seven patches just to get a Clang that would actually build FreeBSD. A PC relative addressing wasn't working, position independent code wasn't working,
08:24
thread local storage wasn't working. So that got us to the stage where you can build. Since then, my colleague Lewis Revel, who's actually sitting in the RISC-V dev room at the moment, who's not here because this relates to RISC-V as well, has submitted another nine patches. So we've had to put 16 patches up there.
08:41
We think there's one bug left. Well, there's one thing we think that's not working, we hope it's just one bug. But actually, this has been a great driver and thank you to the customer who's paid for this work because these open source projects often rely on people actually paying us to do the work. 64-bit RISC-V LLVM has actually progressed incredibly fast.
09:05
So we got it all compiling, the compiler seemed to be pretty much good and it still would not run. And the reason came down to compiler RT. Now for those of you who don't know, compiler RT is the low-level emulation library for LLVM.
09:20
So if you don't have hardware floating point, you do it in emulation and you do it in compiler RT. And RISC-V is quite clever. If you've got a machine that doesn't have hardware floating point, what it generates is hardware floating point instructions and they cause a trap which goes into compiler RT and then you emulate them. The problem was that one of the compiler RT programs actually doing the emulation
09:42
had hardware floating point instructions in it, so promptly trapped and called itself again. And shortly after this blew up, once we'd realized what the problem was, it was fairly easy to fix. And when we get that there, then FreeBSD runs quite well. You can fire it up, it boots up and it all works nicely.
10:03
But after about 10 minutes or so, it tends to lock up. And it's a kernel issue. There's something with the compiler and the kernel that doesn't work because you can build the kernel with GCC and everything else with LLVM and it all works just fine. So we think there's an LLVM compiler bug. Possibly it's a kernel bug that LLVM is technically correct, but only LLVM is exposed.
10:26
So we're still trying to track that down. Three to five days more work, we think. Now those awkward compiler bugs can take a bit of tracking. So we've got to fix that. So far so good. We've got a compiler.
10:40
It more or less works with LLVM, but we've got a bug to fix. But everything else outside the kernel works with LLVM. And it runs one core on a particular hardware board. But we're not in the business of, hey, it sort of works. We're actually into serious professional development. That's what we're paid for.
11:00
So we want to test. And CURE is the FreeBSD test system. And one thing we found with CURE, it really doesn't build cross-compile build. It expects to be built native. So we had to create a compiler toolchain to run on the actual QEMU. So we had to build it so we can cross-compile the compiler toolchain. And then we had to run that GCC compiler on QEMU emulating RISC-V to build CURE.
11:25
Now CURE is quite big. The RISC-V emulation is quite slow. And so we had to leave it several days. That binary is now built. It is many copies. It is locked away in a vault in one copy. So we don't lose that binary. We do not want to have to build it again.
11:41
But the great thing is we now have CURE running. And we can do nightly testing. And it takes about five and a half hours in emulation on a 20 core Xeon server. But that doesn't matter. Because we're asleep at that time. It can go away warming up the machine room. And we can nightly test. And that's good. There are bugs to fix. But broadly FreeBSD is working.
12:04
OK. So what else could go wrong? And the answer is debug. So at the moment we have to debug by putting GDB on the target. And GDB works fine on FreeBSD. But that does presume on a deeply embedded target.
12:22
One we have a fairly fully featured FreeBSD. And secondly that we have the sort of console where I can sit there and it will put up something I can browse on and so forth. And for a deeply embedded system particularly in a secure application which this is. You tend to not want to have more on there than you actually need.
12:43
And that's why for embedded systems you have a program called GDB server. And GDB server runs on Linux. And it runs on Linksos. And it runs on one or two other things. And it used to run on FreeBSD. But no one's run GDB server on FreeBSD for a long time.
13:01
And when you run GDB server then that small little program which doesn't need much operating system resource runs on your thing. And you connect via a serial line or TCP IP to GDB proper running on a workstation somewhere else. And that's the standard way to debug an embedded system. And the code's been taken out.
13:20
So actually we're in the process of rebuilding GDB server, reconstructing it for GDB. It will be a new better GDB server because it will reuse all the FreeBSD from native GDB. And there's two parts to that. One is to get it working for x86. And then we will port that to RISC-V.
13:41
That's a few weeks work. And because it's not critical to bring up this project which is on quite a tight time scale. It won't happen till later in the year. We do need it eventually. We can live with native GDB for the short term. So that's the last thing that we had to fix. So where are we? The answer is we can run embedded FreeBSD for RV64 built with LLVM.
14:05
There's a reference implementation for the Hi-5 Unleashed. But it has some limitations. It only works on a single core on the Hi-5 Unleashed and we're still trying to nail that down. And if you're an expert or want to help us, we're very happy. We've got money. We can hire people to do that.
14:20
So for people like, you know, who need to be paid to do stuff, we understand that. It's a perfectly reasonable requirement. We can do that. So come and talk to me. And it's unstable with LLVM and we should knock that compiler bug out in the next week or two. OK? I would like to have been able to say that the reference implementation will be available there. I'm afraid I've been traveling and it's sitting there and I haven't actually got it onto GitHub where people can help.
14:45
It will appear in the next day or two. If you need it, please shout at me because that will be a motivation to make sure we don't forget to do that. OK? So where are we with FreeBSD for 32-bit RISC-V? There are a number of issues here.
15:02
One is that we can't pick up someone else's work who's done the hard bits for us. So we're actually, Mark's been plowing through, making sure all the bits are needed to run on the 32-bit RISC-V. And because of the bug with LLVM, we thought we'd go with GCC.
15:21
And GCC doesn't work because 32-bit GCC doesn't have an INT128 type. And that's rather baked into compiler RT and GCC doesn't believe it should be there. And GCC, unlike LLVM, doesn't have quite such a fluid view of what types are. We're working around do we make that conditional inside compiler RT or do we just say let's go with LLVM and hope we fix the bug soon.
15:48
And the other thing is we don't actually have a suitable RV32 hardware platform. We've got a software platform obviously in QEMU and we're not quite sure. If anyone's got a good suggestion of a 32-bit RISC-V platform that has an MMU on it, that would be good.
16:05
So that's work in progress. Watch this space. But we will bring FreeBSD up for 32-bit RISC-V. And so that is my talk. It's a whistle-stop tour. Questions? Yes?
16:34
So the answer is we certainly have panics. Have you been looking at them with GDB, Mark?
16:58
So the question was can we, for the record, is can we use GDB to look at core dumps
17:10
because there's been a problem with that on AMD and the answer is we haven't had to use that yet so we don't know if there is a problem. Other questions? Yes.
17:26
So the question is do we see commercial interest in a 32-bit version of RISC-V? And the answer is yes because we're being paid to do this work. And the product we're going and it's very similar to many other RISC-V architectures
17:40
which is where your core engine is a set of RISC-V 64-bits and sitting alongside you have a 32-bit sometimes engine doing housekeeping. It may be doing slow speed networking. It may be doing timers, watchdogs and so forth like that. And that's a setup, for example, used in the low-risk project.
18:00
And commercially you're probably using FreeBSD because you're concerned about security network performance. And if you're going to have FreeBSD on your main cores you don't want a different operating system on your housekeeping core. So you want FreeBSD everywhere. So since that's quite a common architecture setup I would expect that to be a requirement.
18:23
And actually though there has been a focus on the RISC-V foundation on 64-bit operating systems the feedback we get from customers is if you're embedded actually there's still a lot of people who want to use 32-bit there. Any more questions?
18:42
Yes. Good question. The question is, is there an equivalent of GDB server in the LLVM project? The equivalent to GDB in LLVM is LLDB. And we get asked about it the whole time.
19:01
There are things that LLDB does quite well like native debugging on x86 and to some extent on ARM for Apple Core. You can do remote debugging with LLDB. We keep on looking at it because we'd really like to be able to do it and offer it to people. My feeling still is that it's quite a long way behind.
19:22
On the whole GDB servers come in all sorts of flavors. GDB talks a very simple serial protocol to say read memory, write memory, start program, stop program, read registers, write registers and that's basically a GDB server sits there processing those. And the GDB server application is a Linux application
19:43
that's been ported to Linksos and used to run on FreeBSD that provides that functionality. There are lots of other GDB servers OpenOCD is an example for embedded systems. We have our own one for embedded systems when you're running bare metal or with real-time operating systems.
20:00
So I think it boils down to there isn't really but Apple sort of have one that does that sort of thing and there are other ones out there. But it doesn't really help us. The solution is to get GDB server running and it should actually compile with LLVM in the end anyway. And we're out of time. Thank you.