you have been here on stage before, you successfully tempered with the Wii,

you successfully tempered with the PS3 and got some legal challenges over there, some unfounded legal challenges yes, and then you fucked an excuse my French over here by the way that is number 8021 to get the translation on your DCT phone, so you fucked with the Wii U as well, and well console hacking 2016, here we go,

so I'm a lazy guy so I haven't turned on my computer yet for the slides so let me do that, hopefully this will work, so my computer is a little bit special, it runs a lot of open source software, it runs FreeBSD,

it even has things like open SSL in there and Nginx and Cairo I think and Webkit, it runs a lot of interesting open source software, but we all know that BSD is dying,

so we can make it run something a little bit more interesting and hopefully give a presentation about it, let's see if this works, it's a good start black screen you know, it's syncing to disk and file system shutting down,

there we go, and yes I run gen 2 Linux, this is the does wi-fi work moment,

hopefully NTP failed, well that's a bit annoying but it still works, hello, yeah it takes a bit to boot, it doesn't run system D you know, so

it's sane, it's a tiny bit slower but it's sane, there we go, this is the does my controller work moments, bluetooth and the install one, okay it does, all right so let's get started, so this is console hacking 2016 ps4 pc master race,

I apologize for the horrible nazi joke in the subtitle but it's a reddit thing, so yeah pc master race why, well ps4 is it a pc, is it not a pc, but before we get started

I would like to dedicate this talk to my good friend Ben Beyer who we all knew as bushing, unfortunately he passed away in february of this year and he was a great hacker, he came to multiple congresses, one of the nicest people I've ever met, I'm sure some of you have met him would agree with that, and if it weren't for him I wouldn't be here, so thank you, so the ps4 is it a pc, is it not a pc, well

it's a little bit different from previous consoles, it has x86, it's an x86 CPU, it runs free bsd,

it runs webkit, it doesn't have a hypervisor unfortunately, then again the ps3 had a hypervisor and it was useless, so there you go, and so this is different from the ps3, but it's not completely different, it does have a security processor that you can just ignore because it doesn't really secure anything, so that's good, all right so how to own a ps4,

well you write a webkit exploit and you write a free bsd exploit, right, and everything runs webkit and free bsd is not exactly the most secure os in the world, especially not with sony customizations, so this is you know this is completely boring stuff, like what's the point of talking about webkit and free bsd exploits, instead this talk is going to be about something

a little bit different, first of all after you run an exploit well you know step three something step four profit, what is this about, and not only that though before you write an exploit you usually want to have the code you're trying to exploit, and with webkit and free bsd you kind of do but not the build they use and it's customized and it's annoying to write an exploit if you don't have you know access to the binary, so how do you get the

binary in the first place, how do you dump the code, that's an interesting step, so let's get started with step zero blackboard code extraction the fun way, a long time ago in a hackerspace far far away failure of all got together after 31c3 and we looked at the ps4 motherboard and

this is what we saw, so there's an aovio southridge that's a code name by the way, then there's a liverpool apu which is the main processor, it's a gpu and the cpu which is done and it has some ram and then you know the southbridge it connects to a bunch of random crap like the usb ports a hard disk which is usb for some inexplicable reason the internal disk on the ps4 is usb like it's sata to usb and then to usb on the southbridge

even though it has sata like what, the blu-ray drive is sata the wi-fi bluetooth ssdio and the ethernet is gmii, okay how do we attack this, well dddr what just oh i have a screensaver apparently that's great i thought i killed that let me kill all that screensaver real quick

something had to fail it always does i mean of course i can ssh into my ps4 right so there you go okay i could have sworn i fixed that anyway so yeah which one of these interfaces do you

attack well you know usb sata is the io gmii that's the you know the raw ethernet interface by the way all these are cpu controlled the cpu issues commands and the devices reply the devices can't really do anything so you can't write to memory or anything like that you can exploit usb if you hide the bug in the usb driver but we're back to the no code issue so

ddr5 that'd be great we could just write to all memory and basically own the entire thing but it's a very high speed bus it's definitely exploitable if you're making a secure system don't assume we can't own ddr5 because we will but it's not the path of least resistance so we're not going to do that however there's a thing called pca express in the middle there that's interesting pcie is very fun for hacking even though it might seem intimidating because

it's bus mastering that means you can dma to memory it's complicated and complicated things are hard to implement properly um it's robust uh people think that pcie is this voodoo high speed no it's not it's high speed but you don't need like match traces to make it work

it will run over wet string like you can hot wire pcie with pieces of wire and it will work at least at short distances anyway like believe me it's not as bad as you think it's delay tolerant um so you can take your time to reply and the drivers are full of fail because nobody writes a pcie driver assuming the device is evil even though of course everybody should because devices can and will be evil but nobody does that so what can we do

well we have a pcie link let's cut the lines and plug in the southridge to the motherboard to a pc motherboard that we stick on the side so now the southridge is a pcie card for us and we connect the apu to an fpga board which then can pretend to be a pcie device so we can man in the middle of this pcie bus and it's now times one width instead of times

four because you know it's easier that way but it'll negotiate that's fine so how do we connect the motherboard on the fpga um there's of course many ways of doing this but how many of you have done any hardware hacking even arduino or anything like that raise your hand i think that's about a third to a half or something like that at least

and when you hack some hardware you melt some hardware after you blink an led what is the first interface you use to talk to your hardware serial port so we run pcie over rs232 at 115 kilobaud which makes this pcie i said it was delayed tolerant uh so it makes

this pcie 0.00002x and eventually there was a gigabit ethernet port on the fpga so i upgraded to that but i only got around to oh to doing it in one direction so now it's pcie 0.00002 in one direction and point point in the other direction which has to make this one of the

most asymmetric buses in the world um but it works like believe me this is hilarious you can run pcie over serial also we were asking coding so half the bandwidth um yeah it's fine it's fine it's fine so pcie 101 it's a reliable packet switch network it uses a thing transaction layer packets which are basically just packets you send and it can be you know

memory read memory write io read io write configuration read configuration write it can be a message signaled interrupt which is a way of saying hey listen to me by writing to an address in memory because we can write the things so why not write for interrupts it has legacy interrupts which are basically emulating the old set this wire low for

interrupt and high for no interrupt thing you can tunnel that over pcie and it has completions which are basically the replies so if you read a value from memory the completion is what you get back with the value you tried to read okay so it's pcie right so we can just go wild with dma we can just read all memory dump the kernel hey it's awesome right except there's an iomemu in the apu but uh of course the iomemu will you know protect

the devices it will only let you access what memory is mapped to your device so the you know the host has to allow you to read and write the memory but just because there's an iomemu doesn't mean that sony uses it properly so here's some pseudocode and you know it has a buffer on the stack it says please read from flash to this buffer with the correct length

can anyone see the problem with this code well it maps the buffer and it reads and it unmaps the buffer but iomemus don't just map byte food to byte bar they map pages and pages are 64k on the ps4 so sony has just mapped 64k of its stack to the device so we can just

dma straight into the stack basically the whole stack and take over so now we get code execution free bsd kernel dump and webkin and ios lips dump just from mapping the flash okay that's step zero so we have the code but this is not you know that's not the ps4 that we did this on it's a you know it's a giant mess of wire someone here knows about that uh you know flying

over the face but we're gonna make a nice exploit uh we've done that because as i said kit free bsd whatever what comes after that okay we want to do something of course we're gonna run linux and uh how do you go from free bsd to linux it's it's not a trivial process uh but you use something that we call ps4k exec uh so how does this work it's you know

it's simple right you just want to run linux just jump to linux right well kind of uh you need to load linux into contiguous physical RAM setup boot parameters shut down free bsd cleanly hold secondary CPUs make new page tables a lot of random things i'm not going to bore you with all this crap because you can read the code but there's a lot of like

iteration in getting this to work let's assume that you do all this magical cleanup and you get linux into a nice state and you can you know jump linux okay now we jump linux right it's cool yeah okay you can technically jump to linux and it will technically run um for a little bit yeah yeah then it'll stop and you're not gonna get any serial

or any video or anything what's going on here okay let's talk about hardware um what is x86 x86 is a mediocre instruction set architecture by intel it's it's okay i guess you know it's not great it's okay ps4 is definitely x86 it's x86 64 um what is a pc ah pc is a horrible

horrible thing built upon piles and piles of legacy crap dating back to 1981 the ps4 is definitely not a pc then again that's practically sony level of hardware fail so it could be but it's not um okay so what's going on well a pc a legacy pc basically has an 8259 programmable

interrupt controller a 253 programmable interval timer a uart at io3 f8 which is the standard dress for a serial port it has a ps2 keyboard controller 842 it has an rtc a real-time clock with the cmos if everyone knows the cmos right mc146818 is the chip number for that an isa bus even if you think you don't have an isa bus your computer has an isa bus inside

the southridge somewhere and it has vga the ps4 doesn't have any of these things so what do we do well okay let's look a little bit of what how a pc works and how a ps4 works this is a general simple pc system there's an apu or an intel core cpu with the southbridge you know intel calls it pch and the fch there's an interface that is basically pcie though

intel calls it dmi and and calls it umi whatever ddr3 ram and a bunch of peripherals and sata whatever the ps4 kind of looks like that right so you think this can't be that yet what's so hard about this you know because all the crap i mentioned earlier is in the southbridge in a pc right so you know ps4 has a southbridge right right right uh so the southbridge the AMD standard fch

implements intel legacy from 1981 the marvel aolia marvel is the maker of the ps4 southridge implements intel legacy from 2002 what does that mean ah that's no southbridge that's a marvel armada soc so it's not actually a southbridge it was never a southbridge it's an arm system on a chip

cpu with everything it's a descendant from intel strong arm or x scale it has a bunch of peripherals and what they did is they stuck a pci bridge on the side and said hey x86 you can now use all my arm chip so it exposes all of its arm peripherals to the x86 they added some that they really needed for pcs and has its own ram why do they do this well it also runs 3bsd on

the arm in standby mode and that's how they do the whole download updates in the background get content mean update whatever all that crap is because they have a separate os on a separate chip running in standby mode okay that's great but it's also batshit insane uh yeah so quick recap this is what a pci bus number looks like it has a device number so it has a bus number

which is 8 bits a device number which is 5 bits and the function number which is 3 bits so you've probably seen this in lspci if you've ever done that this is what a regular southbridge looks like so it has a usb controller a pci you know isa bridges sat or whatever and it has a bunch of devices so one southbridge pretends to be multiple devices because you only have

three bits for a functioning number so you can only have up to eight functions in one device so the you know intel southbridge says well i'm device 14 16 1 8 1 b and just a bunch of roughly unpatched linux kernel on the ps4 you get something like this so the aeolia first of all

clones itself into every pci device because they were too lazy to do if device equals my number then reply otherwise don't reply no they just said oh just reply to every single pci device that might query so linux sees the southbridge you know like 31 different times which is kind of annoying because it gets really confused when it sees 31 clones of the same southbridge and

then it has eight functions acpi ethernet satis dmc pci express okay eight functions so all three bits turns out eight functions are not enough for everybody uh function number four pci express glue has a bridge config msi interrupt controller icc we'll talk about that later hpet timers flash controller rtc timers two serial ports i squared c all this smashed into one

single pci device so linux has a minimum system requirement to run on anything you need a timer you need interrupts and you need some kind of console the ps4 has no pit and no pick and no standard serial so none of the standard pc stuff is going to work here the board has test points for any 250 standard serial in a different place so we want d message over that okay fine linux

has early con which we can point to a serial port and say please send all your d message here very early because i really want to see what's going on doesn't need irq's you set control equals the type the address the speed and you'll see it says 3200 instead of 115 kilobaud

that's because their clock is different so you set 3200 but it really means 115k and that gets you d message that actually gets you know linux booting on compressing whatever that's that's pretty good um okay we need a timer uh because it's otherwise everything explodes uh linux supports the tsc which is a built-in cpu timer which is super nice and

timer which on the ps4 doesn't exist so that's fail so uh again the ps4 really is not a pc so what we need to do here is define a new sub-architecture because linux supports this concept says is this not a pc this is a ps4 the bootloader tells linux hey this is a ps4

and then linux says okay i'm not going to do the old timestamp calibration i'm going to do it for the ps4 which has a special code that we wrote that calibrates against the ps4 timer and it disables the legacy crap okay so now this is not a pc this is officially not a pc anymore okay now we can talk about acpi uh you might know acpi for all its horribleness and all its

evilness and all this microsoftiness but acpi most people associate with suspense and uh suspended hibernate it's not just power it has other stuff too so we need acpi for pci config for the io mmu for the cpu frequency the ps4 of course has broken akpi tables because of course it would be so we fixed them in ps4k exec okay now interrupts we have timers we have

serial we fixed some stuff uh the ps4 does message signal interrupts which is what i said the non-legacy the nice new thing where you just write a value and uh what you do is you tell the device when you want to interrupt please write this value to this address the device does that and the cpu interrupt controller sees that right and says oh this is an interrupt and then just fires off that interrupt into the cpu that's great it's super you know

super fast and very efficient and the value directly tells the cpu that's the interrupt vector you have to go to okay let's see that's the standard msi way that your computer does msi that way this is how the ps4 does msi the aolia ignores the msi config registers in the standard

location instead it has its own msi controller all stuffed into function 4 which is that glue device yeah glue uh each function gets a shared address in memory to write to and the top 27 bits of data and every sub function because you can't a lot of things into one place only gets the different five bits and all msis originate from function 4 so like

this device has to fire interrupt and it goes to here and then that device fires an interrupt uh like what like this is all what the hell is going on like seriously this is really fucked up and the eyes are missing in the front there um but yeah so yeah driver hell uh now the devices are interdependent and the irq vector location is not sequential so that's

going to work and need to modify all the drivers and like this is really painful to develop for so what we ended up doing is uh there's a core driver that implements an interrupt controller for this thing and then we have to make sure that loads first before the device driver so that linux has a mechanism for that we have to patch drivers some drivers we patch to use these interrupts and drivers we wrapped around to use these interrupts

unfortunately because of the top bit thing uh everything has to share one interrupt within a function uh thankfully we can fix that with the iomemu because it can redirect interrupts so we can say oh interrupt number zero goes to here one goes to here two goes to here so the uh that's great because it's consecutive right zero one two three four five is

obviously going to have the same top bits um but uh we have to fix the act b table for that because it's broken uh but this does work so this gets us interrupts that function and they're individual so let's look at the checklist we have interrupts timers early serial with interrupts we can get some user space we can you know stash some user space some binaries

into the kernel and it'll boot and you can get a console but you get a console and you try writing commands and it sometimes it hangs like okay what's going on there so it turns out that free bsd masks interrupts with an AMD proprietary register set we have to clean that up too and that fixes serial and all the other interrupts this took ages to

find it's like why like interrupts on CPU zero sometimes don't it right yeah um i ended up dumping register sets and i saw this fffff here and not fff what's that but uh yeah like tracking through the stack to find this was really annoying all right so we have the basics we have like a

core platform we can you know run linux on even though don't won't do anything interesting add drivers uh so we have usb xhci which has three controllers in one device again just you know let's make it insane we have sd hci that's a sdi o for the wi-fi and the bluetooth needs a non-standard config needs quirks ethernet needs more hacks it's still partially

broken it only runs at gigabit speeds if you plug in 100 megabit switch it just doesn't send any data not sure why um and then all of this works fine in the next 4.4 and then just three days ago i think i tried to rebase on 4.9 and so we have the latest and the greatest and everything failed and dma didn't work and all the drivers were just throwing their

hands up in the air and what's going on here aeolia strikes back so that's what you know the uh you know the aeolia looks like normally so you have it's again it's an arm sock it's really not a device it's like its own little system but it maps its load two gigabytes of the address space to memory on the pc and then the pc has a window into its registers

that it can use to control those devices so the pc can kind of play with the devices and the dmas to the same address and that works great because it's mapped in the same place and then it has its own ram um you know in its own address space this works fine but now we had an iomemu because we needed it for the interrupts and the iomemu inserts

its own address space in between and says okay you can map anything to anything you want it's great you know it's a page table you can say this address goes to that address linux 4.4 did this it would find some addresses at the bottom of the iomemu address space say you know page one goes to this page two goes to that page three goes to that and say device you can now write to these pages uh and they go to this space on the x86 that

works fine it turns out linux 4.9 or somewhere between 4.4 and 4.9 it started doing this it would map pages from the top of the iomemu address space and that's fine for the iomemu but it's not in the window in the aeolia so now the you know you say ethernet dma to address fe something something something and instead of dmaing to the ram on the pc it dmas to the ram on the

aeolia which is not going to work so yeah effectively the aeolia implements 31-bit dma not 32-bit dma because only the bottom half is usable it's like why did this is all really fucked up guys like seriously and this is littered all over the code in linux so this needed more

patches and it works but yeah uh painful okay devices you know devices work now for something completely different who can tell me who this character is that's uh starshot from space battleship yamato and apparently that's the code name

for the ps4 graphics chip or at least that's one of the code names because they don't seem to be able to agree on like what the code names are like it's got liverpool in some places and starshot in other places and faby jay in other places and we think sony calls it starshot and amd calls it liverpool but we're not sure uh we're calling it liverpool everywhere just to avoid confusion but yeah okay what's this gpu about well it's an amd c islands

generation gpu uh which is spelled ci instead of si because s was taken um it's similar to other chips in the generation so you know at least it's not a batshit crazy new thing but it does have quirks and customizations and oddities and things that don't work

what it is we took bonair which is another gpu that is already supported by linux in that generation and just kind of added a new chip and said okay do all the bonair stuff and then change things and hopefully uh adapt it to the ps4 so hacking on amd drivers okay well they're open source but amd does not publish register docs they publish 3d shader and command queue documentation so you get like all the user space 3d rendering commands that's documented

but they don't publish all the kernel hardware register documentation that's what you really want for hacking on drivers so that's annoying and you're thinking the code is the documentation right just read the linux drivers that's that's great well yeah but they're incomplete and they have magic numbers and it's you know you don't know if you

need to write a new register that's not there and it's really sucks to try to write a gpu driver by reading other gpu drivers with no docs so what do we do we're hackers right we google every time you need information hopefully google will find it because google knows everything and uh any tidbit you can find in any you know forum or uh code dump somewhere that's great one of the things we found is we googled this little string um r8 xx gpu in quotes and you

get nine results and the second result is this place it's a silicon kit token what that okay it's an xml file and if we look at that it looks like it's an xml file that contains a dump of the bonair gpu register documentation but it's like broken xml and it's incomplete that

stops at one point but like okay what's what's this doing here um and why like what what is this where this come from right so let's dig a little deeper okay google what do you know about this website uh well there's some random things like what the hell no dot txt and what the hell yes dot txt and uh some excel files or sorry excel like xml style sheets um

and then there's a thing at the bottom there called rai.grammar.for.txt i wonder what that is and it looks like it's a grammar you know uh bnf notation description for a syntax of some kind of registered documentation file it's it just looks like an AMD internal

format but it's on this website okay so we have these two urls slash pragmatic slash bonair xml and slash rai slash rai grammar dot for txt let's try something how about maybe pragmatic slash bonair.rai nah it's a 404 okay it's pragmatic slash rai slash bonair.rai

ah bingo so this is a full bonair or almost full bonair registered documentation with like full registered field descriptions breakdowns all the addresses it's not 100 but like the vast

majority this seems to be a and d internal stuff uh and i looked this guy up and apparently he worked at AMD at some point so but yeah this is really really helpful because now you know what everything means and debug registers and yeah so i wrote a working parser for this format

not the xml this i was apparently writing a xml parser or something like convert this thing to xml but it was all broken oh he was writing PHP by the way but there you go so i wrote a each register means and it'll tell you all the options you can take a register dump and map it to the you know basically documented you can diff dumps you can generate defines it's very useful

for AMD GPUs and this grossly speaking applies to a lot of AMD GPUs like they share a lot of registers so this is useful for anyone hacking on AMD GPU stuff over 4000 registers are documented in the just in the main GPU address space alone so that's great okay so we have some docs

how do we get to a frame buffer uh so if you uh you know the history has hdmi that's easy right the GPU has hdmi and if you query the GPU information you actually get that it has an hdmi port and a display port port okay maybe it's unconnected that's fine right uh but if you actually ask the GPU it tells you hdmi is not connected display port is connected okay

yeah they have an external hdmi encoder from display port to hdmi because just putting a wire from a to b is too difficult because this is sony so let's put a chip that converts from protocol a to protocol b yeah yeah yeah yeah yeah and okay it's a it's yeah it's a panasonic

display port to hdmi bridge not documented by the way requires config to work that's why it doesn't just work even though some bridges do and you'd think okay it's hooked up to the bus because GPUs have in the past used these bridges and not this one particularly but other

a and d cards have had various chips that they stuck in front and the code has support for talking to them through the GPU I squared C interface right right that's easy yeah you wish this is sony enter ICC so remember the ICC thing in the earlier it's an RPC protocol you

use to send commands to an MCU that is somewhere else on the motherboard it's a message box system so you write some message to a memory place and then you tell it hey read this message and it writes a message back and it tells you that's the reply you access it via aovia not via the GPU you use it for things like power button uh the LEDs turning the power on and off and also the hdmi encoder R squared C so now we have a dependency from the GPU driver to

the aovio driver and two different PCI devices and two different yeah and okay again ICC but it's I squared C you know I squared C is a simple protocol you read a register you write a register that's all you need it's super simple right right now let's make a bytecode fucking scripting engine to issue I squared C commands and delays and bit masking and everything

and why sony why like why would you do this well because ICC is so slow that if you actually try to do one read and one write at a time it takes two seconds to bring up hdmi yeah like yeah I don't even know at this point I have no idea okay and by the way

this thing has commands where you can send scripts in a script to be run when certain events happen so yo dawg I heard you like scripts I put scripts in your script so you can I squared C while you are squared C like let's just go even deeper at this point right because yeah

yeah yeah yeah okay we wrote some code for this you need more hacks it needs all display port lanes up linux tries to downscale doesn't work memory ground with calculation is broken mouse cursor sizes from the previous GPU generation for some reason I guess they forgot to update that so wait all this crap you get a frame buffer but X won't start ah well it turns out

that ps4 uses a unified memory architecture so it has a single memory pool that is shared between the x86 and the GPU and games just put a texture in memory and say hey GPU render this and that was great and this makes a lot of sense and their driver uses this to the fullest extent

so there's a vram you know the legacy uh GPUs had a separate vram and all these integrated chipsets can emulate vram using a chunk of uh system memory and you can usually configure that in the bios if you have a pc that does this and the vs4 sets it to 16 megabytes which is actually the lowest possible setting and yeah 16 megs is not enough to like have more than one

full hd frame buffer so obviously that's going to explode in the next pretty badly uh so what we do is we actually reconfigure the memory controller in the system to give one gigabyte of ram to the vram and we did that in vst kxx so it's basically doing like biose things we're reconfiguring the northbridge at this point to make this work but it works and with this we

can get x to start because it can allocate its frame buffer but okay it's 3d time right uh yeah GPU acceleration doesn't quite work yet so we got at least you know x but uh let's talk a bit about the Radeon GPU for uh for a second so when you want to draw something on a GPU you send

it a command and you do this by putting it into a ring which is really just a structure in memory that's just a list of commands and it goes it wraps around right so that way you can queue things to be done in the GPU and then it does it on its own and you can go and do other things so that's a graphics ring for drawing a compute ring for gp GPU and a DMA ring for

copying things around the commands are processed by the GPU command processor which is really a a bunch of different CPUs inside the GPU that are called f32 and they run a proprietary AMD microcode so this is a custom architecture also the rings can call out to ring to ibs which are indirect buffers so you can say basically call this piece of memory do this stuff there return

back to the ring and that's actually how the user space thing does things so you know this says draw all this stuff and it tells the kernel hey draw all this stuff and the kernel tells the GPU jump to that stuff read it come back keep doing stuff um this is basically how much GPUs work but radion specifically works like you know but this f32 stuff okay the driver

complains ring zero tests failed thankfully it tests them so at least you know it has nice diagnostic and how does the test work it's really easy it writes a register with a value and then it tells the GPU with a command please write this other value to the register runs it and then checks to see if the register was actually written with a new value

so the write doesn't happen it never it's there thankfully thanks to that rai file earlier we found some debug registers that tell you exactly what's going on inside the GPU and it shows the command processor is stuck waiting for data in the ring so it needs more data after a knob command yeah knob is hard let's go stalling so packet headers and this GPU

thing has a size that is size minus two whoever thought that was a good idea so a two-word packet has a size of zero then AMD implemented a one-word packet with a size of minus one and old firmware doesn't support that and thinks oh it's three fff so i'm just gonna wait for

a shitload of code in the buffer right it turns out that hawaii which is another GPU in the same gen has the same problem with old firmware so they use a different knob packet so there was an exception in the driver for this and we had to add ours to that but again getting to this point many many many hours of head banging um yeah okay we fixed that now it

says ring three test failed that's the sdma ring that's for copying things in memory and it works in the same way it puts a value in ram tells the sdma engine hey write a different value and checks this time we see the write happens but it writes zero instead of the dead

beef or whatever okay so i tried this i put two right commands in the ring saying right to one place what to a different place and this time if i saw what it did is it wrote one to the first destination and zero to the second destination i'm thinking okay it's supposed to write dead beef which is what you see there it says you know dead beef is that word with the

value it writes one well there's a one there i wasn't there before it was a zero because it padding right so yeah it turns out they have it off by four error in their sdma command parser and it reads from four words later than it should again this took many hours of head

banging and it was like randomly try two commands oh one one one yeah uh so it reads two words four words too late but only in ring buffers indirect buffers work fine that's good because those come from user space so we don't have to muck with those we can work around this

because it's only used in two places in the kernel by using a fill command instead of a write command that works fine again how do they even make these mistakes okay but still the gpu doesn't work the ring test pass but if you try to draw you get a bunch of page faults and it turns out what happens is that on the ps4 you can't write the page table registers from actual

commands in the gpu itself you can write to them from the cpu directly you can say just write memory you know memory register right mmi all right but you can't tell the gpu please write to the page table register this so the page tables don't work the gpu can't see any memory so everything is broken linux uses this free bsd doesn't it uses direct writes and we think

this is maybe a firewall somewhere in the liver pool some kind of security thing they added we can directly write from the cpu but it's like breaks the regular like it's not asynchronous anymore so this could break things that's a really hacky solution i would really like to fix this and i'm thinking maybe the firewall is in the firmware right but it's proprietary and undocumented

firmware so let's look at that firmware it's a thing it needs microcode right the cp thing it's undocumented but we take the blobs out of free bsd and that's great because we don't have to ship them um let's dig deeper into those blobs so how do you uh reverse engineer an unknown cp architecture that's really easy you run an instruction and see what it did

and then just keep doing that thankfully we can upload custom firmware so it's actually really easy to just have like a two instruction firmware that does something and then writes a register to a memory location and that's actually really easy to find if you just that first like write the memory instruction is really easy to find in the binary because you see like gpu register offsets that stand out a bit in the in one column so long story short we wrote f32 dis which is

a disassembler for the proprietary AMD f32 microcode i shamelessly stole the instruction syntax from arm so you may recognize that if you're used to arm assembly and this is not complete but it can disassemble every single instruction in all the firmware in liverpool for pfp mec mec and rlc which are five different blocks in the gpu as far as i know this has never been done

before uh all the firmware was like you know voodoo black magic thing that's been shipped not even the uh non-amb kernel developers know anything about this so um and you can disassemble the you know desktop gpu stuff too so this could be good for debugging strange

gpu shenanigans and uh and non ps4 stuff all right alas it's not in the firmware uh it seems to be locked in hardware uh i found a debug register that actually says there was an access violation in the bus when you tried to write this thing and i tried a bunch of workarounds and i even uh bought an AMD apu system desktop dumped all the registers diff them against the

one i had on linux and tried like setting every single value from the other gpu and hoping i find like some magic bits somewhere but no uh they probably have a setting for this somewhere but you know it's a sea of ones and zeros good luck finding it uh it does work with the cpu right workaround though so hey at least we get 3d and it's actually pretty stable

so if there's a race condition i'm not really seeing it so checklist what works what doesn't work we have interrupts and timers the core thing you need to run any os we have a serial port we can shut down the system and reboot and you think that's funny but actually it goes through icc so again it needs some interesting code there i actually just implemented that uh what

four hours ago uh because you know pulling the plug was getting old um the power button works uh usb works there's a funny story with usb is it used not to work and we you know said uh fix it later there seemed to be a special code missing and then someone pulled a repo from the usb not working branch and tested it and said oh it's working it seems we fixed it by

changing something else uh the hard disk works which is via usb blu-ray works i wrote the driver for that also four hours ago three hours ago now yeah something like that and it spent 20 minutes looking for someone in the hack center that had a dvd i could stick in to try it apparently i'm from the past if i ask for dvds so yeah but it does work so

that's good wi-fi and bluetooth works bluetooth uh ethernet works except only a gigabit speeds frame buffer works hdmi works it's currently hard coded to 1080p so uh yeah it does work uh we can fix that by you know improving the encoder uh implementation 3d works with the ugly uh you know register right hack and speed if audio works um so that's good hdmi

audio doesn't work uh mostly because i only got audio grossly working in general recently and i haven't had a chance to program the encoder uh to support the audio stuff yet because again more uh more annoying hacks there and the real-time clock doesn't work and if you think simple well the clock like device is simple but ever since the playstation 2 the way sony has

implemented real-time clocks is that instead of reading and writing you know the time on the clock which is what you would think is the normal thing to do they never write the time on the clock instead they store an offset from the clock to the real time in like some kind of storage location and there's a giant mess of you know registry it's called in the ps4 and i

don't even know where it's stored it might be on the hard drive it might be encrypted so basically getting the real-time clock to actually show the right time involves a pile of nonsense that i haven't had a chance to look at yet but we have ntp right so it's good enough all right oh and we have blinking lights important uh you know the power led does some interesting things if you're on linux so that's good um so the code um we can get the

ps4 kx code on our github page that has the kx sec and the hardware configuration and the ps4 linux branch which is the uh our fork of the kernel uh based on 4.9 which is the latest

public version i think uh you can get our radion patches which are three i think really tiny patches for user space libraries just to support this new chip uh really simple stuff the knob thing and a couple commands uh and the rai and f32 this thing i mentioned you can get radion tools at uh at that github repo i just pushed that right before this talk uh so if you're

probably want to run before the guys at that website realize it really should take that down but i'm sure the internet wayback machine has it somewhere so uh yeah okay well uh that's everything for the uh the story of how we got linux running on the ps4 um and uh you know

you can reach us at that website or failoverful on twitter uh thank you so i hope that wasn't too fast sorry i had to rush through my like 89 slides a little bit

because i really wanted to do a demo and then again this kind of is the demo right but we can try something else so maybe i can shut this uh if i can if i can aim with my controller um yeah this is really not meant as a mouse that's not right button

come on uh yeah i think it is close close maybe yes so we have this little icon here i wonder what happens if it works do we have internet access hopefully wi-fi works actually just check real quick because this could work really badly if we don't uh ping eight update

eight right yeah we have internet access okay wi-fi works okay i wonder what happens if we click that it takes a while to load this this is not optimized for uh

so the uh the cpus on this thing are a little bit slow but hey you know i mean it works and now it's a real game console and this is uh there we go okay

so uh yeah i think we can probably take some q a because this is a little bit slow um to load but we can try a game maybe um well if you ask for q a i think there

will be some questions so shall we start with one from the internet testing testing okay hey um the internet wants to know if um well most of your research will be published or if stuff's going to stay private uh well all of this i mean like the publishing is basically the code which and you know the explanation i just gave uh as i said

everything's on github um so all the drivers we wrote all the you know i mean in that case i guess also the spec is the code if you really want to i could write some wiki pages on this but roughly speaking you know what's in the drivers is what we found out the really interesting bit i think is that f32 stuff from the uh from the amd gpu stuff and

that we have a repo for but absolutely if you have any you know general questions on in particular device or any details uh for you to ask i don't know uh you know again it would be nice if we wrote a bunch of docs and everything but it's not really a matter of not wanting to write them it's you know lazy engineers not running to write documentation but the code is at least you know the things we have on github is fairly clean so

okay so someone is piling up on four guys if you have questions you see the uh the microphones over here just pile up over there and i'm gonna come on four please just a small question how likely is it that you upstream some of that stuff because i mean so um there's two sides to that one side is that we need to actually

get together and upstream it the code some of it has horrible hacks some of it isn't too bad um so yeah we we want to upstream it uh we have to sit down and actually do it i think most of the like custom x86 space machine stuff in the kernel is doable the drivers are probably doable

some people might scream at the interrupt hacks but it's probably not not terrible and if they have a better way of doing it all years the other kernel devs uh the radium stuff is white fishy because of the encoder thing that is like really non-standard and also

understandably AMD GPU driver developers that work for AMD may want to have nothing to do with this and in fact they know for a fact that at least one of them doesn't but we can i mean they they can't really stop us from upstream thing things into the next kernel right so i think as long as you know we get the code into a state where it's doable it's fine but most likely i think um i think most likely the non-gpu stuff will go in first if

we have a chance to do that and of course if you want to try upstreaming it you know go ahead it's open source right so over to microphone one please hi um first i i think i i should implore you to try and find travel hudson and uh control him into uh using your your

bsd k exec implementation in heads uh instead of having to run all of linux in it um as a joke but my real question is um if the reason you used uh gen 2 was because system d was yet another hurdle in getting this to run well um i run gentoo on my main machine i run gentoo on

most of the machines i care about uh i do run arch and a few of the others and then i live system d um but uh the reason why i run gentoo is first it's what i like and use and second it's super easy to uh use patches on gentoo you get those things we put on the github which are just patch files it's not really a repo because they're so easy it's not worth cloning everything

just get those patch files stick them on etsy port touch patches you have a little hook to uh patch it and that's all you need so it's really easy to patch uh packages in gentoo that's one of the main reasons yes number three please um will there be new exploits new way to boot linux on ps3 with

modern filmwares uh because uh finding one with humor 176 is really rare that was 405 ah but um again our goal is to focus on the i just told you the story of the pre-exploit

thing because i think that's good uh like hacker story a good knowledge to try new platforms and the linux thing we're working on the reason why we don't want to publish the exploit or really get involved in the whole exploit scene is that there's a lot of drama uh it's not like it's not rocket science in that it's like super custom code this is what can free bsd it's actually not that hard and we know for a fact that several people have reproduced this on

various firmwares so like there's no need for us to be the exploit provider and we don't want to get into that because it's a giant drama fest as we all know anyway so please like you know diy at this time okay thanks and what is the internet saying testing okay

if the internet wants to know if you ever had fun with the bsd on the second processor oh uh that's a very good question and i myself haven't i don't know if anyone else has looked at it briefly uh one of the commands for rebooting will boot that cpu into free bsd and there's probably fun to be had there um but we haven't really looked into it

and over to five please um i was wondering if any of that stuff was applicable to the ps4 vr additional however it's called the new one uh sorry did you ever test it sorry say that again um the sony brought up a new piece for oh the pro you mean the ps4 pro yes yeah so

boots on the pro we got that far gpu is broken so we would like to get this ported to the pro and also working it's basically an incremental update so it's not that hard but the gpu needs a new definition new chip all that stuff yeah i get a lot of fancy frames yeah but yeah

as you can see you know 3d works and uh there you go good you will hear a bus when you hear the bus look down at the floor look up and down in this game yeah well then number three please i want to ask you if you want to port

this radium patches to the new amd gpu uh cpu a gpu driver because amd now supports the southern islands a gpu yes that's a very good question uh actually the first attempt we made at writing this driver was with amd gpu and at the time it wasn't working

at all and i was a bit concerned about uh its freshness at the time and it was experimentally supporting this gpu generation i'm told it's uh it should work so i would like to port this you know move to amd gpu not that we have a you know working implementation and we got the cleanup code much better we know where all the nits are i want to try again with amd gpu

and see if that works that's a very good question because the uh you know newer gen might to require the driver maybe so yeah thank you well then i'm going to guess we asked the again okay um the internet asks or states that a year about a year ago you argued with someone on twitter that the ps4 wasn't a pc and now you're saying it kind of is us something and

what's about that so again it's my reason for saying it's not a pc is that it's not a it's not a piece it's not an ibm personal computer compatible device it's an x86 device that happens to you know be structured roughly like a current pc but if you look at the details so many things

are completely different it really isn't a pc like on linux i had to define you know subarch ps4 it's an x86 but it's not a pc and that's actually a very important distinction because there's a lot of you know things you've never heard of that are x86 with not pcs like for example there's a high chance your monitor at home has an 8186 cpu in it so yeah so nobody's

piling at the microphones anymore is there one last question from the internet uh yes there is um and um the question is um if if there was any decryption needed uh no so this is purely

uh you know you exploit webkit you get user mode you exploit the kernel you get kernel mode jump linux there's no security like there's nothing like stopping you from doing all this stuff there's a sandbox in free bsd but obviously you exploit around the sandbox um like there's nothing you know there's no hypervisor there's no monitoring there's nothing like

saying oh this code should not be running there's no like uh integrity checking you know they have a security architecture but as this tradition for sony you can just walk around it so yeah the on the uh the ps3 was notable for uh the fact that the ps jailbreak which is a usb

effectively piracy device that was released by uh someone um that uh basically used to use the usb exploit in the kernel and only a usb exploit in the kernel to effectively enable piracy so when you have like a stack of security and you break one thing and you get piracy that's a fail this is basically the same idea except i have no idea what you need to do

to do piracy and i don't care but yeah and sony doesn't really know how to architect security systems that's it all right thank you very much here we go that's your applause thank you