All right. We're going to get started after lunch. Rekhaan would like to welcome Jesse Exum, who's going to talk about reverse engineering ISC controllers. Thank you. So hello, everybody. Hope you all had a good lunch. My name is Jesse Exum. And a couple of years ago, I went out

to try to build a processor out of entirely 7,400 logic chips, thinking it was a good idea. Three years later, I don't have a processor. And I'm talking to you about reverse engineering in-system controller, in-system configuration controllers. So seriously, something went wrong in the middle.

So today, I'll be talking about a couple of things. First, the intro to why I got started in this and my attempt to build a processor that quite literally ended in flames. And then in my work in reverse engineering, the Digilent and Xilinx JTAG controller protocols

and firmware, as well as some re-implementation of firmware and other things along that way. And then based on the learnings from step two, I'll talk about some of my proposition for a better way to interface with these types of devices. And then hopefully, we'll have some time for questions.

And I was told by the recon team to try to make this accessible since that was my first hardware reverse engineering project. So some people may find some of the pieces completely basic. But I hope that it's very useful for the people who need it. And the end goal, I believe, will help most people in the room.

So I still have that up. So I took a college class a couple of years ago. And I learned about logic gates and decided that I wanted to build a processor with that. Took a Coursera class on it, computer architecture

by David Winsloff, very good class for anyone interested in how that stuff works. And I found that other people had built such devices, computers entirely built out of these discrete chips. So I looked up what they did. And everyone pretty much considered it novel and almost ridiculous that anyone had done it.

And even the people who designed it basically said, yeah, that was a waste of time. Boards were huge, power hungry, hard to modify. And everyone suggested FPGAs. So immediately on the path to an FPGA. But anyone who's worked with FPGAs enough knows that they are their own problem. So to get started, I purchased a Xilinx Spartan 3e development

board from Digilent. It had Ethernet, VGA, and PS2, which is enough to build pretty much a full computer that can go on the internet and do normal stuff we expect them to. It also had a built-in JTAG controller. So you could just plug USB into it and program the device.

I didn't know what JTAG was at the time, but this was quite nice. As everyone who first gets these boards, I started with blinky light examples with schematic capture. That ended up being a disaster because the tools are not made for that. And bumped into an HDL.

From there, started building little projects, as most people who try FPGAs do. Learned about, pretty quickly, the big differences between CPUs and FPGAs. And then immediately ran into a problem. I was only really able to do any of my programming work

on Windows because the Xilinx's tools at the time, ISE, didn't really behave well with Linux. Now technically it does, but I was only able to see it actually working on CentOS, running kernels 2.5 or older. After you load a proprietary binary blob into the kernel and taint the whole thing. So I wasn't particularly interested in that.

So instead, I found that they had LibUSB drivers and was excited, but they weren't loading. So I S traced the impact process and found that it was trying to load LibUSB directly by itself without letting LD do it. And it was loading it from an old CentOS location of LibUSB,

which had since been replaced by an LD shortcut file, which is just basically text. It was manually grabbing that file, importing it, and failing, of course. So I thought if I LD preloaded it, it would work. And it did, one out of five times, because of race conditions that were in the software.

So I was thinking maybe I could look into this and try to fix it, but there's obviously no source available for it. And if I tried to reverse engineer it to fix what was happening, it's part of a 15-gigabyte software suite that's written in a combination of C++, .NET, and Java. So I did not want to touch this.

And even if I did want to touch it, it would be against the EULA. So I considered switching to other vendors. And my friends on IRC explained that even though the exact problems I'm facing aren't common across all vendors, they all basically equally suck in their tooling. So I was a little let down for the moment, but it didn't matter.

Because shortly after that, I went to work on my board and plugged in my 12-volt power supply into my full 5-volt board and blew the whole thing. So if anyone's interested in how to waste $300 very quickly, I have some experience on that.

So I was out of board, had no plan at that point. I was just really frustrated. So I pieced together some money and a bit of a plan and got several different digital boards, since that's what I was used to using, each of different levels in the hopes that I could make a better tool that worked for all these boards.

And maybe I could start with the lowest level one and work my way up in complexity, assuming that the simplest and the oldest one was the easiest to work on. But sometimes I make dumb assumptions. So I had a whole range of rational goals at this point from completely rebuilding the FPGA synthesis

and place and route and thought of building my own FPGA, but I was just pissed. So I found that the flashing programs are just as what I was having those problems with, the actual loading of the program onto the chip. And I wanted it to work with Linux. And since I only had messed with digital boards,

I wanted that to work. So I set out to build my own tool. And I needed to know a couple of things to do that. First off, how do you talk to these chips directly? And since I had this board with this little magic chip on it, how do I get that board to talk to the chip and do the things that actually

are needed to program it? So for those who are not familiar, this will be a very quick overview of JTAG. I thought when I heard about it that it was only a system for programming chips, but I found out very quickly that I was wrong. And the great EEV blog had an amazing video on it that helped me out a lot.

Pretty much when you're testing old circuit boards, you had to put a pin to everything you wanted to check to make sure there was not shorts between the board. And that was getting impossible for packages that held the pins on the bottom. So instead, it was proposed to pull all that circuitry inside the chip and have it be able

to report its state via serial. But this turned out to be very extensible. And vendors jumped on this very quickly as a bus that they could put anything in. The benefits is tons of important things were able to be put in there that you only needed one plug to debug your board, flash your board,

check for manufacturing defects all with one plug. But the issue is is that all these new features came around after the JTAG proposal was made, like mostly flash and systems on a chip. So there was no real support for it, and it was a wild west of everybody doing whatever they wanted.

On the plus side, it does have auto-detective chips, and it's a bit of a mess for people who know it, but it has a good core. And finally, while I was researching this, I learned about the JTAG initialization process, which was necessary for the next step. I wanted to see what this board does when I talk to it.

I knew what it should do from reading the JTAG docs, so I got my oscilloscope, which is possibly my prized possession, and probed the state machine control line of JTAG and the clock, and used Digilent's custom GUI program, because I was sick of looking at Xianlix's tools,

to just initialize the board and check what's there. Captured the layout, sorry, captured the waveform, compared it to the documentation I expected, reset, transitioned to state, read out 32 bits of data, perfect. I now have that working. Well, good view into it working.

Now, this little thing on here, I needed to figure out how to control it. It was all done by USB, so I had a couple options, either trying to break open its firmware, but I don't like cutting open chips, because I don't want to burn off my hands. And I didn't really want to open up all the software

that was running this, because it was very big as well. So instead, I found that I could just use Wireshark to sit there and capture all the USB packets, and it worked quite well. So now I had this little test bench set up, that I could use Digilent's tools or Xianlix's tools,

to send commands to my board, to send commands to the chip, capture the commands to the board, and capture the resulting waveforms on the chip. And immediately, I felt the need to do a replay attack, to make sure that this worked again, and that they didn't have some sort of crazy like time signature on the packets. But when you're dealing with 8-bit microcontrollers,

it turns out no one really has room for that. So that was lucky. Replaying the packets worked fine. So now I should be able to make a program that does it myself. But first, of course, I need to know what they did. It took a lot of reading of the LibUSB specification

to understand the packets. So if anyone's interested in learning some more about USB, these are the resources that were most helpful for me. Once I had all of the packets captured in Wireshark, I took them out by their command IDs, and made the big table of them, said how many times they appeared, listed them off, and wrote descriptions of how I saw them used,

and theories about it. But my first mistake was I started writing down what I think they did explicitly, and not pointing out where I didn't know, like where my theory wasn't supported. So I erase observations in lieu of facts

that I thought I came up with. So for beginners doing this, don't deal with facts. They're misleading, deal with observations. And remember that what you don't know is just as important as what you do know. After that, editing the replays, reordering things, removing stuff, changing bits, just seeing what happened worked really well for figuring out what these commands do,

and filling out my table. It's pretty much impossible to get 100% accuracy, because even if you had the source of the system, the company could change it one day or extend it. But I got pretty solid on the commands I had. But pace started slowing down, and I still was missing a couple bits, and I didn't know all the commands

because I hadn't seen all of them. And then I found out, by chance, that Digilent actually not only provides the GUI, but an SDK that lets me write my own C programs that explicitly call all the commands of the board. So I could have skipped most of that stuff, but it was useful if this wasn't here. Now with this, I was able to use my same test bench

and just run each individual command, see what it sent, and better fill out my documentation. Also gave me a lot of variable names that I had just been guessing on, or didn't have a consistent naming convention for, and that helped out a lot. In the end, from this, I learned about

a ton of new commands that I didn't know, and gotta check where my theories were correct and wrong, and that helped a lot. So there were several more messages that were in this Digilent protocol that I had no idea what they were doing. They were all in the initialization sequence

when the board first started up, so I couldn't really get them to appear in any different ways. Most of them were, in fact, just reads from the board where it sent back empty strings of a fixed width, and I couldn't tell what they were. But the most annoying ones were these E8 and EC messages. I thought the sequence would go,

the computer would send the board an E8 message with a two-byte random number that I didn't know where it came from. Then the computer would read a four-byte number off of the device with the EC command, and then write zero back as another two bytes, just zero, zero, to the machine, and clear it out.

And every time the data that was written and the data that was read was completely random, I had no idea what it was. So I had a pro to the little program I wrote that used the Digilent library, and I think I also used frickin' Visual Studio at some point for some reason. But it had a decent debugger sometimes.

So I stepped through that, and the first thing I ran into was address space layout randomization. All of my breakpoints kept being deleted every time I reran the program. And being a noob that I was at the time, I just kept remaking my breakpoints every time. But anyone who runs into this, you can disable it.

Just don't do that for production. So inside the library, I stepped into the initialization functions for the board and immediately I was faced with just hundreds and hundreds of functions where everything was being jumped and executed based on calculated addresses

that you couldn't tell unless you ran the program. So I had to actually debug it, and static analysis was basically out for me at my skill level at that point. But as I walked through it, I finally found the Windows code for messing with USB, and it called E8 and provided the number,

the lower 16 bits of the uptime of the core of the computer confused me, kept going, saw that the EC message read back the four bytes, XOR the two bytes that were sent to the chip the first time, and XOR that result with every single thing that came back, and then checked if that was equal to the string digi.

I had spent so long on this and it was so confusing, and then I realized that my friend on IRC was right. If you see a bunch of XORs, it's usually just trying to confuse you, and it was just OEM verification, completely unnecessary for running the board normally. But now it's documented, so.

And then there was a couple of the commands, but I'll get to those in a little bit later, but those are the ones that were particularly interesting to me. So I started a new project, and slightly in spite of Digilent Adepts programming, I just called it Adapt, and that code's actually available on GitHub.

I'll provide the link in a minute. But I converted each of the packets I had in the list to Python functions, and then wrote a little main function that called them all in sequence. So instead of replaying, I was just producing the packets with the same code.

So with this, I was able to talk to the chip and send arbitrary JTAG commands to it, but this was far from programming it. Looking into how to program it, I ran into IEEE 1532, also known as the boundary scan definition language specification, I guess. The motivation is because all those JTAG control,

all those JTAG devices have such different control protocols for just common operations, since they were all extended in different ways by vendors a long time ago. Let each of the vendors produce a file for each of their chips that specifies the order of commands that you have to run to use the device.

Seems pretty straightforward. You basically get a script of like, I want to flash it. Here's a big sequence of commands. And that worked out pretty well at first, but immediately there was issues, because they actually weren't strict enough in their implementation of the standard.

So I made JED parsers and BSDL parsers so I could load the data I needed to produce the file. And I started running into some confusion about how to actually flash the file, so I turned to IRC again. IRC again. And SiliconPron's Dr. Andrew Zonenberg, who spoke here a while ago about actually working with the same chip that I had on my default board

I was working with, the Xilinx CoolRunner 2. I presented to him my idea that I could just load these BSDL files and everything would work great. And he shut down my naive vision pretty much immediately, and educated me about how it actually worked. In the case of this chip, yes, it's all specified in the BSDL files,

but the JED file that you get is actually in a virtual memory space, and every single bit has to be translated to a new location and then loaded in line at a time. And those lines also had to be loaded in gray code order, which both of those things cannot be specified in the BSDL specification. So it's almost useless, which is a bit of a shame

because it really tried. So it seems that a lot of more recent projects are just going class per chip, and just ignoring the BSDL files outright. But whatever. So started writing the code, it flashed it, everything worked except sometimes it would be totally wrong

and I found out that bits that were supposed to be null and did nothing, had to set them to one, because if I set them to zero, things randomly failed. It's great when you have hardware that you can't see how it works inside. So I got this one digital board down, and I wanted to see if the other boards were similar.

Mostly my plan was that for each of the boards I had, I wanted to talk to the board and then be able to program the chips. So Spartan three, E, Spartan six, et cetera, as I went up, assuming that that'd be pretty easy.

So for all the other boards I had, I hooked them up and ran the same observations on them and found that their protocols were basically identical. But there was one little thing that was weird. Every time these boards ran, these newer boards ran, they always seem to have this A zero command being run. And there was like kilobytes of it, just tons and tons of these commands.

So I did the thing anyone would do when faced with something kind of scary, I skipped it. And the board didn't work. Instead, it would reply to its name when you asked it and another identifier like serial number, but it actually just wouldn't do anything. Then I tried rerunning the big capture of A zero messages

and it worked fine. At that point, I was a little too dense to realize it was firmware, but I picked that up pretty quickly after that. So I found that all the boards that were getting this A zero message had a different USB controller chip acting as the JTAG controller.

And for the simple boards I had, it was AVR chips, but for the boards that, I don't know why that happened, but for the boards that were requiring A zero messages, it was always the Cypress Easy USB FX2 chips, which a couple of people I've talked to here

have all just gone, oh, yeah, that. So I didn't really wanna deal with firmware at that time and like, well, I guess I'll go look at something else because I don't wanna see this. So I turned to look at some of the other controllers that are available. Originally, I only went to target boards that had built-in controllers,

but after talking to people, I realized that no one really uses them unless they're just testing something. And everyone who's really doing work has these controllers and plugs them in and does whatever they want or can reverse an existing machine with their controller. So I thought, okay, I'll add this to it.

So I also found that there was often a single controller or more per vendor. And I purchased a couple of controllers and went to use them with my chips and found that if it wasn't Xilinx, it wasn't working with the Xilinx chips. Started looking in why there was no descriptions on how to do that. Found out it's entirely because the same lack

of documentation on these digital end boards affects these controllers. And the only people who have that documentation are the companies that make it. And the companies that make it only want to support their chips. So the end result is you can only use a controller and a chip and a piece of software together if it all has the same brand name.

If it all has the same brand name. Which felt ridiculously silly to me since it's all based on standards just with extra little layers of crap wrapped around it. So I don't know, I just don't think that, I don't think of fashion statements, what we need in engineering. So, this actually took me like an hour to make, so.

And when I considered cutting the slide, I'm like, there's no way. So this type of lock-in only benefits vendors because if Xilinx is selling their controller and it's $200, it's the only one that'll work, well, you have to pay $200.

If you're a company, you probably don't mind it, but if you are a hobbyist or a reverse engineer trying to break open an Xbox or something, then it can get quite expensive quite fast. So, because I had Xilinx chips and I was able to watch the Xilinx software talk to the Xilinx chip over the Xilinx box,

all parts of the outfit matching, I decided to buy a Xilinx controller, one from China, way cheaper, and start with that. So this is the Xilinx platform cable USB, which for some reason has USB on the end, but I never was able to figure that out. So I monitored the programming on Windows because that's where it worked,

and it had, I had some problems replaying the packets. So I plugged it in, hooked up the monitoring stuff, captured the packets, cool. Ran it in Linux, didn't work, wasn't responding. So I went back to Windows and I monitored the packets that went to it since it was plugged in, and I found more A0 messages.

I opened up the box and looked at it, a freaking Cypress Easy USB FX2. They are literally everywhere. So at this point it was clear I needed to start messing with firmware. Just a brief introduction so people know how this guy works. These chips usually are only programmed

with their USB device ID and vendor ID. They appear to the computer and say, hi, I'm this, and the driver will take it and say, ah, an uninitialized device. Send it A0 messages, tell it to turn on, that'll disconnect itself from USB, turn itself back on, and appear as a different device. And the purpose of this is that you always have the firmware that works with your driver.

You never have to deal with legacy or incompatible versions. So every time this thing plugged in, you had to flash it with the firmware. And that's the role of the Xilinx kernel driver that only worked with Linux 2.5. So I wasn't interested in that, and I looked for a more open solution.

And I found FXLoad, which is an open tool that is able to talk to these FX chips and send them information over USB to flash them. And I hooked that up with UDEV with some rules that I have documented on my GitHub so that when you plug it in and you take the firmware file out of the Xilinx repository tree thing and put it in a location, you plug in the device,

light turns on, it's flashed with its firmware. So now I could work with it in Linux. Now I could start messing with the protocol and seeing what I could do. First thing to note is that it's very different than the Digilent protocol. The Digilent protocol had, well JTAG has two wires

that you mostly want to write to and one you want to read to on a clock. And the way the Digilent protocol would have it is functions like write TMS, one of the pins. But that would hold TDI at one location, or write TDI, hold the other at one location, or write TDI and TMS, or write TDI, TMS and read TDO. So just pretty much every combination

you could think of of all those things. The Xilinx protocol had just one JTAG related message and it was always really big. It did have a bunch of other little settings and commands that did things like, I found one set the speed and a couple other things like that. But this one big command actually specified

all of those states of when you should clock, when you should write, when you should write, when you should read for every single state transition all the time in chunks of 16 bits. So this was just very different than what I expected. And I found a bit of help from an old document.

Unfortunately, I couldn't find the document for this presentation, but they had some corrected descriptions of the format of the data, which saved me a lot of time. But they had this amusing thing of thinking that when you set the count of that bits you're sending to divisible by four, that the device would freeze, when really it was just zero indexed.

So when they sent four, they were telling it five and it was expecting another chunk of 16 bits that they never sent it. So added that to the documentation. At least that'll help someone else. So yeah, extended the documentation with speed settings and the laziest OEM verification I've ever seen.

Literally just takes the byte and flips it backwards. And the way they do it to check that it's valid is they do every single byte that could be sent. They send zero, they send one, they send two. I guess that's secure, but I don't know. I don't work at Xilinx, I'm not a professional.

So I then went to my software that I'd written for the Digilent stuff and I thought, okay, I'll add this controller, so I should be able to just switch it out. Proved my API, added a state machine so I couldn't easily say things like JTAG, transition to state instead of saying 11010111, whatever. And that made all my code look a lot cleaner

and a lot easier to work with. But I also found that the abstractions that I built were all based on the Digilent function. So I had the read TDO, write TDI, write TDI TMS, all those things that I implemented. And then when I tried to build the driver for the Xilinx controller, I was basically taking this one message it handled,

which was universal, and just putting in to each of these weird restrictions, okay, well, put zeros for all this, I guess, because it's write TDI. And I didn't really think much about this at the time, but this ended up being a bit of a pain.

Because when I started, I was able to flash finally the chip using both controllers. And I found that they both took about the same amount of time, which was concerning because the Digilent controller on Windows took something, I don't have the numbers exactly, but like 15 times longer than the Xilinx one. And if they're both going the same speed, then I've clearly done something wrong.

So I took another look at the Xilinx cable's messages that were being spit out of impact of Xilinx's tool, and found that the way I was doing it is I was sending little tiny messages one at a time that were doing tiny bits of JTAG operations, where Xilinx was taking all the data and turning it into as big of messages as they could

and flushing it at once. Then I realized that my model of this just wasn't supporting that. So I needed to look in better ways to do it. But before I started designing anything, I thought I should look at other controllers and make sure I don't make the same mistake. So I got two other controllers, one from Altera,

which ended up being kind of not that special, it works, it's fine, it's just wasn't interesting, wasn't what I was looking for. And then I found OpenJTAG, which broke all my expectations. I really respect what these people are doing. They're making open hardware JTAG controllers mostly targeted at open source developers

so that you can easily work with the system without having to know all the low-level details that I'm explicitly having to deal with here. And the way they do that is all the state machine transition stuff that I was keeping track of, the board itself keeps track of that. And instead of telling it send these TMS bits to transition the state machine and clocks and all that,

you kindly ask it through an FTDI message, I think, to just set the state and then it will do that for you. So what this meant is I had several different controllers over here that all gave you very fine-grained control over bits

at different levels of granularity of the actual packet size. And then this other controller, which gave you no granularity, but this high-level operation. So I then see that there's three types, but I can really group those down into two types.

The Xilinx digital and Altera controller are the individual bits that you specify, and this OpenJTAG controller, and maybe some other ones, are all these higher-level commands. And currently, the way my system was built, I physically could not handle the OpenJTAG controller because everything I was doing was in terms of bit transitions,

and it just would have been like trying to convert it to the bits and then back, and it just wasn't gonna work right. So I started to see a pattern come. There's these four types of stuff, like higher-level operations, like flashing the chip, individual JTAG operations, like writing individual commands,

which are made up of state machine control changes, which are made up of bit manipulations, and so on. Chip control easily resolved to read-writes, read-writes easily resolved to state machine changes, state machines, JTAG pins, et cetera, but it doesn't go backwards. So it started to feel like it was a compiler,

and that maybe I could write some sort of grouper-compiler-optimizer thing for this. So I did. This actually goes in order 0321, but I liked that better when I was building it, and purple refers to read barriers where it actually needs to get data back to do something to determine what to send next.

So the packet actually has to be sent there and flushed. So at the top, if people, I think actually you can't see that, cool. Don't really pay too much attention to all the bits at the top, because that's not what this talk is for, but we can resolve it to the layer 03 right below it, which has the purpose saying,

reset and read the first ID of a device, check the second device ID, et cetera, then enable the in-system configuration register, and then select the flash line and read the line. Now those are the operations that those bits above it describe, and we can resolve this down to a lower level.

This can be send a reset command, then shift to the data register state, read a register, et cetera, et cetera. And we can lower that even lower to level one of write TMS data, write TMS data, write TDO. But those write TMSs, despite being two different logical operations, are really doing the same thing.

So we can just group them, as you can see below it where the write TMSs coalesce. So by doing this, we're actually able to take several, yeah, you can see my mouse. We're able to take what would normally be like one, two, three, four, like 20 messages or something and convert it into here is one, two, three, four messages,

maybe five. And for the Xilinx controller, since it had such a general purpose protocol, it was actually able to just be three messages. I tried this, and after building this system, and it ran as fast as the Xilinx programmers did, sorry, the Xilinx software did

when using the Xilinx programmer. So very briefly on this, the point of it is to take all the commands that you're writing and using, and when you say, I need to transition the state, don't run it, add it to a queue, and flush it later and the flushing happens, you have the option of resolving down to whatever messages the controller you have selected

actually understands, and then an optimizer that does an optimization pass over it and runs it out. I didn't think at first that this would be worth it, but it turns out that the speed of messing with like a kilobyte of data and moving it around memory, even in Python, is significantly faster than the time it takes

to send bulk packets to USB. So whatever optimizations we can do here will speed us up a lot. So I had that, and the system was working fast, worked well for multiple different controllers, and I was pretty satisfied with my results at this point. But then I went to show someone and remembered,

oh, well, okay, let's see, you have to get this firmware, and then you have to get these JDAC files, and then you have to get this file, oh, dammit. You know what, you need to download the Xilinx tools so we can pull these three files out of them. And yeah, like literally the way you are supposed

to get that firmware image is download 16 gig, 15 gigs of files to copy out a 21.8 gig kilobyte file, because that's, you have to agree to their EULAs. So I was sick of this and wanted people to be able to use my work without having to bend over backwards. So I thought maybe we should just open source

the firmware of these controllers. So first thing I did is, I don't know if this is actually right or whatever, I think I might have dodged a bullet, but the EFF has talked to me and said they'd help me out if I have any problems.

Basically, I agreed to the EULA when I downloaded the tool, but I didn't wanna open up their firmware files, so I did a Google search for their firmware file name and found all sorts of FTP repositories just lined with the old versions. So don't know enough about the law to know what that matters,

but at least I feel like I kinda did, oh, whatever. I'll put my email later and don't just, don't email me Xilinx. So now that I had a firmware file and I knew roughly what the FX2 was, which I'll get in a bit, but anyone who doesn't know, it's an 8051 microcontroller from like 85 or something.

I found a schematic of the actual controller box I had on some German site. It was kind of awesome. I don't know where they got it, but I'm really happy they have it. So I had a schematic. I knew what the chips were roughly. And I already knew this,

but the second, just the serendipity of this all is so weird. I was working with the site, I was trying to program the Xilinx CoolRunner 2 chip, and it turns out the box I was using to program it has a hardware accelerator that is the same chip. So it's just FX2s and CPLDs of the same type

all over the place. So I guess I should be thankful. And before we get into actually how the software works, I wanna point out that the way this works with hardware and why it's able to be so fast is that all the data that comes down through USB gets piped straight into the chip and moved over to a general purpose IO

like scripted interface on this chip. And it is able to run waveforms against the, sorry, in the FX2 chip, the USB one. And it's able to run waveforms with a state machine against whatever other chip it's attached to, in this case, the CPLD.

So it literally takes the 16 bits of data we send over USB in groups and takes one 16-bit group at a time, puts it on the 16 data pins, sets a mode, enables it, and clocks it until it says it's done. So the CPLD is actually doing the individual JTAG work, but there's still interfaces between the FX2

and the CPLD that's doing the hardware acceleration that we have to understand. Okay, I guess that's mostly that. So a couple brief points on the FX2 architecture, mostly for people who haven't had to deal with any FX2 chips themselves, particularly people who haven't had to deal with any very old microcontrollers.

This was a very big learning experience for me. It's Intel 8051-based, which is Harvard architecture. There's been a couple talks on that, but no one really mentioned what it is, so I'll give a couple points. Separate memory space is for code and data. Benefits is you can't overwrite code because you physically can't write to the code.

It's separate memory, separate memory space, and usually read-only. And also, there has to be different instructions for each memory set that you read, which adds complexity, but it's not too bad, I guess. This particular chip has a whopping 256 bytes for its stack

and that includes the memory mapped registers, R0 through R8. So it's really not holding much. And because of this, there's almost no local variables in this entire program. Everything's global because they don't have any place to put it. This chip is the old 8051 core attached to a USB core

that takes up about twice as much space in silicon than it, and it is completely aware of how to talk USB itself and optionally can pass data to the microcontroller through shared memory. And with interrupts, which was very fascinating. I'll call it fascinating,

how they got the interrupts working. And then there's this GPIF waveform thing, which is that scripted state machine that you're able to pass it data. And you can literally take 32 groups of 16 bits or more and just say, okay, I'm gonna put this in this buffer,

get to that, go, and the processor can just do something else while the hardware runs through the state and does all these complicated requests. So I see why they used it. Oh, and because there's so much hardware, like acceleration stuff, writing to registers, triggering events all over the processor,

reading the manual is absolutely necessary for this chip. So a load-in is IDA Pro, and it's supported, the 8051 supported by IDA, but it doesn't have decompiler support, which would have been nice, but it just meant I had to do a bit more digging

to get stuff working. At the time I started this, IDA 6.5 did not have easy USB FX2 options when you're loading the ROM. You just had to manually map all the memory and specify everything, which took a couple of hours at least. Luckily in 6.6 they fixed that and added a bunch more data,

but they don't have all the support. There's the weird stuff they did with interrupts, but that just takes an hour or so, so it's not too bad. Oh, right, those lack of interrupts meant that there was tons of code all over the system

that was just, or all of the file that were just unknown blobs, because they were jumped to from only one place, which was an interrupt that was not detected. So it just saw it as no entry point. So inside the firmware, I already point that, whatever, I already said that already.

So there was a lot of other blobs even after I got all of the interrupts working, particularly after this guy, which was really confusing to me. It's a call to a function that then immediately is followed by unusable garbage data. And I was very confused by this because I'm expecting it to go to the next function when it's done.

And even more confusing to me, the code right after this blob were all stubs that were not referenced by anything. So everything seemed weird here. But yeah, so I opened up that function, and the first thing it did was pop twice off of the stack, which was very weird

because nothing was pushed to the stack. Then after talking to IRC people for a while and realizing how much of a noob I was, I realized it was just popping off the return register, or sorry, the return address, which was pointed here. So it was using the return address to get the location of the data structure to parse

to produce what ended up being a switch statement. And there's a lot of this type of stuff on these old APID architectures. This is part of the standard KL library for every time you do a switch, it builds this type of structure. Okay. Me, pop, stack, and then loops. Yeah, I already said that, no surprise.

Actually, the person who helped me with that is presented here a bit ago. He was pretty cool. Oh, shouldn't I answer that too fast? Serious function popped and enabled it, okay. So there's some other weird KL compiler artifacts I found

just because it's fascinating to me. Um, if you're trying to deal with a C pointer, it's able, C assumes you're in one constant architecture, or one constant memory space, but that's pretty much impossible when you're dealing with these types of Harvard architectures with multiple different systems. You can have an address zero

or in three different memory spaces. So if you have a pointer of zero, where are you pointing? And KL's solution to that, and apparently a lot of people's solution is to actually make pointers three bytes despite being a 16-bit memory space, and one of those bytes is a number just specifying what segment to use. And then the pointers, instead of just being done with a move,

are actually handled by entire functions dedicated to parsing the structure. And of course, since it's an eight-bit system, 32-bit math has to be done as functions, which is always kind of amusing to find, but there are some really weird places of like seeing it do a 32-bit left shift, or right shift, eight times,

and then an and operation on the low eight bits to pull out the lower byte when it's literally just four eight-bit registers, and they could just grab the second register without like, I think, 500 instructions. But that's what compilers do, particularly low-le optimized compilers for systems that aren't super popular.

So if any of you do any reverse engineering inside of, dammit, there it is.

Okay, if any of you are doing any work in small chips, particularly old ones, and you haven't had a lot of experience with it, expect things to look like they make no sense, and usually that means it's a either compiler optimization

or lack of optimization, or just part of the standard library that was handwritten. So looking through this code, I was able to find out a couple things about this controller. Originally, the Xilinx controller. Originally, I thought you could send a 16-bit register of how many bits you were going to transition in one packet.

I found out that they actually hijacked an unused eight-bit field from one of the other USB fields, I guess, and actually made it a 24-bit register count, which means that you can send up to 16 million, 700, 700, 700, 116 transitions in a single request of this controller.

So if you have a system where you're writing to a large chip and you don't need to read back stuff to determine what you're gonna do next, it's possible to program several controllers with one bulk USB message. So people tell me this controller's over-engineered. I am actually kind of a fan of it. So looking through here, I found some new commands

and also found out how to initialize a CPLD upgrade system. So I haven't tried reading the firmware off that, but they probably locked it. It'd be nice to be able to free that as well. Harder controlled statement, I already told you that too.

So there was one more unknown binary blob in the firmware and it was 716 bytes long. So in a 16-bit address space, that's actually kind of big. So I needed to look into this because all the GPIF stuff that was loading the waveforms to talk to the second chip were reading from RAM addresses that were uninitialized

when I looked at them statically. So I couldn't get anything out of it. So I looked at the 761 byte thing and here's the actual interrupt, the start function. It's split into two pieces. The green part is the main loop that runs for the entirety of the device running normally and the red stuff is initialization.

This big loop here actually reference that large blob of data and looped over it incrementing the address one at a time and reading the data out. So I took it and I implemented a memory space

and a couple of memory spaces in Python, just an array that could hold stuff, dictionary, whatever, and implemented this branching logic that's here in Python and copied all the bytes over to it, ran it and it extracted a bunch of data.

It turns out they were doing this because in Harvard architecture, you can't initialize memory with the program because that memory has to all be written at runtime. So they instead compress it into code and run it. But again, because of magic compiler nonsense, I think they could have just kept it as code and run it directly out of code

instead of compressing it here, but whatever the compiler wanted to do. So I took those blobs of data and I wanted to understand how these chips were talking. So I took, I looked up Cypress's website and they actually have tools for developers to build these waveforms and they can export the form as a C file

and you can import a C file of that same form to reproduce the waveform and work on it. So I produced a default one, copied all the data I read out of one of the pieces used for the GPIF data structure and loaded it in. And then I have the waveform that's actually used to communicate between the two chips.

There's several of them and they're used in many different places in many different ways, but this is the primary one used for the standard JTAG operations for writing bits to the system. Okay, so yeah, I have pretty much everything I need to start working on the firmware. I understand how the system works, I understand how the chips are talking,

I understand the USB messages, so let's go. I need to go fast on this piece. So started assembling a toolchain, bought a couple boards because I didn't actually wanna use this device for debugging because it has only a light out and I didn't wanna just blink lights at myself for debugging. It turns out I had to do that later,

but I didn't wanna do it early on. So I got a couple development boards, including Cypress's official development board, which they wanted to sell me for 600 bucks, but I found on eBay for like 150, so that was great. I used SDCC, the small device C compiler for this. Found a great library called FX2Lib by,

I cannot pronounce that, but if you ever do FX stuff, open source, it's very good. First tested, just getting lights to blink, took forever to get the chain working, started getting USB in there, USB descriptor tables are awful. I don't like working with them, but they're necessary.

Finally got USB messages working where I could send stuff and turn lights on and off, got debugging over serial working, made the basic commands for the controller, was pretty satisfied with this, and then started working on the actual device, which of course had the problem of only having a blinking light and USB commands, so there was a couple points where I was running code and having it blink at different speeds

to tell me what part it was in the code. Thank you. No debugging and my, whatever, normal debugging stuff. The end result is it actually worked, and I have an open source version of the Xilinx FX2 firmware that you guys can use if you want, as well as documentation on how to use it.

I think this is pretty cool because I see a lot of people doing things like buying bus pirates and other tools for various things, but a lot of people will have these types of controllers, and these guys can actually just talk JTAG or SPI with these commands, and all you have to do is know how they work and have the firmware. So,

there's a couple pieces I need left on this. I have a couple unknown commands. This piece over here is the JTAG operations. This piece right here is magic, and it doesn't seem to do anything. It literally is reading addresses from parts of RAM, what seem, addresses from RAM based on the things,

the messages that you send in through USB, so I think it's a huge debugging structure, but I'm not sure. So, I need to add some tests to it since I haven't really written stuff for hardware. I haven't had experience writing tests for that. Found out that the JTAG commands actually are the same as the SPI commands after dealing with impact again.

Oh, I went back to my docs after six months because I had a job, and I read them, and I went, I hate me, and then I rewrote them all, and then I went back another six months later, and I looked, and it just keeps repeating. If you guys go and look at them, please be gentle.

They work, but... So, improving the docs is something that's nice, and I was considering, since once I get this testing, and I know it works for people in production and not just me, I think it would be kind of cool to package this into an open source firmware blob, maybe distributed through Debian or something.

Okay, so super quickly, I opened up the Atlas test board from Digilent and got its firmware by reading the stuff sent over USB with Wireshark, decompiled that, figured out a bit of how it worked, but I didn't implement the stuff yet. There's some cool features in there.

So, to very quickly run through this, the firmware in docs is a lot of fun and really useful, but I think the JTAG compiler thing that I came up with is actually gonna be more useful, particularly with open source tools, because tools like OpenOCD and AVRDude and other projects

all implement their own controller support, and it's all completely unique and incompatible with each other, and they don't all have the same controllers, and they build it the first way I did it where it's completely unable to take control of the hardware, take advantage of the hardware. So, if you buy Xilinx's new $200 controller,

it's gonna behave just as well as the one you bought for like five bucks, and I believe this is because the focus of open source tools and proprietary systems are different. The open source people just want it to work because they're trying to get something done. They just wanna be able to plug it in and it work, and it doesn't matter how slow it is because they're doing it for free,

but the proprietary tools wanna be able to say we're the fastest, so they put all the work into that, but since they put that work in there, we should be able to utilize it too. And as for OpenOCD, it's an amazing tool, but there is a lot of technical data as anyone who's messed with it. It's open on-chip debugger, but many of my friends refer to it lovingly,

I promise lovingly, as force obsessive compulsive disorder because of the ridiculous amount of configuration options you have to provide it to get it to work, and we shouldn't have to deal with this because this software was originally written in the early 90s, and now we have auto-detective USB and Avahi for detecting over network and all this stuff that should just detect these devices.

So my proposition is that we build a library that actually does the stuff I was talking about of compiling down the commands, and probably in C++, I have all the specifications of how it should work. I just need to get someone who's interested

and willing to talk with me through this so I can design the interface of the actual API. But it should be a lot of the controllers the way I showed. It should, I have no idea what common interface means. Well, it should be able to come with it. It's open source firmware,

and programs should be able to individually access each layer. So if you wanna send direct messages to the low-level layers to get exact control, by all means, you can. Send higher messages, it'll compile down for you. And of course, there's a bit of a problem if you try to send low-level messages to a higher-level controller. It just won't, but that's the price you pay. And if we have this library up,

new projects, like some stuff I've seen from Project Ice Storm, and anybody who wants to build new systems to program certain types of chips, don't have to think about controllers at all. They don't have to build any abstraction for this. All right, common interface was maybe we could have common command line arguments for specifying this stuff

that'd just be passed through to the backend library. So every open tool that's working on this that uses this library doesn't have to deal with these controllers, because none of them want to. They just wanna program the chips. So yeah, questions I have on this is like, should it be a library or a service? Like Pulse Audio being a service, I know that's sometimes a dirty word. Figuring out what language it should be.

Probably C++. I was thinking C because I thought maybe it should go in the kernel, but then I'm like, I don't know. There's no real reason. And because these controllers are actually able to support more than just JTAG, which I probably should have mentioned earlier. That's why I call them in system configuration controllers because it works with JTAG and SPI and IDC

and all these different things. We should have an interface that can grab these devices and present all the features they have and you can grab the JTAG interface or the SPI interface and it would just work. And yeah, so if anyone is interested in that and either wants to talk to me about it or is interested in using it or integrating it

into a product once it's actually working, I would absolutely love some talk about that because I just figured out how it should be built. I just haven't built it yet. And since I have no time left, I want to thank, man, I can never pronounce this right, Dan Ukeru, you, for inviting me to come here and talk.

Dr. Andrew Zonenberg for just helping me with so much of the CoolRunner2 stuff. David Karn, who's here as well, for everything 8051 related he told me. John McMasters for helping running the Silicon Prawn community. And of course, Silicon Prawn for being a great community.

And then lastly, my good friend, Matt Karpeles, who helped me turn this talk into a somewhat coherent thing instead of an autistic mess. And my girlfriend, Caitlin, who listened to every version of this talk, I'm so sorry to her. And all my friends who gave me inputs and helped me actually get the proposal ready and working.

So that is it, and I would say questions, but I think I'm out of time.