Reverse engineering a solar roof datalogger
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/62039 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 202369 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
Computer hardwareInformation securityLocal ringReverse engineeringKernel (computing)Internet der DingeInformation securityComputer hardwareInstallation artExpected valuePoint cloudMereologyTouch typingComputer animation
02:05
Execution unitFrame problemMotion captureRevision controlCommunications protocolInternetworkingControl flowData typePlot (narrative)Product (business)QuadrilateralCore dumpArmSocial classGraph (mathematics)Inverter (logic gate)Ultraviolet photoelectron spectroscopyKey (cryptography)Server (computing)LoginSimilarity (geometry)EmailBeta functionElectric currentFlagoutputGUI widgetBlogField (computer science)Interface (computing)Limit (category theory)Endliche ModelltheorieData storage deviceData loggerComputer networkAddress spacePlotterPower (physics)VoltmeterField (computer science)NP-hardLine (geometry)Inverter (logic gate)RhombusMultiplication signFlag1 (number)AverageEndliche ModelltheorieVarianceError messageSoftware developerPlastikkarteEmailPoint (geometry)GodProduct (business)Multi-core processorReverse engineeringSoftware bugSoftwareOcean currentAddress spaceLimit (category theory)User interfaceCloud computingServer (computing)Data loggerFunction (mathematics)Similarity (geometry)MereologyHacker (term)LoginFrequencyGUI widgetBitGraph (mathematics)outputWeb 2.0NumberData storage deviceComputer animation
11:36
Binary fileStandard deviationInstallation artData storage deviceComputer filePhysical systemStructural loadLocal ringMenu (computing)Inverter (logic gate)Bit rateDirected setString (computer science)Random numberSheaf (mathematics)EmailLink (knot theory)Data typeInformationoutputFunction (mathematics)Server (computing)Hybrid computerDevice driverParameter (computer programming)Loop (music)Event horizonTrailNumberThread (computing)Set (mathematics)WritingReading (process)BlogInclusion mapField (computer science)Military operationFlagDiscrete groupTexture mappingComputer programString (computer science)WhiteboardServer (computing)Keyboard shortcutLibrary (computing)Revision controlExecution unitInformationBinary codeSheaf (mathematics)File formatoutputSource codeCodeWeb 2.0LoginMereologyFormal languageOpen sourceBit rateFlagMappingMenu (computing)Thread (computing)Inverter (logic gate)Total S.A.Hand fanReading (process)Computer fileNumberPhysical systemPoint (geometry)Multiplication signStandard deviationBitTouch typingInstallation artLoop (music)Event horizonEndliche ModelltheorieConnected spaceComputer animation
19:13
Function (mathematics)Repository (publishing)Scripting languageBinary fileSystem callString (computer science)Execution unitQuery languageView (database)EmulatorComputer fileCache (computing)File formatBlogPersonal digital assistantPlot (narrative)AverageFlagSoftwareVideo game consoleoutputDivision (mathematics)Power (physics)Connected spaceDevice driverSoftware testingWebsiteRadical (chemistry)outputWhiteboardQuery languagePoint (geometry)Computer fileSoftware testingKeyboard shortcutMultiplication signVideo gameAddress spacePointer (computer programming)Personal identification numberLoginFlagWordStructural loadAverageString (computer science)NumberOpen sourcePlastikkarteDevice driverFile formatProjective planeLengthMathematical analysisInformationUniform resource locatorBinary codeSoftwareState of matterRepository (publishing)Power (physics)Functional (mathematics)Endliche ModelltheorieArithmetic meanArmKey (cryptography)1 (number)PlotterRevision control
26:40
Program flowchart
Transcript: English(auto-generated)
00:22
Okay, so last talk of the Devereux, Paolo Bonzini is going to be talking about reverse engineering a solar roof panel, sorry roof, yeah, whatever. Whatever he said.
00:54
Here, yes. So as a quick introduction to myself and to set the expectation straight, I'm not a hardware guy, I'm not a security guy.
01:02
This is basically something I did for fun, so I'm a beginner in these topics. On the other hand, I'm not a total idiot either. I know assembly pretty well, I work with compilers, kernel stuff, so I know X dumps enough to do this stuff.
01:21
So anyway, this all starts almost five years ago when I bought a solar roof for my family, and the installer asked about having this optional data logging component, and I didn't really want to have anything cloud related because IoT is short for Internet of Things
01:40
that shouldn't be on the internet, so I didn't want it to be on the internet, but it's entirely local, so I said sure, why not, and this is what I got from them. This is the normal solar roof setup, the stuff that I won't touch with 10 football. This is the part that this talk will be about.
02:02
And it was already suspicious from the beginning, but I mean, who knows, maybe they bought it on sale, I don't know, but the plot thickened when I by chance had Wireshark
02:25
running, and I saw there was a Raspberry Pi that I didn't know of. I have other Raspberry Pis, but none with this IP address, so in part actually a few years later when I was not really preparing for this talk, but I noticed this on their
02:43
website, so these are the specifications for this. It has quad-core ARM Cortex, it even has a micro SD inside, okay, so you know what kind of Raspberry Pi is going to be in there, but anyway, let's take a step back and go
03:01
back to 2018, and let's see what this thing does. So it's pretty nice, I mean, you can only see this from outside, but actually there's extra things around. Here you can see that they power it through the GPIO, in fact the power brick they actually cut out the USB part, and they just screwed the wires to the GPIO here.
03:29
So what does it do? It basically logs data to the SD card every five minutes, it lets you plot nice graphs, unfortunately I don't have any picture of the graphs because I don't use the software anymore, but it also has five relays, you can see them here, like some of them
03:45
have normally closed, normally open, some of them only have normally open. And it's based on five volt inputs, which is a bit weird because usually you use 12 volts or 24 volts DC for communication, five volts is a bit weird.
04:00
And it's not very useful also because you have to actually put the wires on the wall and it was already hard to find a place for the whole solar roof things retrofitted into a relatively old house. Another interesting thing is that it has a built-in UPS, because the inverter has like a 10 ampere line that stays up 24-7, if the grid goes away it's still battery
04:26
powered. So this is another incentive to actually reuse the Raspberry Pi for something else. And also on a slightly lower level, what does it do? It has port 22 open, unfortunately there's no way to upload keys. It has a remote update, but it triggered exactly once when I plugged it in and then not anymore
04:44
for four years. The web server is not Apache, it's not Nginx, it's just an embedded web server with a vanilla server header in HTTP. But it has a nice JSON API, it's easily discoverable with the Firefox developer tools
05:01
and that's how I used this thing for some time. So for example there is the API login, there is the API dash which is basically what is used for the dashboard and it returns the instance data like one minute old at most with some weirdly cryptic names. OK, var is not cryptic, temp is not cryptic, but the other ones are a bit weird.
05:21
And it also has the possibility to get CSV data with similar headers for a particular day or average across the whole year. The daily one is the most useful because if you have a lot of data for the same header, it's easy to figure out what it might be. So for example this one, ENPH, V probably stands for voltage, W stands for watt, but
05:49
H you may not know offhand, but if you look at the number you can see that pretty clearly that it's probably the frequency of the grid or something like that. Because it's about 50. So you get lots of data, you get voltage, current, power values, you get a few computed
06:04
fields that are easy to recognize because the name starts with X. So for example X home is the current consumption of the home independent of whether there is data coming from the, sorry, there is power coming from the solar panels or maybe
06:22
instead it's coming from the battery. The actual data from the inverter are a bit not suited for example to plotting useful graphs, but they have a few computed fields that put things together. There's also a few dozen flags that are interesting.
06:40
They are almost all zero and so they will be a bit harder to reverse engineer. And also you can see the five volt inputs and relay outputs in the logs. I actually never use these so they are always zero, but they are named like in one, in two, in three, in four, in five, so it's pretty easy to figure out. So what you can do with this is already do simple minded hacks with cURL.
07:02
So for example, you can gather all the yearly data and do your own plots. For example, you can see here is when I do laundry because the weekend and Tuesday is where I consume more power. And another thing that I did very early on was push data to MQTT to get some nice
07:20
widgets on my phone that could give me instant data without opening the web interface. And also for home automation, at some point I was using this to turn on and off some Zigbee smart plugs. But this was not any reverse engineering at the Raspberry Pi level.
07:40
All I did was just looking at JSON stuff and doing just stuff with cURL. But this was already enough to find some interesting bugs. There are some weird stuff in the logs. So for example, here you can see this is probably stands for day and hour and this stands for month and seconds, but it's a value that has a decimal point in it so it
08:06
makes no sense and I will show later what happens. Also, there are some fields that end with L and H, but sometimes they're swapped. So you can see that L is the one that stays always the same and H is the one that keeps going up and down, up and down.
08:23
And also with L and H, they didn't really care a lot about them because also if you plot them, the plot looks like this. So this is the total solar production since the day that I installed it. And this is how much it had produced in January 2021, this is in December.
08:45
And it's really weird. What actually happened is that they put the low as the signed value, but it should have been unsigned. So whenever it goes from 2 to the 15 minus 1 to the 15, it actually flips back
09:04
to minus 32,000, whatever. So you can see that this is exactly 655.36 kilowatt hours. And if you fix it, it's still wrong because there are some bugs. But I mean, this was not supposed really to be used by the buyers of the appliances.
09:28
People were just supposed to look at the nice plots. Also, this is way more worrisome because there are flags that seem to be passing correctly. If you look at the logs, it keeps giving these weird names that they keep going on
09:43
and off like a few times every day. And the name is very scary. It's battery charge over the current hard limit and same for discharge. And the weird thing is that you can see this error in the web interface all the time, but you never see it on the inverter.
10:01
The inverter has a little panel with errors, and you cannot see that any time. So that's weird. And I didn't really like that. I asked customer support, which were otherwise very nice, but they just said, yeah, don't worry, it's fine. So all the time while I was doing these things for fun, I had the lingering
10:22
thought that the microSD card would die over me because they say diamonds are forever. I don't know if it's true, but certainly microSD cards are not forever. So even worse, I couldn't just go and say, okay, it's 10 years warranty, please send me another SD card.
10:41
Because the newer models have moved the storage to cloud, and they only have five years of service, which is especially close to the time before my own SD card died. So people won't pay for the cloud service. You won't have to repair the data logger. Anyway, I never did anything about it until last year.
11:03
Last year, I got the data logger to shut down on me twice in a week, and I naively thought that it was just a network problem, and I said, okay, I will just take out the SD card and set up a static address. Actually, the SD card was dying. I don't know why it really had all these problems in general.
11:22
Actually, they didn't have any problem until last November, and last November, it really died in a matter of a week. So I have no idea what happened, but I'm very thankful to the SD card gods for... So when I opened it, this is what it looks like, and this is the behind of Raspberry Pi 3.
11:42
It's potato quality, but... Okay, the first impressions was that it's a standard Raspbian install, so I guess I can forgive them for not obeying the GPL, because they didn't do anything weird. I mean, it was just Raspbian. And there were two statically linked binaries in OMPI.
12:01
One was running on TTY1. Nobody ever noticed that it was running there, because there is no room for an HDMI cable. So it was there, though. The main use of that OMPI thing was that if you plugged a keyboard through the USB and you press there, it rebooted.
12:21
So it probably was a quick way for them to do some testing. But it had some nice ASCII art of the company logo. The other one is the one that we care more about. The data is placed in home pi storage, and the nice thing is that Strace is installed. So I was already thinking of ways to get some data from this,
12:45
because with Strace, it's so easy. Anyway, there's basically, again, stock install. Some files are newer than the others, so that's an easy way to see what's going on and what they changed. There's a couple system D units.
13:03
There's an I2C RTC. This is nice, because at the time, I hadn't even looked at the PCB or anything, so I didn't know what was in there. And knowing that there is a real-time clock on the board is nice to know. They disabled Bluetooth for no particular reason.
13:20
Well, there is a reason, but I will say it later. So what I do is just I copy the binaries to my computer, I add my SSH, and I turn it back on, and it works. So one thing that you need to consider before doing this kind of work is what about the warranty, especially for something as expensive as the solar roof?
13:42
The thing is the connection to the inverter is through USB. It's not direct RS485. There is a screw term. This thing is away from the board, so I didn't touch it. I just removed the screws to take a picture, and that was it. It's basically RS485 to USB adapter.
14:01
So also, the inverter has the baud rate. You can customize it. And the user menu has basically three choices, date and time. What was the other one? It was the total that it reports for the produced energy, so you can reset it to zero if you give it to another person
14:23
or for whatever reason you want to change it, and the baud rate for RS485. So the user is supposed to know that it is based on RS485 and is supposed to look at RS485 data. I don't think this is any kind of lawyer analysis,
14:43
but still I'm pretty sure I didn't break the warranty on the inverter, which is the really expensive part. So anyway, I got the binaries. The first thing I do is using strings, and just doing strings by plus is an awesome source of information.
15:01
And what I noticed by doing quick search, for example, for the API endpoints and for DevTTY USB, is that all the strings are together, and this means the program is unlikely to be written in C, because in C they will be null terminated and they will be all one after another. So could we go, because I mean,
15:22
there's not that many languages that you would use to write a web server in and that produce a large binary. Certainly it wouldn't be Rust because it was four years ago. So it wasn't as fancy as today. And anyway, if you do read-elf, you can see some section headers,
15:40
the gosim, tabgo, pcl, and tab. This is basically the Go format for the debug information. So it's almost certainly Go, like what? Certainly Go. Another thing that you can do for strings is grab for GitHub, because... Why wouldn't people use RPIO libraries from GitHub?
16:06
And also you can find some nice names and things, and this is the name of the model of my inverter, so I know that what they call it in the source code
16:23
or in the files, it can be handy. So anyway, the thing is, there is running this, I can just, I have SSH access, so what I can do is just trace it and see what happens. And one nice thing is that DT2 opens and closes the dev-ttui-usb0 every minute,
16:41
so it's pretty easy to also get not just the board rate, but also the parity, the stop bits. Okay, that's very little, but okay. I will go fast. So with Go, the thing is it has an event loop that can move a sub-outine from one thread to another,
17:01
so you need to track the file descriptor numbers. So what you get here is something that is basically, you can recognize to be mod boost. This is a write, sorry, a read 16-bit registers request. This is a read one-bit registers request. So what I did is I basically took this from the logs
17:24
and I put it in a small C program to debug, to print to the code what it was. I mean, I could probably do something with Wireshark or whatever, but Strace gave me already some C string, so I put it in a C program. And this is enough, for example, to compare with the CSV files,
17:43
at least for the 16-bit registers. So for example, I can see now that these are probably low and high. So this was the minutes and seconds, the hour, the day, the month, and the year. This is 21 in hexadecimal.
18:01
And I can also see that some values are fixed points. So this one is the version multiplied by 100, and this one is the temperature multiplied by 10. For the discrete inputs, it's a lot more complicated. I could find in strings some nice names of the fields.
18:23
So for example, GRN was no voltage from the grid, and so on. It's a bit weird that they put it on alert that there was a blackout, but they didn't put as an alert that the fan broke, but whatever. And this also doesn't make lots of sense
18:41
to average the bools, but whatever. So anyway, this is already nice because I have the names corresponding to all the fields, but I don't have the mappings of the discrete inputs. That's what ModBools calls the one-bit Boolean values to the flag. And I knew that this was the part that was broken in the code.
19:01
So this was probably not going to succeed, but actually it was successful. I used radar2 for this, and I will super, super quickly go through radar2. This is what I learned about radar2 because I had never used it before. So the comments are one letter per word. So ADF means analyze data in functions.
19:23
There are some people here that are old enough to remember Lotus 1, 2, 3. That's the way the menus worked in Lotus 1, 2, 3. Basically, you had one letter per word in the command that you wanted to execute. And the main ones are seek, print, and slash for search.
19:42
And another interesting thing to know is that the state of your work is saved in a project, which is actually a single file in a Git repository with thousands of comments in it that say all the nice things about your binary. You can get some information from the
20:06
debug info that I showed earlier. There's a nice command to do all the analysis that is possible, but it doesn't really work for a static binary linked binary. So I use these ones instead. It starts giving some nice information.
20:22
So for example, in the project file, after I do the analyze strings commands, I can see that it has these CS commands, and now when I do the disassembly, it actually prints this as a string and not as instructions. Likewise, when I do a XD,
20:41
I can see that it loads these, and it also says what is the data that is loaded from here. This is probably in the custom pool. So for example, this instruction is the one that loads the address of the TTY USB 0. One thing that you can do is also you can write your own commands and add them to the file.
21:00
So for example, here, I know that this location is a data operand for this instruction, and I can tell radar2 for everything that is in here, make it dumped as bytes, not instructions. So after I add all these things to the project file, it will not be dumped as a rubbish ARM instruction.
21:23
It will actually print the word. Then you can also search keys. For example, if I search for the two flags that went up and down, I can see that they are here, and here they are closed, so I decided to search for this address.
21:42
You have to put it backwards because it's little endian, so this is the first byte and this is the last one. But then I found that they were found relatively close, like 68 byte apart, so you dump them, and it's very helpful that it even tells you where the hits were from the previous searches.
22:03
And here you can see some nice things. It seems to repeat every 68 bytes, and it's always pointed to the string followed by the length. And also, there's these nice numbers
22:21
which might be maybe the numbers of the discrete inputs, who knows? And if you go back and back, you can see using the seek command, I can go back 68 bytes at a time, and sooner or later, I get to a point
22:41
where the format changes, 68 bytes before, there's nothing like what was afterwards, so this was the beginning of the array. And now I know exactly which discrete input was with which name and so on.
23:02
I can also tell whether to print data, so for example, this was a floating point number, and if I print it, it's 0.1, so the guess is that this would also be something related to the fixed point values. And now it's time to actually find the pointer to this,
23:21
I search for this address here, and I find it here, it's also addressed followed by the number of entries, so it's probably some kind of array descriptor for Go, and then I search for the address of the descriptor, and then if I go here, I see that there is
23:43
a reference from the query function of my model of the inverter, so I guess we have a winner. And in fact, what I did then was rewrite the software using Modbus, it outputs the logs in exactly the same format, so I still have a continuous log
24:01
from the date of the installation, except I didn't leave it exactly the same, I fixed the bugs, so no averages of dates, the scary flags are not logged anymore, I can see now whether it's using the battery or not, and I even got the grid non-flag during a blackout,
24:22
so I guess that's full confirmation that it works. It also does the same thing that I was doing before with cURL, now I do it natively, so every minute I export it to MQTT with Home Assistant, I don't have the plot functionality, but I can get it from Home Assistant, so that's fine, this is the source code, and sooner or later
24:41
I will try to put an Ansible playbook, so that if the SD card dies once more, I will just have a very quick way to deploy it. As a bonus for the last minute of the talk, I have a picture of the PCB, because at some point I wanted to update to the Debian 11, the nickname changed, so it didn't get the network anymore,
25:03
I had to really connect the Raspberry Pi to the keyboard and monitor, so here's the PCB, it's a work of art, it's all two whole components, don't ask me why. The inputs are voltage dividers,
25:22
so learning that the blue resistors are the more precise ones, finally paid off after like 40 years of my life. Here's the battery-backed RTC, there's a power LED and an alert LED connected to the GPU, this is nice, this is a driver IC for the relays
25:42
and also for the LEDs because these are powered in five volts, not 3.3, and there's also, this is nice if somebody wanted to hack further on it, these are test pins and they are connected to eight more GPIOs on the Raspberry Pi, I think. But there's something really weird here, because this part, these two terminals are not exposed,
26:06
they're unused, and this is a bias resistor, this is a terminal resistor, this is another RS485 transducer, because remember that it was connected through USB, and it actually works, I have no idea why it's there, but if you look at the website,
26:21
there is probably an older version of the board where you can actually read here common A and B, so at some point they wanted to use it, and then they didn't, so on the brochure picture, this is not used on the website feature, they still have the older version, and that's it.